Questions tagged [producer-consumer]
27 questions
33
votes
3 answers
How to implement a message queue over Redis?
Why Redis for queuing?
I'm under the impression that Redis can make a good candidate for implementing a queueing system. Up until this point we've been using our MySQL database with polling, or RabbitMQ. With RabbitMQ we've had many problems - the…
djechlin
- 2,212
- 1
- 15
- 26
16
votes
1 answer
Difference between Consumer/Producer and Observer/Observable
I am working on the design of an application that consists of three parts:
a single thread that watches for certain events happening (file creation, external requests etc.)
N worker threads that respond to these events by processing them (each…
user183536
- 163
- 1
- 1
- 5
6
votes
2 answers
Understanding the Single-Writer Principle
The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and how instead we should be using only single…
BVN
- 81
- 1
- 4
5
votes
2 answers
What are the distinction and relation between batch processing and stream processing systems?
Design Data Intensive Applications says
Batch processing systems (offline systems) Chapter 10
A batch processing system takes a large amount of input data, runs a
job to pro‐ cess it, and produces some output data. Jobs often
take a …
Tim
- 5,405
- 7
- 48
- 84
5
votes
2 answers
kafka consumer properties session.timeout.ms vs heartbeat.interval.ms
I am reading Kafka, the definitive guide and I came across the below point for consumers.
heartbeat.interval.ms must be lower than session.timeout.ms, and is usually set to one-third of the timeout value.
May I know why is it, I thought both should…
Ramya B
- 59
- 2
5
votes
1 answer
Parsing Multiple Files and Their Contents in Java using Multithreading without ExecutorService
I'm learning concurrency in Java and went over the tutorials on Oracle website. While I have understood some of it, a greater portion eludes me. I was thinking of a hypothetical problem (though it may or may not be a good case of using multiple…
Shehryar
- 163
- 1
- 1
- 5
4
votes
1 answer
One producer - multiple consumers queue, best way to guard against rare duplicates?
I have a classical one producer - multiple consumers queue, each item can be processed independently. On rare occasions (<1%) there can be multiple queue items related to the same object the consumer is processing. It is perfectly fine to have that…
Dmitry Streblechenko
- 151
- 4
3
votes
2 answers
Is it a good practice if using Object when a method can accept more than one type of object?
public class TaskA implements Runnable {
....
}
public class TaskB implements Runnable {
....
}
I have two runnable class as TaskA, TaskB. I want to have a TaskManager to run these two tasks and have possibility to add more task in the future. So I…
c2340878
- 79
- 1
- 3
3
votes
1 answer
Producer/consumer problem with 2 mutexes?
The setup
Let us consider a variant of the classical multithreaded programming problem: the producer/consumer problem. We have some finite number of resources (for simplicity, let there be just a single resource). The producer thread continuously…
Dmitry Serov
- 139
- 1
- 5
3
votes
1 answer
How to recognize consumers in AMQP architecture
I'm doing an project where I'm using a microservice architecture. All the services within this architecture communicate with each other using an AMQP broker (RabbitMQ).
When something happens in a service, it publishes an message to RabbitMQ and the…
CGeense
- 191
- 1
- 5
3
votes
0 answers
An efficient way of starting an arbitrary number of consumer threads?
I have a Producer/Consumer implementation where the number of consumers is configurable (this is a form of configurable throttling). The producer and consumer are kicked off like this:
var cts = new CancellationTokenSource();
var processCancelToken…
user833115
- 131
- 1
3
votes
0 answers
How to schedule large numbers of user-defined tasks?
Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails).
You could run a cron which SELECTs a database table for unprocessed events before the current time (or even a long…
james.haggerty
- 173
- 2
2
votes
1 answer
What is the currently best C# implementation for a Producer/Consumer scenario?
I have a relatively simple task where I need some 10 consumers to consume work to be produced into a queue, continuously.
This is my first time implementing this design pattern, so I have been searching the web for different approaches.
The ones I…
Veverke
- 383
- 2
- 15
2
votes
3 answers
How to build rest calls dynamically to interface with different APIs?
Lets say I am building an application where clients can book bus tickets. The only thing they do is select, origin, destination and the type of service (express or standard). The system will then book the ticket against the right bus company api.…
masber
- 327
- 1
- 5
- 9
2
votes
2 answers
Solve Synchronization
I want to solve Synchronization with a Message Queue
In the wikipedia page of message queue, I read:
Most messaging systems support both the publisher/subscriber and message queue models in their API
I used python-rq (celery in the past), with fun…
guettli
- 180
- 1
- 15