4

I am porting an event-driven architecture to a container environment (Docker) and I wonder how to take advantage of replication while using pub-sub: if I just increment replicas I end up with many process doing the same elaboration and I don't think is a good way to go. I thought to replace pub-sub with producer-consumer, so consumers waits for events from a queue and only one grabs it, but I would like to avoid tightly coupling.

Is there any solution to have a "load balanced" publish-subscribe? Also commercial tools can be checked.

Marco Stramezzi
  • 665
  • 2
  • 6
  • 16
  • Where are you seeing a tight coupling in the producer-consumer scenario? – Bart van Ingen Schenau Sep 17 '19 at 07:12
  • 1
    Have you had a look at kafka’s design ? It seems to address exactly this kind of issues. If scaling is limited, you can also consider splitting the processing part from the queue management so that you can multiply workers without adding the complexity of a distributed queue processing – Christophe Sep 17 '19 at 07:15
  • @BartvanIngenSchenau If N consumers exist (not replicas of the same but different logics), I need a queue for each of them, otherwise only one consumer handle the event. Am I wrong? – Marco Stramezzi Sep 17 '19 at 07:17

1 Answers1

1

What you need is supported out of the box in RabbitMQ (and many other frameworks as well). This is one of the basic feature.

What are you needing here is a feature called Competing Consumers.

In RabbitMQ When you want one message to consumed by one of the many consumers. You set the bindings such that the message is routed from the exchange to one queue, and make multiple consumers listen to this queue. Then only one of the consumer will pick up the message. This is called competing consumers

When you want multiple consumers to consume one message. You set the bindings such that the message is routed to many queues from the exchange. Then the messages can be consumed by multiple consumer each listening to different queue

You could read https://www.rabbitmq.com/getstarted.html The scenarios are shown as 2 and 3 in the link above

Nachiappan Kumarappan
  • 1,404
  • 1
  • 9
  • 19
  • Thanks for your suggestion, but what you are showing is a producer-consumer implementation. I am asking if is possible to avoid it and keep using pub-sub. – Marco Stramezzi Sep 24 '19 at 06:24
  • @MarcoStramezzi, Yes... The link I showed was using RabbitMQ as a producer-consumer. RabbitMQ supports pub-sub patterns as well. You could check the link https://www.rabbitmq.com/tutorials/tutorial-three-python.html To get this to work we have to set the queue to exchange bindings, and we can have competing subscribers. – Nachiappan Kumarappan Oct 04 '19 at 12:30