2

I want to solve Synchronization with a Message Queue

In the wikipedia page of message queue, I read:

Most messaging systems support both the publisher/subscriber and message queue models in their API

I used python-rq (celery in the past), with fun and success.

.... but I don't understand the (academic) difference between "publisher/subscriber" and "message queue models".

What is the difference between "publisher/subscriber" and "message queue models"?

guettli
  • 180
  • 1
  • 15

2 Answers2

5

As per the wikipedia page you linked to regarding message queues:

Message queues provide an asynchronous communications protocol, meaning that the sender and receiver of the message do not need to interact with the message queue at the same time. Messages placed onto the queue are stored until the recipient retrieves them.

(Bolded emphasis mine).

From the wikipedia page for the Publish-subscribe pattern:

In software architecture, publish–subscribe is a messaging pattern where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead characterize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.

(Again, bolded emphasis mine)

To summarise - in any messaging pattern, the concept of sender and recipient always exists; the differences between message patterns concern the way in which a message is delivered to a recipient.

Message Queues

  • A message queue pattern is a form of reliable message delivery - i.e. the sender ensures that the data they send will arrive into a queue (a message buffer). Though does not guarantee the recipient will actually receive it.
  • The message buffer/queue retains the message for as long as needed, until the recipient is ready to collect their message (or the queue is purged/destroyed).
  • Typically, a message queue will exist on some kind of middleware or broker platform (i.e. a sub-system whose purpose is to act like a 'post office' for messages, ensuring that messages are routed to the right recipients)
  • Typically, each recipient will have their own queue.
  • It is incumbent on the sender to make sure the recipient queue(s) is/are available before sending the message.
  • One common pitfall of message queues is the risk of stale data - i.e. messages which have been enqueued, and are out-of-date by the time they are received. A common (simple and possibly naive) solution to this might be for a publisher to ensure all queues are freshly purged before sending their first message.

Publish/Subscribe

  • Describes fire-n-forget patterns where a sender sends a message without knowing or caring whether the message will be delivered.
  • The sender doesn't know or care anything about the recipient; the pattern could work with Zero, one, or many recipients without any effect on the sender.
  • "late subscribers" may miss out on earlier messages which had been sent before they started subscribing.
  • The "middleware" for publish/subscribe can be as simple as a basic notification system (e.g. the Event Aggregator pattern), or it might be a more advanced message broker system.
  • One or more recipients use some message meta-data (such as the message type and/or other descriptors) to "listen" for any messages of that type
  • Senders attach the meta-data (e.g. message type, etc.) to the message which recipients can "subscribe" to in order to receive the message.

Lastly, there can be some cross-over in terms of the implementation of these patterns; a publish/subscribe pattern can be implemented using a broker which supports message queues - these patterns are closely related because they both achieve the goal of de-coupling senders from receivers, so there will naturally be a lot of overlap in the way each of these patterns are implemented.

Modern messaging middleware tools tend to support both patterns straight out-of-the-box with minimal differences in the code needed to use those patterns. (A popular example might be RabbitMQ).

Ben Cottrell
  • 11,656
  • 4
  • 30
  • 41
  • Thank you. I see this big difference. The pub/sub pattern works like this: If there is no one processing the message, then the message will be lost. The message does not get stored until someone processes the message. Is this correct? – guettli Apr 10 '17 at 09:25
  • @guettli Correct - that is one important difference; another important difference with a message queue is that the sender takes some responsibility in ensuring the delivery to a queue, whereas in pub/sub the sender simply doesn't care anything about the recipient. – Ben Cottrell Apr 10 '17 at 09:39
  • I have no use case for pub/sub in my context. If the sender does not care for a reliable message delivery, then why does he do it at all? I guess you can compare this to UDP vs TCP. If a UDP package gets lost, no retransmission happens. In my case reliable delivery is important. Thank you for answering my question. – guettli Apr 10 '17 at 10:38
  • @guettli A possible use-case for "fire-n-forget" might be a 'stream' of time-relevant update/snapshot messages which are frequently being invalidated by subsequent updates/snapshots. – Ben Cottrell Apr 10 '17 at 11:31
  • @guettli I would also add that another possible option for reliable delivery could be a [Request-response pattern](https://en.wikipedia.org/wiki/Request%E2%80%93response) which is useful for scenarios where a sender needs to be told when a message has been received and processed. (i.e. the sender waits until either a timeout or response). – Ben Cottrell Apr 10 '17 at 11:32
  • I want to implement a worker-farmer pattern: One central process (farmer) and several workers. – guettli Apr 10 '17 at 12:49
1

In general, publisher/subscriber messaging systems work on topics. 1 or more publishers (producers) publish messages into a topic, and 1 or more subscribers (consumers) subscribe to a topic and read information from the topic. The underlying implementation of this concept can be done using message queues, for example, each subscriber for each topic will have its own message queue which publishers push data into.

For message queues, it is generally 1 or more producers pushing messages into a single queue, and a single consumer reading from the queue.

When you need a many-to-many messaging model, the publisher/subscriber concept will fit the bill, but if you are looking one-to-one or many-to-one, the message queue model is simpler to implement.

If you just need to use a C++ queue with synchronized access to objects without mucking about with mutexes, I have had success using the lock-free queue: https://github.com/cameron314/concurrentqueue

William L
  • 104
  • 3