Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [rabbitmq]

58 questions
8
votes
2 answers

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. Any other client that subsequently submits a…
Johan Martinsson
  • 189
  • 2
8
votes
1 answer

Making API calls with celery

I'm designing a system for a client where the requirements are: they upload a JSON file (one object/line) make a call to an API with the JSON object as the payload record the state (success/failure) of each API call in a database make one retry if…
user154706
  • 101
  • 1
  • 5
5
votes
1 answer

REST APIs vs Message brokers for Backend to Backend communication

We are a wholesale marketplace and we have three main products in our startup. The products are: 1- the marketplace app that allows customers to view products and purchase online. 2- the seller app that allows the sellers to list their products and…
Eslam Nasser
  • 61
  • 1
  • 2
5
votes
1 answer

MQ integration: How to notify consumers of upcoming message format changes?

We have multiple microservices communicating over MQ. As MQ messages are the interface/contract between the services, whenever we make changes to the MQ message published by a service we need to make the same adjustments on the services which…
kenneho
  • 151
  • 1
3
votes
1 answer

Data processing pipeline design for processing data

I have a use case for which I need to build a data processing pipeline Customer contact leads data coming from different data sources like csv, data base, api has to be first mapped to a universal schema fields. There could be ~100k rows coming…
Sam Rohn
  • 139
  • 1
2
votes
3 answers

RabbitMQ or REST API Public

Currently working on designing an application that will take in JSON data from remote client applications. The data these client applications are sending is going to go into a Message Broker, namely RabbitMQ, to be later processed. In the…
Skirek
  • 21
  • 2
2
votes
2 answers

How to share API between microservices?

Currently, our architecture uses an "API-first" approach in building our product. This product is divided across multiple teams handling different microservices. The above image succinctly shows our approach. There are JSON Schema files that are…
Nishant123
  • 165
  • 4
2
votes
1 answer

How to decouple messaging middleware from microservices?

We have a bunch of microservices written in different programming languages. These all communicate with each other over RabbitMQ(AMQP) middleware. There are consumers and publishers in each microservice which listen to incomming messages and publish…
Nishant123
  • 165
  • 4
2
votes
1 answer

AMQP messaging: How to generate documentation for consumers?

My application publishes various messages (e.g., telemetry, changed-events, ...) to a RabbitMQ message-broker. The users have no access to the source-code and therefore, will need a separate documentation with all available exchanges, messages and…
SoftwareArchitect123
  • 51
  • 6
2
votes
0 answers

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-server and n-clients. The server has several tasks to…
Daniel Stephens
  • 121
  • 10
2
votes
0 answers

Data pipeline architecture: airflow triggered by message broker

Let us say we have: a web app with a Postgres DB that produces data over time, another DB optimized for analytics that we would like to populate over time. My goal is to build and monitor an ETL pipeline that will transform the data and write it…
sunless
  • 151
  • 3
2
votes
2 answers

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find any info in the manual. I'll outline a basic…
Daniel Benzie
  • 269
  • 1
  • 5
2
votes
1 answer

MQ Architecture: How to handle old messages in the queue and releasing a consumer/producer upgrade

So for example if I have a producer that sends a message {id: 1} to a consumer that expects that json object. And there's a ton of messages sitting in the queue waiting to be processed. I'm also writing some code in the producer & consumer so…
Luke Xu
  • 123
  • 3
2
votes
1 answer

Message Broker vs. iPaaS

Does anyone have suggestions/recommendations/guidance they can offer on when to use a message broker like RabbitMQ/Kafka vs. an iPaaS solution like Mulesoft/Jitterbit/Dell Boomi/Workato? Can they coexist peacefully? Or should one of the two…
Andy
  • 397
  • 3
  • 11
2
votes
3 answers

Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?

My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro service receive async messages from our internal…
Dherik
  • 2,406
  • 20
  • 33
1
2 3 4