Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [queueing]

19 questions
13
votes
3 answers

Are we queueing and serializing properly?

We process messages through a variety of services (one message will touch probably 9 services before it's done, each doing a specific IO-related function). Right now we have a combination of the worst-case (XML data contract serialization) and…
Bryan Boettcher
  • 2,754
  • 4
  • 21
  • 31
4
votes
1 answer

Seeking Guidance on Thread Safe Scripting

I am trying to write a script that will take multiple paths to files on various servers, search them all simultaneously, and return a single list of results to a user. Initially, I was just using Python threads to do this, however I quickly ran into…
Elias51
  • 43
  • 5
3
votes
3 answers

Strategy for ensuring consistency when integrating using an API?

How do I ensure consistency between two separate systems which store and maintain the same set of data, when the method of communication between them is REST APIs and webhooks? Is it possible? I find myself coming across this problem when working…
Alex
  • 159
  • 5
3
votes
2 answers

Is it worth it to use a message-queue framework in this case?

We have these things called "executions" that are run at certain times. They have a delay property that basically says "execute me after delay microseconds`. Currently we are using Cassandra as a queue, but this is definitely not scalable, and the…
Vivin Paliath
  • 195
  • 9
3
votes
0 answers

How to schedule large numbers of user-defined tasks?

Say you're running a website which generates a large number of tasks that should be run at particular times (e.g. reminder emails). You could run a cron which SELECTs a database table for unprocessed events before the current time (or even a long…
james.haggerty
  • 173
  • 2
2
votes
1 answer

How to detect end of queue in a parallelized web crawler?

Let's say I want to program a parallelized web crawler which has a shared FIFO queue (multi consumer/multi producer). The queue only contains URLs. How do I detect the end of the queue? A worker process is always consumer and producer at the same…
jervis
  • 41
  • 1
2
votes
2 answers

Shared & exclusive locking

I need a "lock" that can be either shared or held exclusively, that will provide the following behavior for the following sequence of events: Process A: Requests and is granted a shared lock Process B: Requests an exclusive lock, and is blocked by…
Chap
  • 723
  • 1
  • 7
  • 18
1
vote
1 answer

AWS multiple SQS queues on one bucket without fanout

I am coming into a project that uses an Amazon S3 Bucket for a process. For my application, I need to know (within a reasonable time - order of a few minutes, at most) whether a new file has beeen added to the bucket, and then download that…
SRNissen
  • 151
  • 5
1
vote
0 answers

How to have only one consumer handle a long running task at a given point?

I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items for all Users. Within a ListOfItems requests per…
Yos
  • 155
  • 1
  • 10
1
vote
1 answer

Queuing emails in Laravel when using an api driver such as Mailgun?

In Laravel (for the record I'm using v5.6) there is an option to queue emails for background processing. I have a scheduled job to send out daily email alerts to users. Currently the job drops the emails (mailables) in the Laravel database queue and…
adam78
  • 291
  • 3
  • 9
1
vote
1 answer

How do I set up short-lived queues?

This scenario seems pretty ordinary, and yet, strangely, messaging systems (like Google Cloud PubSub and Task Queues and ActiveMQ) do not seem to support it -- they assume that topics/queues are long-lived. A frontend webapp server sends a…
Joshua Fox
  • 1,090
  • 10
  • 18
1
vote
1 answer

Difference between server Jobs and Workers?

I am trying to understand the difference between server workers and server jobs. Google has not been very helpful because the keyword "jobs" is being construed as an employment term... I have yet to find an article answering my question. :/ Would…
comixninja
  • 121
  • 3
1
vote
1 answer

How to queue work orders by date and priority?

In my organization I work on orders based on date, so earliest work orders first. We also have an option for order to be marked "priority" in which case those orders are done first but not always. I've asked my boss some questions and I couldn't get…
Liger86
  • 13
  • 2
0
votes
3 answers

Queue for tasks and notifications

I've been a heavy user of "batch jobs" to perform different kinds of logic in systems. Over the last couple of weeks, I've been thinking and reading about other approaches and I wanted to bounce some of my ideas and get some feedback and trips. So…
Markus Knappen Johansson
  • 192
  • 6
0
votes
0 answers

What would be the best way to queue asynchronous calls received simultaneously by a web service cluster?

I have an API where every time is called there are multiple calls to multiple webservices in the background, sometimes it takes up to 20 seconds to process the request calling up to 10 different web services depending the specific use case. The API…
Mario Arturo
  • 109
  • 1
1
2