Software Engineering Stack Exchange
Software Engineering Stack Exchange

Questions tagged [queue]

68 questions
10
votes
2 answers

How to best design a job queue with constraints?

Consider the following situation: You have a program which creates numerous 'jobs' that need to be processed and places them into a queue. You have other worker programs which grab the next 'job' in line so they can process that job. Each job has a…
merp
  • 101
  • 1
  • 5
10
votes
1 answer

Message Queue for RTOS for microcontrollers

I'm currently writing an RTOS for microcontrollers. The whole thing is written in C++11 - if anyone is interested, and the link to the repository is at the bottom. Currently I'm writing a class that is a simple data queue for passing objects between…
Freddie Chopin
  • 241
  • 1
  • 5
9
votes
3 answers

When designing a job queue, what should determine the scope of a job?

We've got a job queue system that'll cheerfully process any kind of job given to it. We intend to use it to process jobs that each contain 2 tasks: Job (Pass information from one server to another) Fetch task (get the data, slowly) Send task (send…
Stu Pegg
  • 568
  • 2
  • 13
9
votes
13 answers

What are some good, simple examples for queues?

I'm teaching CS2 (Java and data structures), and am having some difficulty coming up with good examples to use when teaching queues. The two major applications I use them for are multithreaded message passing (but MT programming is out of scope for…
Michael Ekstrand
  • 367
  • 1
  • 2
  • 8
8
votes
2 answers

Fair distributed task scheduling with RabbitMQ

Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. Any other client that subsequently submits a…
Johan Martinsson
  • 189
  • 2
6
votes
2 answers

Design question for handling large volumes of messages in multi-tenant queue

I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario. Producer produces events based on various inputs (user interactions, api, etc...)…
Taylor
  • 169
  • 4
6
votes
3 answers

Large doubly-linked-list (or other) storage on disk for queue system; options for how to store

I'm looking at building a message queuing library in Go, which will be used as part of a larger application. A doubly-linked-list seems like a sensible approach for an in-memory data structure, but let's say this list grows to be 4 million items and…
Brad Peabody
  • 239
  • 1
  • 7
6
votes
0 answers

I need to write my own version of a bounded queue

I need to write my own version of a bounded queue. It must be thread safe. The consumption of this queue is based on a priority system, where a consumer makes a request for an object in the queue. The queue can have many producers and many…
TheCoder
  • 179
  • 2
6
votes
3 answers

Slow throughput - still worth using a dedicated message queue?

Here are some requirements for a queue: Every few days add ~100k tasks with various priorities Workers will pull tasks at typically less than 10 / second Tasks need to be completed by ~2 unique workers (for error checking), and then depending on…
hoju
  • 163
  • 5
6
votes
3 answers

Use Queue or stick to native f# lists

I need something that represents a queue in F#. The queue will be filled with items once, and (as the application progresses) items will be taken from this queue and moved to other lists. Should I use the Queue class in .NET, or should I avoid it,…
TimothyP
  • 233
  • 3
  • 8
5
votes
3 answers

Queue vs Threads

I'm implementing a data processing software. The software gets form the network thousands of events that must be processed in according to rules. I implemented a multi-thread service, which receives the event and insert it into the processing…
ender.an27
  • 153
  • 1
  • 6
5
votes
1 answer

Querying large amount of data for parallel processing

I have a dataset containing list of users (around 50M). Each user has an email address, name, and some more data columns. I want to send a weekly email to those users, and the content of the email will be based on the user's data. Each user should…
ItayMaoz
  • 159
  • 3
4
votes
4 answers

Designing an efficient implementation of a random access queue based on a linkedlist

I'm currently working through Robert Sedgewicks "Algorithms in java" (3rd edition, german) on my own and am currently at one of the more complicated questions there. I think I may have the starting point for a solution, but I'm not sure if its…
Philipp Doerner
  • 163
  • 1
  • 9
4
votes
1 answer

What happens differently when you add a task Asynchronously on GAE?

Google's doc on async tasks assumes knowledge of the difference between regular and asynchronously added tasks. add_async(task, transactional=False, rpc=None) Asynchronously add a Task or a list of Tasks to this Queue. How is adding tasks…
Ben
  • 165
  • 5
3
votes
4 answers

Iterating through stacks and queues?

Are stacks and queues allowed to be iterated over, conceptually? It “feels wrong” for some reason but I can't come up with a reason why it would be bad to see through the whole thing that's stacked/queued. Obviously no taking the element out of…
Lazar Ljubenović
  • 150
  • 7
1
2 3 4 5