4

I have a job that runs at a fairly high frequency (~1x every second).

This job needs to be executed constantly as long as the web app is running. However, I really only want 1 instance of the job running at a time.

Currently, I just fire up a thread that performs the job in the background, sleeping for ~800ms each loop. That causes a problem when I want redundancy, because the thread starts up as part of the app.

I know I can do something simple, like add a table in a database and have a "primary" and "secondary" server lock the record, track who's in charge, etc. But it seems a bit ugly, so I was curious if anyone had experience with a similar situation, and what solution you ended up using?

Jerod Venema
  • 141
  • 5
  • 1
    Is it important that the app is a single instance that fails over? Or are multiple workers that each take a piece of the load (and if one fails another picks up the slack) acceptable? –  May 13 '15 at 03:24
  • Single instance is simpler - due to some performance characteristics of the data I'm using, I (currently) pull the dataset, work on it in memory, and then push only the changes that are necessary. Typically I might have 50-100 objects at a time being considered, but every loop will typically only modify 0-1 of them. If I used multiple instances, I'd have to pull subsets, and figuring out which "subset" a given worker is working on would be tricky. That said...maybe there's a good way to get a "subset" on each loop somehow, so if server A goes down, server B would get both subsets... – Jerod Venema May 13 '15 at 20:36

4 Answers4

3

I see this ultimately as a problem of interprocess communication (IPC). You have one process generating work and then another one consuming the tasks and doing the work.

This falls into a classic producer and consumer layout (tutorial from RabittMQ).

The nice thing about message queues is that they guarantee the contract of "this message is guaranteed to be delivered to one, and only one consumer (if it is consumed at all)." This simplifies the IPC as you don't need to worry about the semaphores and shared memory segments or other channels of communication.

Beyond this, you can start to get fancy. You can have two consumers running - though only one will grab the message indicating the task from the queue. You could have the producer pumping out single "consider this" tasks as fast as it can (up to a certain queue fullness) and the consumers processing them once every 800ms - though the exact implementation depends on the problem.

The message queue has an advantage over databases in that its what its designed to do. You can do similar things via mechanisms in the database, but then you will start worrying about transactions and locks. Some databases have gone on to take additional abilities of message queueing as part of the core set of features as described in this post, but a message queue has a general advantage over databases being able to handle more concurrent messages, blocking until you get a message, avoiding deadlocks and races.

How one designs a solution using a message queue for this specific problem is still a bit up in the air as there are many aspects of the domain that are left undescribed, but in general the message queue should be one of the first tools that comes to mind when one starts thinking about how to send messages between one or more processes.

  • I think you've nailed it, and I need to reconfigure to a message queue here. A bit awkward given some of the other system constraints, but the right solution I think. Appreciate your feedback. – Jerod Venema May 14 '15 at 12:24
3

I can recommend the Hangfire as a solution. Key highlights for your case (extracted from the web site),

Persistent

Background jobs are saved into a persistent storage – SQL Server, Redis, PostgreSQL, MongoDB and others. You can safely restart your application and use Hangfire with ASP.NET without worrying about application pool recycles.

Reliable

Once a background job created and placed into a storage without exceptions, Hangfire takes the responsibility to process it at least once. You are free to throw unhandled exceptions or kill your application – background jobs will be re-tried automatically.

Distributed

Background method calls and their arguments are serialized and may overcome the process boundaries. You can use Hangfire on different machines to get more processing power with no configuration – synchronization is performed automatically.

Low Flying Pelican
  • 1,602
  • 9
  • 13
  • I think this may actually be the best option...my job runs 1x a second, so that's the level of granularity I need, but the time-based stuff is known ahead of time - I need to check it 5s after creation, 10s after, 30s after, etc. So pushing that to a scheduled job that can handle the distribution may work... – Jerod Venema May 15 '15 at 15:30
  • upvoted any ideas on a similar solution for node – PirateApp Apr 16 '19 at 07:04
0

Good Question,

I would move the task out to a seperate service. This gets around your link with the website but you'll still have to make the service redundant. You can go two ways with this.

1: services dont know about each other. If you can change the job so that no error occurs when it runs concurrently with itself this is the best solution as you can simply fire up 2 or more services and not worry.

2 the services know about each other. This is more complicated but worth it if the job is long running as it allows you to spread the load over workers. You can use a queue (msmq,zmq,rabbit) or a database table (mark jobs as in progress) to affect the communication

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • #2 is the only way I can do this realistically, I'm looking into queue options but unsure yet if this will resolve the problem, since my actions are driven not be events but by time. – Jerod Venema May 15 '15 at 14:01
0

The task should be running as a separate 'service' which starts when the server starts. You can use a singleton pattern to ensure there is never more than once instance running. For a .net specific solution look at Mutex. It creates a system wide singleton pattern.

While you say the process should only be running when the web app is what is really important is that the process completes some periodic work when the web app is running. It can accomplish this a number of different ways. One would be to have the service listen to see if the web app is running between sleep cycles.

The one thing you don't have is the ability to detect if the service fails to run and restart it. There are ways to accomplish that as well plus throw in alerting but if you are going to all that you should look into making your 'service' an actual windows service.

Gerald Davis
  • 136
  • 4
  • This doesn't really address my problem. It doesn't matter that it's running as part of the web process, it matters that I'm tied to a single instance of the job running at a time due to the application logic, which running as a service doesn't resolve. – Jerod Venema May 15 '15 at 14:00
  • Well it depends on what you mean by service. Service as just a generic term for a background process. That is the point of Mutex. Mutex will ensure there is only a single instance of the background process running at one time. If you want a more comprehensive solution creating a windows service will give you restart on fail, logging, etc while still ensuring there is only a single instance of the service running. It would not be possible for two jobs to be running at once. – Gerald Davis May 15 '15 at 14:33
  • I don't need just a single instance *on a single server*. I need to synchronize *across servers*. So the mutex is irrelevant unless it has synchronization across physical machines. – Jerod Venema May 15 '15 at 15:33
  • No mutex does not extend beyond the physical machine. That makes it more difficult you may want to update the question to reflect this. Does the critical process (the one which can only have one instance) need to run on multiple physical machines? Bypassing the problem is going to be a lot easier than solving it. Multiple machine synchronization especially for an arbitrary number of machines is non-trivial. Even if you have multiple webservers does the backend process need to run on multiple servers? Do you need full redundancy? Can one machine be designated a master node? – Gerald Davis May 15 '15 at 15:45
  • The server does reference that: "have a "primary" and "secondary" server lock the record, track who's in charge, etc". :). I think I can use "LowFlyingPelican"'s solution, where I delegate out the jobs to a system that already supports multi-server distribution. – Jerod Venema May 15 '15 at 18:48