Organize and maintain a lot of cron jobs

Question

I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc.

We have started to move the hourly timings by a few minutes so that they don't overlap, but I don't know if this is a good idea.

Besides, I have noticed that it is becoming increasingly difficult to maintain such a file

Do you have any advice, ideas on how to improve it, change it ?

I was thinking of combining cron with a queuing system (RabbitMQ) so that cron would call tasks at a certain time but they would go to RabbitMQ. This would prevent tasks from overlapping.

I wonder if this makes sense.

Below is a sample piece of cron:

1   4   *   *   *   yii task1

1   5   *   *   *   yii task2

1   8,17  *   *   *   yii task3
10  9,17  *   *   *   yii task4
30  9,17  *   *   *   yii task5

*/4  *  *   *   *   yii task6

*/15 *  *   *   *   yii task7
5 10,14,17  *   *   *   yii task8
50 *  *   *   *   yii task9
5    *  *   *   *   yii task10
10   *  *   *   *   yii task11
15   *  *   *   *   yii task12
10   0  *   *   *   yii task13
10   3  *   *   *   yii task14
30   4  *   *   *   yii task15
30   0  *   *   *   yii task16
40   5  *   *   *   yii task17
40   7  *   *   1-5   yii task18
30   22 *   *   *   yii task19

59   23 *   *   *   yii task20
7    6-16,23    *    *    1-5    yii task21

5    3  *   *   *   yii task22

*/10 6-22 * * *    yii task23

*/15 6-22 * * *    yii task24
*/5 6-22 * * *    yii task25

0    6-22    *    *    1-5    yii task26
20   5    *    *    1-5    yii task27

Answer 1

I don't use the particular technologies mentioned, but a complicated system of scheduling can easily get out of hand.

Almost 30 distinct schedules (and growing) strikes me as far too much even for developers to reason about, let alone for others in a business to know when things will actually happen during the working day.

I'd suggest simplifying the schedule down to basics - like daily, semi-daily, and hourly - and then attach the necessary tasks to one of these.

It's unusual for a properly designed system to require hourly reporting - that kind of frequency often suggests that either the report should generate on demand by the user (if results always have to be reasonably fresh), that users should be counselled to relax into the arms of a slower schedule (like daily), or that there are deeper systemic problems which are causing users to poll the system unnecessarily.

Answer 2

I would suggest combining jobs that should be run on the same schedule (hourly, twice daily etc.) but should not run in parallel into shell scripts which execute them in sequence, so if one job runs a little slower they still don't overlap, and if one runs a little faster you can proceed to the next one without waiting on a specific minute. This would significantly reduce the number crontab entries to those with different schedules.

If your dependencies are more complex, you might consider a full blown job scheduling system, but it looks like that would be overkill in your case.

Since your task names are completely anonymous, it's hard to detect whether there is some natural sequence or everything is basically independent of everything else.

A complete change of architecture from scheduled job executions to continuous event flow processing using a message broker would be a pretty big change, but it might make sense when almost-immediate data updates are required.

Answer 3

Build system that recomputes crontab.

Recomputation can be as simple as copy paste Cron definition, or some staggered schedule for related jobs.

On the other end work on a great wrapper for all your jobs, so that you have good statistics for all those jobs. It could for example note start/stop/next expected run. Or report failed jobs.