Considerations for delegating long running to CLI from PHP/Apache

Question

I'm working on a web project that executes CLI commands to perform running tasks to spare users from having to wait since PHP/Apache combo cannot (to my understanding) create a separate process for said long-running processes.

• Is the this the best approach for running long for running long processes that do not force web users to wait?

• Is there any time-bombs one needs to look out for when taking this approach (running many CLI processes from PHP/Apache?

Answer 1

There are multiple ways to solve something like this:

• If you're hosting on AWS for example, you might want to look into their SQS and Lambda services. I'm not really familiar with other cloud providers but they probably have similar services of their own
• A more DIY approach to the queuing-technique is to use Apache Kafka for example (or other similar tools)
• If your application is written using the Symfony framework, there is a technique in there where you can keep executing stuff AFTER you return the response to the user. In short: it's done by listening to its kernel.terminate event and doing stuff when that event gets triggered
• If your application is written the Laravel framework: they have quite a nice queueing service which stores the queue in the database/filesystem/redis/whatever and then can be executed from the command-line, therefore also by cronjobs
• probably 10 other approaches that don't come to mind right now.

Maybe add a few more details about your setup and what you need, that will make it easier to narrow down the best solutions.

Regarding your 2nd bullet point, the things to look out for the most:

• have a really good way of monitoring those tasks. Since they're in the background, they tend to be more invisible, so you need to break that disadvantage as much as possible. Plan this into your solution from the start
• have some way of notifying the users when the task is done, if it makes sense. For example: the user uploads an avatar and your background-task needs to resize that image before it's displayed in his profile. Even if notifications don't make sense RIGHT NOW for your use-cases, plan to support them because they might make sense in the future and it will make things a lot easier
• Decide what to do when a task fails. Do you tell the user? Do you tell the DevOps guys? Do you retry it? If yes, how many times before you give up? Do other tasks depend on its output? Does it leave the database-data in an inconsistent state?
• If those tasks take a lot of processing power, how do your protect against someone (non)intentionally flooding the system with too many of them?

Some of those things above make sense for your situation, some don't. There are probably many more. Shouldn't be too hard to come up with a more robust list based on your requirements.

Answer 2

Assuming a Linux operating system (on the web server), you might consider installing the batch and/or at utilities and use them from your web application (BTW, you could consider better tools than PHP for it, e.g. ocsigen). You definitely should master Linux system programming techniques since you are using several processes simultaneously (the web server and your CLI running command).

You could be interested in FastCGI (or use HTTP proxy techniques on your web server, with perhaps some HTTP server library like libonion used in your own server program with an event loop monitoring long-lasting tasks) ....

The important issue is how to notify your web user of the completion of the long lasting Linux process started on the web server. Maybe inter-process communication techniques (see poll(2), pipe(7), unix(7), ....) are relevant (between long lasting programs and web service code). Read also ALP. Of course you want to protect your web service against DoS attacks and you need to care about job control (so be at least aware of setrlimit(2), time(7), signal(7), fork(2), execve(2), wait4(2), dup2(2) etc...).

You could be interested in inotify(7) facilities. The long lasting job could run in some batch script ending with e.g. an echo done > somefile to create a file, and you would use inotify facilities for synchronization purposes in your web server.

Assuming recent graphical browsers used for your web application, you might consider using Websockets and AJAX techniques: on the web server side, you would use polling techniques or some event loop, and when the job is completed you'll asynchronously send on the websocket some JSON message, with some AJAX javascript code in the browser to handle it and display something nice when that happens (so passive polling in the browser too...)

PS. See also this question. Several (but not all, and not most) points there are relevant for your needs.