Opinions on logging in multiprocess applications

Question

We have written an application that spawns at least 9 parallel processes. All processes generate a lot of logging information.

Currently we are using Python’s QueueHandler to consolidate all logs into one file. Unfortunately, this sometimes results in very messy files which are hard to read (making it difficult to track exactly what’s going on in one thread).

Do you think it is a viable option to separate all messages into dedicated files? Or will this make things even messier, due to the high number of files?

What are your general experiences when writing log files for multiprocessed/multithreaded applications?

grep is a livesaver with log files – ratchet freak Dec 08 '12 at 00:26 — ratchet freak, Dec 08 '12 at 00:26

score 3 · Accepted Answer · answered Dec 08 '12 at 00:47

3

Tag each log line with the process id as an identifier, then use grep when looking at the logs. Or something else besides the process id.

answered Dec 08 '12 at 00:47

David Paigen

256
1
2

score 1 · Answer 2 · answered Dec 08 '12 at 00:45

I prefer letting each process log to it's own file and later merge them together if needed/wanted. This way I don't have to concurrency/contention on one single log file. Being separate processes the log events are usually pretty disjoint anyhow.

Performance wise this is the most efficient way to handle it. The joining of log files is typically wasteful since no one reads most of it anyhow, I would stick to doing so on demand.

If you are mostly using your log files as debug instrumentation instead of auditing purposes and it's helping you to get your code to work or debug multi-threading issues the joining of log files may result in loss of event ordering unless you can use an acceptable high enough resolution timer to include in your log events. However your safest bet would be to log to a log service of some sort that synchronizes for you.

Thanks for this opinion. We do indeed do audits on these logs. A specialiced service souls also add a little bit more security maybe. — chkorn, Dec 08 '12 at 08:35

score 1 · Answer 3 · answered Dec 08 '12 at 11:41

It really depends on your application, and how you analyse the logs. Consider the high end server extreme case where you have many threads all interacting. Consoldating this at log time into a single queue, and writing into a file in time order allows you to determine fairly precisely the order events occured in is one solution, but a better solution would be to have multiple logs, have each log event record the tick time, and then use splunk to perform the analysis.

Opinions on logging in multiprocess applications

3 Answers3