How should I handle logger failures?

Question

In several of our company's applications, we use a custom logger. It's fairly robust, though we may replace it with something like NLog in the future. One of the logger's tasks is to log any exceptions encountered in the application.

One concern I've always had is that the exception handling within the logger allows for a silent failure. That is, if the log isn't written for a given exception (due to an error in the logger), how should I handle it and (somehow) log the exception in the logger itself?

Let's say the WriteLog function throws an exception. Should I try to call the function some number of times or until the exception isn't thrown? Should I try to write the thrown exception with the logger (which would likely just result in exceptions all the way down. . .)? I have been lucky enough to not encounter this situation except when we were first implementing the custom logger. On the other hand, I have no way of knowing at the moment if the logger has failed to log application exceptions (due to its own exceptions).

I have tried searching online and on some SE sites, but it's been fruitless so far since all the posts deal with errors in a logger (but not potential exceptions and how to log them) or with exceptions outside the logger.

Answer 1

When you encounter exceptions within the logger itself, you shouldn't use the logger to log its own exceptions. The reason for that is that:

• You may be stuck in an infinite loop. Imagine that within your logger, you have a conditional branch which wasn't tested (and generates an exception). Imagine that once the condition is met, any further reported exception is handled by the same branch. This means that from the moment the branch is executed, you're in an infinite loop.

• You may be stuck in a temporary loop, generating thousands of exceptions per second. Imagine you're reporting exceptions to a remote server. An issue with the server causes another exception, which causes another one, and so on, until the connection is back.

What you should do instead is to fallback to a safer way to log the exceptions. For example, if your logger sends the exceptions to a remote server, send the exceptions within the logger to syslog instead. If your logger records exceptions in Windows Events and this action fails, store the failure exception in a simple text file.

Once you have that, the next question is how do you know that those exceptions occurred: if you have dozens of applications running on thousands of servers, you can't possibly SSH each of them on regular basis to check whether they were logging something locally.

One way is to have a cron job which checks for those “exceptional logs” and pushes them to the location where other exceptions are stored (eventually using your logger, but beware of infinite or temporary loops!).

Answer 2

If logging is critical to your application, then one should stop the application if logging fails.

If not critical, then being somewhat defensive one could have a secondary component to handle logging failures that logs/alerts to a secondary source. But even that is not fool proof and you will have to consider what happens if the secondary logger fails while it is monitoring the primary logger.

A good strategy is logging to a local file and if that fails, maybe logging that failure to the event log, generating an email alert, saving to a database, etc. With the logging frameworks that are available this should be foolproof unless the machine runs out of disk space or some other rare condition.

Ideally your better off failing silently as that will make the application less complex.

More importantly, to handle logging failures one should be monitoring the logs from a 3rd party. Over time you should be able to discern how many events a healthly application is logging. If it starts logging low or no events, then through monitoring you can see the problem occurring and potentially alert through that 3rd party mechanism.