Why should 'boneheaded' exceptions not be caught, especially in server code?

Question

I am confused because in quite a few places I've already read that the so-called 'boneheaded' exceptions (ones that result from bugs in code) are not supposed to be caught. Instead, they must be allowed to crash the application:

• Vexing exceptions, by Eric Lippert
• A comment under Eliding Async and Await, by Stephen Cleary
• Answer below Is it a good practice to use self-defined exception?, by Draco18s no longer trusts SE

At least two of the three above people are established authorities.

I am surprised. Especially for some (important!) use cases, like server side code, I simply can't see why is catching such an exception suboptimal and why the application must be allowed to crash.

As far as I'm aware, the typical solution in such a case is to catch the exception, return HTTP 500 to the client, have an automatic system that sends an emergency e-mail to the development team so that they can fix the problem ASAP - but do not crash the application (one request must fail, there's nothing we can do here, but why take the whole service down and make everyone else unable to use our website? Downtime is costly!). Am I incorrect?

Why am I asking - I'm perpetually trying to finish a hobby project, which is a browser based game in .net core. As far as I'm aware, in many cases the framework does for me out of the box the precise thing Eric Lippert and Stephen Cleary are recommending against! - that is, if handling a request throws, the framework automatically catches the exception and prevents the server from crashing. In a few places, however, the framework does not do this. In such places, I am wrapping my own code with try {...} catch {...} to catch all possible 'boneheaded' exceptions.

One of such places, AFAIK, is background tasks. For example, I am now implementing a background ban clearing service that is supposed to clear all expired temporary bans every few minutes. Here, I'm even using a few layers of all-catching try blocks:

try // prevent server from crashing if boneheaded exception occurs here
{
    var expiredBans = GetExpiredBans();
    foreach(var ban in expiredBans)
    {
        try // If removing one ban fails, eg because of a boneheaded problem, 
        {   // still try to remove other bans
            RemoveBan(ban);
        }
        catch
        {

        }
    }
}
catch
{

}

_{(Yes, my catch blocks are empty right now - I am aware that ignoring these exceptions is unacceptable, adding some logging is perpetually on my TODO list)}

Having read the articles I linked to above, I can no longer continue doing this without some serious doubt... Am I not shooting myself in the foot? Why / Why not?

If and why should boneheaded exceptions never be caught?

Answer 1

Silent But Deadly

When writing enterprise software, you will eventually learn an essential truth: the worst bug in the world is not one that causes your program to crash. The worst bug in the world is one which causes your program to silently produce a wrong answer that goes unnoticed but eventually produces a massive negative effect (with severe financial implications for your employer). Thus, error messages and crashes are A Good Thing^TM, because they indicate that your program detected a problem.

Amazing Grace

Now, this seems to conflict with another enterprise virtue, which is "degrade gracefully". Blowing up and not returning any response at all hardly looks like "graceful degradation". And this is why many folks will try very hard to return some response, if they can. Indeed, this is why many frameworks, like Spring, will catch all top-level exceptions and wrap them with a 500 response, as you describe. In general, I think this is OK. After all, most exceptions don't really require a restart of the entire app server if you can just kill/restart a server thread. A sane framework will be careful to not catch Java Errors, like OutOfMemory, for obvious reasons.

But there is one more point to consider: once you get beyond a single server, you will likely have a load balancer in front of your service. And when the LB times out or gets a closed connection, it will generally return a 500 to its client. Thus, the LB will often transform your "server crash" into a client 5xx automatically! Best of both worlds.

Worst Case

In your scenario, what is the worst that can happen if you don't catch the exceptions? Your answer: "Well, my game server dies, and nobody can play!!!" But that's not the worst case. The worst case is, everyone is playing your game, but griefers are ruining it. Players file a bug report and tell you that bans aren't working, but you look at the logs and everything looks fine. Or, legitimate players are getting banned by griefers, and instead of being able to rejoin in a timely manner, the bans are lasting indefinitely, because your server happily ignores failures. The worst thing isn't your game crashing. It's your player trust crashing. Good luck trying to reset that.

Answer 2

Exceptions should be allowed to crash the system if the system has been left in an unrecoverable undefined state. If you can't put the system back in a defined state that ensures data integrity and security then you crash so the system can be rebooted into that defined state.

Whenever you catch an exception you're taking responsibility for doing all that recovery yourself. You should only leave your catch empty when you are absolutely sure the system is still in that defined state and nothing needs to be done. This happens when the calling code has a better idea of the system state than the code that threw the exception.

That's all there is to it. Calling an exception boneheaded is just being pointlessly rude. In some contexts you simply don't know what will happen to system state when you fail, so you throw and let the calling code figure it out. If the calling code can't figure it out then you crash and let the operator figure it out.

Now if you somehow know that failing won't put the system into a bad state then you don't need to throw in the first place. Throwing was never meant to be the only way to fail to do what you were asked to do. You can return -1, or the empty string, or null (ick), or a null object, or an empty collection, or a "maybe" monad, or print "Your princess is in another castle", or just do nothing, quietly. Yeah, I know. But sometimes that's ok. Why? Because at times failing to do what you're told is correct, expected, and not interesting.

Answer 3

It doesn't really matter if it's a "boneheaded" exception (e.g. a Java unchecked exception) or not.

The only question to ask yourself is:

"Can the program sensibly continue?"

A "server" is a program that processes messages. Typically, each message is mostly independent from the others. That is: if there is an error processing on message, it makes sense to continue receiving and processing future messages.

This would apply to an HTTP server, a job server, etc.

Some errors may not be sensibly recoverable, like OutOfMemory. But if you can recover from an error (including possibly logging it some way), then you should.

Answer 4

You are absolutely correct. An exception in server-side application code ("boneheaded" or not) should not crash a web server.

The confusion is because the articles are not clear about what it actually means to "catch" or "crash". If we followed advice never to catch "boneheaded exceptions", then a single application bug should bubble up and cause the whole operating system to crash. This is not what anybody wants!

Rather you should think of a system as multiple layers of isolation. An unexpected crash in one layer should not cause outer layers to crash. An operating system will isolate crashes in individual processes and will not let a crash in one process take down other processes or the whole OS. A web server will typically isolate crashes in single requests and not let them affect any other request.

An application may itself have multiple layers. E.g. an application supporting plug-ins might isolate crashes in individual plug-ins. If a plug-in crashes it is disabled, but the rest of the application continues.

But isolating crashes can only work if the units of isolation are independent. Web requests are independent from the perspective of the web server. A crash in one request will not cause any other process to fail or receive invalid input.

So the bottom line is that you shouldn't catch boneheaded exception in your own application code, you should let a higher level (the framework) catch them.

Catching a boneheaded exception in your own code does not really make any sense. A boneheaded exceptions means there is an unexpected bug in the section of code. In other words, the code does something wrong, but you don't know exactly what or why. At best nothing happens when you execute the code, at worst some state is corrupted which might cause other parts of the system to behave erroneously and corrupt more state.

So if the code detects that there is a bug in the RemoveBan method... why do you want to execute it again? I can't imagine any scenario where you would want to repeatedly execute some code which you know does something wrong.

Answer 5

You learned an important thing: Whenever you read a rule on the internet that must absolutely be followed, you must start thinking about it and decide for yourself whether you should follow the rule in your specific case or not. And you should also think about whether the rule as you understand it is a good rule or not.

Here some rules that you can think about: A server should never crash unless absolutely necessary. A client on the other hand is absolutely allowed to crash if you make sure there is no substantial data loss, users will often not even take notice.

So for a client, crashing is not at all a bad response to a boneheaded exception. For a server, it is a much worse response. Still, you have to weigh up: You have an exception that is totally unexpected to you. You have no idea how to handle it properly (reporting an exception isn't handling it). How much damage could be done by continuing when you know something has gone badly wrong? So do you want to continue operations, with unknown and potentially unlimited damages, do you want to crash the server, causing some damage, or do you want to restart the server (not quite crashing, but restarting it as gentle as possible).

Answer 6

I think that what you are failing to appreciate is that that the real-life consequences of errors can be much worse than simply having the server go down. Just for instance:

• Erasing a database that is essential to the functioning of a company
• Granting access to confidential information that shouldn't be granted
• Approving a financial transaction that shouldn't be approved
• Disapproving a financial transaction that should be approved
• Destroying physical equipment under computer control
• Killing a patient receiving radiation therapy under computer control
• Causing flight control problems that result in the crash of two aircraft, killing 346 people

Ok, the list is a bit of a scarecrow. Ignoring exceptions is just one many possible errors that could have terrible consequences like these. But it is important to keep in mind that an exception being raised is telling you that one or more of your assumptions about the operation of the program is not valid, and you really need to think about the possible consequences of that.

Since you are writing a hobby game where none of these outcomes is remotely possible, you could just blow a raspberry and ignore this issue. However, if part of the reason you are writing this hobby project is to educate yourself, and to prepare yourself for writing software that other people will use, then you have to think carefully about this stuff, and practice writing code as if its correct functioning mattered. Yes, this is a lot more work.

No, you don't have to write handlers and recovery code for every conceivable exception. You can just deal with the ones that you think are important and know how to handle. But unless you are absolutely sure that an exception has no impact on the continuing correct functioning of the program, then it has to get passed on to the next higher level of control. That gives it the chance to recover or to abort the program and prevent incorrect functioning.

Answer 7

I am surprised. Especially for some (important!) use cases, like server-side code, I simply can't see why is catching such an exception suboptimal and why the application must be allowed to crash.

There's nothing wrong with a crash, in fact, it can be very helpful to have an application crash early. @PaulDrappers answer of 'Can the program sensibly continue?' is really the key way of looking at it.

For example, if my application has been incorrectly configured - then I want it crashing immediately on startup. By quickly and obviously failing, this allows developers/DevOps to quickly identify and rectify the problem. The alternative, returning 500 errors, is less obvious and if intermittent may even pass health checks until a customer complaints.

On the other hand, if I get a sporadic error on a single request, I will typically 'crash' that single request (e.g. return 500), but otherwise, let the application continue running. After all, a single DB timeout shouldn't typically crash a website.

Though this is dependant on the application - if it's processing a single task at a time in a fault-tolerant pipeline (e.g. some kind of job or message processor), letting the application die might be appropriate (e.g. an orchestrator might have a failure policy we want to follow).

Whatever the situation I will try to log every error regardless if it's handled or allowed to bubble up.

Answer 8

Your example demonstrates exactly why you shouldn't catch 'all possible' exceptions.

If your GetExpiredBans call fails, your code simply carries on as if it had been sucessful. The unbanner server is up and running and looks good, but actually it's not working at all.

Now if you know that RemoveBan occasionally fails due to a network problem for example, then you could catch that specific exception and implement a retry, or skip that player and move to the next. It's an understood problem and you know the desired behaviour.

If your code throws an exception you werent expecting then it's best to stop and have a human look at it.

Now you bring up the case of a HTTP server returning 500s instead of crashing.

Here the server is running an external program, ie your webpage. It's not the webserver which has a bug, its someone elses code. You wouldnt expect DOS to crash if you ran a buggy program, returning an error code and moving on is the same as that page crashing.

Answer 9

Personally, I handle all exceptions.

And email them to myself.

It doesn't have to be email, you could just log them, but if you don't know what these errors are happening, how can you ever hope to correct them?

By all means, try to recover, or gloss over the error - to the user, but not to yourself.

For AJAX requests, this might mean sending a 500 response. For GUIs, the classic MicroSoft "something bad happened" (with its implied, and "we coders know what it is, but we are not going to tell you").

IMO, you need to inform your "user", whether human or software, but you also need to inform yourself, the development/maintenance team. These might be same action, or two different actions, but they both need to be informed.

Answer 10

There are actually two different types of exceptions that need to be handled differently.

First of all there are the "Non-bonehead" exceptions like "File not found". These exceptions are kind of expected, even if you check for the files existence beforehand it doesn't prove that it will be there when you go to use it.

These you typically catch as low/specific as possible, everyone understands this already because it's how everyone says to handle "Exceptions", so let's ignore it.

The other is unexpected exceptions. This covers both programming errors and some situational errors you aren't expecting to encounter (like the OS rips a disk out from under you). These are what you are calling "Bonehead" exceptions but they DO happen.

The important thing is that these are NOT ignored. If the only way to get a team to pay attention is to crash the app hard, then DO SO, but if you can get their attention another way, I recommend catching the most general "Exception" type just inside each primary thread loop and dealing with it in such a way as to get the teams attention, and then try to continue. You'd be surprised at how often this can keep your app running pretty much perfectly while you fix it. It's even allowed me to recover from an out of memory situation.

Not catching it is really not good compared to just handling it in a way that gets attention.

Also VERY important:

In Java (at least) by default when a thread throws an exception it is silently eaten, it doesn't crash your app or give you any indication that part of your program failed, but if that was a long-running thread, everything it powered is completely (invisibly) gone!

This can be fixed by installing a default exception handler, but be careful because without the handler or a try/catch, allowing such an exception to just silently kill a thread is the worst possible solution--it's possibly the most expensive thing you can do to another developer (or yourself!) I've spent weeks tracking down exceptions that were eaten by threads and empty catches!

Answer 11

There are two broad categories of exceptions: those that arise from the programmer's misuse of an API (e.g., invalid arguments) and those that arise from external causes (e.g., file not found). Exceptions from external causes should always be caught; exceptions from misuse of APIs should never be caught. This is because it's realistically achievable to avoid making these kinds of mistakes in the first place, and because these kinds of bugs usually manifest right away and should be identified and fixed, not handled in the code.

Distinguishing these categories of exceptions is much easier when you have language support (i.e., checked vs. unchecked exceptions) or when working with APIs that clearly distinguish them (i.e., different exception base classes.) Avoiding API misuse is also much easier in languages and IDEs that provide instant access to the complete documentation (e.g., Java/Javadoc in IntelliJ IDEA.) It's considerably more difficult in languages and IDEs that only provide instant access to a severely limited summary of the documentation (e.g., C#/Visual Studio.) In my experience, good IDE support for instant access to complete documentation vastly reduces the number of API misuse bugs that are ever introduced in the first place.

TL;DR:

1. Constantly refer to the documentation to avoid misusing an API. Corollary: avoid using APIs that are inadequately documented.

2. Do not catch exceptions arising from misuse of APIs. Corollary: avoid using APIs that do not clearly distinguish between misuse and external errors.

Answer 12

An exception you don’t know how to handle at a particular place should not be handled there. It should be passed up the call stack until it can be handled properly. If it is not expected anywhere, there is at least some place where an operation can be canceled without causing a “wrong answer” or a crash.

Answer 13

Exceptions (Boneheaded or not) are Exceptional

If something is exceptional, by definition you have not considered it, nor know how to handle it.

Think of it like this. The sales assistant on the checkout of a store knows how to sell stuff, perform exchanges, etc..

They do not know how to handle Reporter Interview. The correct approach is to throw an exception to their Manager. Their manager may know how to handle this, and if so will then catch the exception and deal with the Reporter Interview. Otherwise what should the manager do?

Pass the exception up to their Manager. Eventually (if not dealt with earlier) this will reach the CEO/Business owner/Directors and have to be dealt with there. Even they may not know. In which case the exception escapes, and the Reporter Interview is not dealt with.

The worst scenario to occur is for the Reporter Interview to be dealt with by one of those people who does not have a clue how to handle it. Imagine the Reporter reporting on an poorly worded comment from the Sales Assistant or the Local Manager? Massive damage to the business reputationally, and quite possibly financially.

Business Software, be it in an office, or an online game is essentially just automating the process that would have otherwise had to be done manually by an actual worker. Ergo you want it to behave like a well mannered employee. It should only deal with the situations that it has a good well-defined response/activity for. Everything else should be handed over to its managers (usually its support team) for clarification (new code) and/or action (manual tasks).

Answer 14

To add an extra bit of flavor:

While you should allow your program to fail, it's not necessary that you make it fail catastrophically.

If your language allows you create your own exceptions, a great approach is to do something like this (Python):

# Exception is the base "user" exception class
class MyAppException(Exception): pass
class MySpecificError(MyAppException): pass
class MyOtherError(MyAppException): pass

...

if __name__ == '__main__':
    try:
        do_it()  # or whatever starts your app
    except MyAppException as e:
        # This catches any exceptions that *you* raised, and
        # is entirely the programmers fault for not catching before here!
        print('Well, we forgot to catch *this* exception:', e)
    except Exception as e:
        # This catches anything exceptions that may or may or may not
        # be within the programmers control, but include things like
        # KeyErrors or IndexErrors
        print('This was unexpected:', e)
    except:
        # Maybe not this one - it catches SystemExit & others. But, you *could*.
        print('Well, this was very unexpected')
    finally:
        # Do some cleanup stuff - close sockets, or whatever, and then quit
        exit(1)  # or some other failure code - it's up to you.

You can also re-raise your exceptions in Python if you'd like to really crash crash.

Taking this approach, you know:

1. If you missed any exceptions that the developers should have known about
2. If you missed any exceptions that you probably should have known about, but just didn't think of.
3. If there was just something else entirely that failed - solar flares or something that was really outside of your control.

But you don't bother recovering and trying to keep running - as everyone else said, if you can't recover then you really shouldn't. But you don't have to die ungracefully - it's OK to give a last gasp, "Well, that was unexpected!" before keeling over. And in my experience, that's even a good idea because then you can try to fix it or account for the problem. Or just decide that it was something that really wasn't recoverable and you're just going to alert on that or something.

Answer 15

You have to consider the reason/purpose for handling an exception, such as:

• To perform some sort of corrective or recovery action so that the application can meaningfully continue; this implies that the exception and its consequences are well understood
• To perform some sort of cleanup so the application is restored to a consistent state; again, the consequences of the exception must be fully understood
• To capture information about the exception for logging and later analysis
• To fail gracefully, such as to issue some sort of user-friendly/user-meaningful message in an appropriate UI instead of whatever raw message or code is directly reported by the exception; in doing this, it's generally wise to capture the more technical details for analysis by the development team.

If there is no useful purpose served by your exception handler, then a handler shouldn't be there and the exception should be allowed to "bubble up".

"Bonehead" exceptions implies conditions which were not anticipated when the application was initially coded. Un-anticipated exceptions should have as much detail captured as feasible so that the cause can be determined and appropriate action taken. This may be done automatically by the programming framework; if so, then you may not want to explicitly handle them.

Answer 16

I know I am late to the party, but I think there is some nuance that is missing in the existing answers. To quote sage advice from any number of martial arts movies:

The best way to block a punch is to not be there

To bring it into software engineering, we can repurpose the quote this way:

The best way to handle exceptions is to never throw them in the first place

Even though it was never explicitly called out in the first link you provided, that was essentially what the writer was trying to steer the readers toward.

Reasons why exceptions are considered bad:

• They are very heavy to recover from (stack information takes time to unwind)
• They were never designed to be used for normal program flow
• They were designed to allow you to do your best effort to recover
  • That recovery may only be to close open resources and release locks before shutting down

That said, exceptions should not be propagated to users, particularly in a web context:

• It exposes the internal architecture of your application, making it vulnerable to future attacks
• For the web, instead of more appropriate responses like a 400 Bad Request or 404 Not Found, you return a 500 undefined server error

So what do we do?

• Minimize the use of exceptions whenever possible (i.e. a TryParse that returns boolean instead of Parse which throws an exception)
• Be purposeful about our exceptions
• Never propagate exceptions to the response, but prefer logging to debug with
• Fix the bugs that cause exceptions to be thrown
• Verify your environment in code during startup if you have external dependencies--it's better to fail on start up then figure out at runtime that the service or application couldn't have worked anyway.

Answer 17

This cannot be more straightforward.

There are only 3 types of code:

1. Code that never throws an exception.

2. Code that you know will generate exceptions. For example, connecting to an external RPC server, it is expected that sometimes it will timeout. You should add try...catch for this type of code.

3. Code that you didn't expect, but does throw errors. Since you didn't expect it to throw errors, there's nowhere to add try...catch, because you cannot add an infinite number of try...catch, that's simply not possible. Once an error happens, you just debug and fix it, that's all.

How hard is that?