Gee this sounds familiar.
I was once the manager for a group that had a large application which had a few unhandled exceptions that were eventually caught be a global catch-all-and-display-the-world-has-ended handler. By default this used to allow the application to keep running.
Your point about application state in this case is the same one I made: the application should not continue running because its internal state is unknown, and most likely suspect (after all that's how an exception came to be raised in the first place).
I wanted the application modified so that the user could save data and then the application would exit - no opportunity to continue. This was met by fierce resistance, mainly from developers who argued that it had continued for a long time and there had been minimal damage - just lots of complaints. In the end I had to back down - the user was allowed to continue but dialogs were changed to strongly suggest to users that they should exit and restart. I still think this was wrong.
In then end, every exception that is seen needs to be analysed to find the underlying cause, and appropriate fixes, code changes, or whatever need to be applied so that exceptions don't happen - or if they do they are caught as part of normal program flow and locally handled.
(Sometimes, sadly, exceptions are propagated out of things like libraries. One of the rules I was taught a long, long time ago was: "do not ever rely on exceptions for normal program flow". It seems this is not always regularly practiced any more.)
I feel your pain - but in the end, the principle is a simple one and you should be sticking to it: When the last chance exception handler is fired, things are really sick. The program should exit. Doing otherwise gives users the misleading impression that they can continue working, but internally the program has an unknown or broken state. Continuing to work only leads to things getting progressively worse. In the end this translates into loss of customer satisfaction. Customers are also pissed off by a program that raises an exception and exits - if there is unsaved program state you should try and save it, perhaps in a manner that makes it clear that what is saved may not be reliable. At least you reduce the chance of data loss. But continuing to operate... bad bad move. You have to choose the lesser of 2 evils.