How can I debug exceptions that are not easily reproducible and only occur in a production environment?

Question

I am working on an issue where the exception only occurs in our production environment. I don't have access to these environments, nor do I know what this exception means. Looking at the error description, I'm unable to understand the cause.

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Would someone please advise me on how to approach this kind of problem?

should this be moved over to StackOverflow? I think you'd get more response there. — DXM, Jan 25 '12 at 07:36
@DXM - it would be off topic for Stack Overflow, as it's too general. The OP is after strategies and techniques rather than a specific solution. If the code that was failing was included then perhaps it might work on Stack Overflow. — ChrisF, Jan 25 '12 at 13:42
In my experience, most problems like this arise from security configuration issues and can be difficult to figure out. As others have mentioned, good logging will help reveal it. — jfrankcarr, Jan 25 '12 at 15:15

score 18 · Accepted Answer · answered Jan 25 '12 at 07:00

18

In general, better debug logging. Figure out what you want to know, add it to the code, and have that in the logs so that you can work it out. Capturing more details of the environment at the time also help - what request, when, etc.

In specific, I would look for a common pattern in clients hitting this - and if you found one optimise - but then go and capture the TCP layer traffic.

Looking at the SSL messages exchanged should give you some idea what is going wrong in the protocol, or at least what the common properties of the request are. Once you have that it should be closer to being debugged.

As a guide, I would guess this comes from one of three things:

Something that isn't SSL talked to the SSL port. (port scans are common, but HTTP to the HTTPS port also happens.)
The client doesn't share an acceptable set of ciphers with the server.
The client offers a certificate, and the server has a hissy-fit. (Uncommon, but possible.)

answered Jan 25 '12 at 07:00

Daniel Pittman

3,660
21
19

1

maybe the server offers a self-signed cert or signed by a CA that the client doesn't know / trust – Carlos Campderrós Jan 25 '12 at 12:11
I think I've seen #3 happen when one of the parties has expired certificates. – FrustratedWithFormsDesigner Jan 25 '12 at 14:36
I've done quite a bit of debugging on production systems. Never have I actually used a debugger, it's always either been logging or writing key values to a particular part of the screen. – Loren Pechtel Jan 26 '12 at 05:00
thanks all for your advice.I am sure there are the pragmatic ways to solve a production bug. – C4CodeE4Exe Feb 01 '12 at 04:19

score 4 · Answer 2 · edited Jan 25 '12 at 12:23

I would recommend to use a logging strategy with a configurable maximum log level. A utility like log4j ( http://logging.apache.org/log4j/, http://en.wikipedia.org/wiki/Log4j) might do the job.

Configurable log level (or verbosity) is important to be able to find the reason of an error, possibly without having to re-deploy your software.

If such strategy is not enough to find the error, try to find how to produce/read the logs produced by the applications with which yours is communicating.

You might also implement some mechanism to automatically get more information about errors by e-mail.

More generally, you can read some articles about instrumentation, which is a larger topic that includes logging and tracing.

How can I debug exceptions that are not easily reproducible and only occur in a production environment?

2 Answers2

Linked