How do I unit test nuanced mathematical procedures?

Question

I'm writing some code for a scientific simulation and I'd really like to use this as an opportunity to get some unit testing practice under my belt. I understand the idea of tests in theory, but I just can't quite see how to apply it to my problem to prove a mathematical result comes out correctly.

For example, a big portion of my algorithm involves doing a numerical convolution to shift and alter a signal; as I only have discrete samples of such a signal, I need to resort to an interpolation procedure before doing any piece of the convolution. Convincing myself the interpolation works is easy enough graphically (plot a known function analytically and plot the interpolation on top of it), but very very difficult numerically (it already depends on the sample rate and interpolation order---both of which can give significant variation in the error of the interpolation).

Of course I can just say "for this signal, sample rate, and interpolation order things agree to a precision of epsilon" but that seems like such a small portion of the test space as to be hardly worth doing. What are some techniques I can use to convince myself that things like interpolation (or matrix inversions, or numerical integrations, or...) work a little bit more generally?

Answer 1

Tests are always just examples – they can never prove correctness. But even if the test space is vast we can extract a lot of value a few from well-chosen examples. Many inputs are basically equivalent to each other, with just a small change of one value between each other. We can group all similar inputs into an equivalence class and then test only one instance of that equivalence class.

The instance from the equivalence class can be chosen more or less randomly, but edge cases tend to be the most important, because there are likely to be bugs in edge cases. It therefore makes sense to test around the minimum and maximum value for each parameter.

There are various test coverage criteria that tell us how effective our tests are. One such criterion is statement coverage: What percentage of statements was executed by the test suite? If statements are missing, we can construct test cases to exercise a particular code path. Depending on your programming language, there should already be tools to calculate test coverage.

By creating such tests, we build confidence that the program is doing what we think it's doing. By creating an automated test suite, we can ensure at the click of a button that the behaviour didn't change, for example after making an unrelated change. In particular, this confidence allows us to refactor the code later in the project.

Moving forward, it is a good idea to express requirements as test cases. Whenever the program is lacking some functionality or when the program's behaviour differs from our expectations, we should write down a test that asserts our expectations. Then we can write the code to make the test pass. This is an easy technique to grow the test suite over time, and prevents regressions: that a change to the system re-introduces a problem you've already fixed before.

Not all tests have to run a complete scenario. If there are single functions within the project that you consider high-risk (i.e. they aren't immediately obvious), then you should test them directly. This requires that the function is isolated from its environment (no global variables, no calling functions where you aren't sure whether they are correct). For mathematical functions like matrix operations, this is probably already the case so they are fairly easy to test. Again: construct equivalence classes, and test with interesting instances from these equivalence classes.

Answer 2

The way i go about testing such complicated pieces is this one:

• list all edge cases the function/method has to cover (that should have been done already)
• create these test cases manually (e.g. in your case: use random input, run the function over it and review the result manually.)
  • one test case with no edge case
  • as many test cases as there are edge cases where every case covers exactly one edge case
  • test cases with common and/or important combinations of edge cases (since, with many edge cases, one ends up with a huge mess of test cases if one wanted to cover all possible edge case combinations)