10

We know that writing JUnit tests demonstrates one particular path through you code.

One of my associates commented:

Manually writing unit tests is Proof By Example.

He was coming from the background of Haskell which has tools like Quickcheck and the ability to reason about program behaviour with types.

His implication was that there are lots of other combinations of inputs which are untried by this method for which your code is not tested.

My question is: Are manually writing unit tests Proof By Example?

hawkeye
  • 4,819
  • 3
  • 24
  • 35
  • 3
    No, not _writing/using_ tests. _Claiming_ that your unit tests are proof that there's nothing wrong with the program is Proof by Example (an inappropriate generalization). Tests aren't about mathematically proving code correctness - tests are, by their nature, experimental checks. It's a safety net that helps you build confidence by telling you something about the code. But you are the one who has to choose a good strategy to probe the code, and you are the one who has to interpret what that data means. – Filip Milovanović Dec 22 '17 at 11:12

3 Answers3

10

If you are randomly choosing inputs for testing, then I suppose it might be possible that you're exercising a Proof By Example logical fallacy.

But good unit tests never do that. Instead, they deal in ranges and edge cases.

For example, if you were to write unit tests for an absolute value function that accepts an integer as input, you wouldn't need to test every possible value of input to prove that the code works. To get a comprehensive test, you would need only five values: -1, 0, 1, and the max and min values for the input integer.

These five values test every possible range and edge case of the function. You don't need to test every other possible input value (i.e. every number that the integer type can represent) to get a high confidence level that the function works for all input values.

Robert Harvey
  • 198,589
  • 55
  • 464
  • 673
  • 11
    A code tester enters a bar and orders a beer. 5 beers. -1 beers, MAX_VALUE beers, a chicken. a null. – Neil Dec 22 '17 at 08:48
  • 2
    The "5 values" is pure nonsense. Consider a trivial function like `int foo(int x) { return 1234/(x - 100); }`. Also note that (depending on what you're testing) you may need to ensure that invalid ("out of range") input returns correct results (e.g. that ``find_thing( thing )` correctly returns some kind of "not found" status if the thing wasn't found). – Brendan Dec 22 '17 at 17:07
  • 3
    @Brendan: There's nothing significant about it being five values; it just happens to be five values in my example. Your example has a different number of tests because you're testing a different function. I'm not saying that every function requires exactly five tests; you inferred that from reading my answer. – Robert Harvey Dec 23 '17 at 15:57
  • 1
    Generative testing libraries are usually better at testing edge cases than you are. If, for example, you were using floats instead of integers, your library would also check `-Inf`, `Inf`, `NaN`, `1e-100`, `-1e-100`, `-0`, `2e200`... I'd rather not have to do those all manually. – Hovercouch Dec 23 '17 at 17:23
  • @Hovercouch: If you know of a good one, I'd love to hear about it. The best one I've seen was Pex; it was incredibly unstable, though. Remember, we're talking about relatively simple functions here. Things get more difficult when you're dealing with things like real-life business logic. – Robert Harvey Dec 23 '17 at 17:28
  • @RobertHarvey it really depends on the language. Hypothesis (Python) is the gold standard, and I believe Hedgehog (Haskell) and Theft (C) are widely respected, but things get a little more tricky when you move into OO languages. This is probably a good starting point: http://hypothesis.works/articles/quickcheck-in-every-language/ – Hovercouch Dec 23 '17 at 17:40
  • @RobertHarvey If the function is complicated enough, the number of possible cases to consider is very large, and even the process of deducing the test cases itself is prone to error. What do you do then? Pick a few sample cases and just test for those? – Jake1234 Feb 16 '23 at 22:38
  • @Jake1234: Refactor your functions into simpler ones that are easier to test. – Robert Harvey Feb 17 '23 at 14:00
8

Any software testing is like "Proof By Example", not only unit testing using a tool like JUnit. And that is not new wisdom, there is a quote from Dijkstra from 1960, which says essentially the same:

"Testing shows the presence, not the absence of bugs"

(just replace the words "shows" by "proofs"). However, this is also true for tools which generate random test data. The number of possible inputs for a real-world function is typically bigger by orders of magnitudes than the number of test cases one can produce and verify against an expected result within the age of the universe, independently from the method of generating those cases, so even if one uses a generator tool for producing lots of test data, there is no guarantee not to miss the one test case which could have detected a certain bug.

Random tests may sometimes reveal a bug which was overlooked by manually created test cases. But in general, it is more efficient to carefully craft tests to the function to be tested, and make sure one gets a full code and branch coverage with as few test cases as possible. Sometimes it is a feasible strategy to combine manually and random generated tests. Moreover, when using random tests, one has to take care to get the results in a reproducible manner.

So manually created tests are in no way worse than randomly generated tests, often quite the opposite.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • 1
    Any practical test suite using random checking will also have unit tests. (Technically, unit tests are just a degenerate case of random testing.) Your wording suggests that randomized tests are difficult to achieve, or that combining randomized testing and unit tests is difficult. This is typically not the case. In my opinion, one of the biggest benefits of randomized testing is that it strongly encourages writing tests as properties about the code that are intended to always hold. I'd much rather have these properties explicitly stated (and checked!) than have to infer them some point tests. – Derek Elkins left SE Dec 23 '17 at 01:41
  • @DerekElkins: "difficult" is IMHO the wrong term. Random tests need quite some effort, and that is effort which reduces the available time for handcrafting tests (and if you have people just following slogans like the one mentioned in the question, they will probably do no handcrafting at all). Just throwing a lot of random test data on a piece of code is only half the work, one has also to produce the expected results for each of those test inputs. For some scenarios, this can be done automatically. For others, not. – Doc Brown Dec 29 '17 at 08:03
  • While there are definitely times where some thought is needed to pick a good distribution, this is usually not a major hang-up. Your comment suggests you are thinking about this the wrong way. The properties you write for randomized checking are the same properties you would write for model checking or for formal proofs. Indeed, they can be and have been used for all of those things at the same time. There are no "expected results" that you need to produce as well. Instead, you simply state a property that should always hold. Some examples: 1) pushing something onto a stack and... – Derek Elkins left SE Dec 29 '17 at 09:18
  • ...then popping should be the same as doing nothing; 2) any customer with a balance greater than $10,000 should get the high balance interest rate and only then; 3) the sprite's position is always within the screen's bounding box. Some *properties* may well correspond to point tests e.g. "when the balance is $0 give the zero balance warning". The properties are partial specifications with the ideal of getting a total specification. Having difficulty thinking up these properties means you are unclear on what the specification is and often means you'd have difficulty thinking up good unit tests. – Derek Elkins left SE Dec 29 '17 at 09:18
0

Manually writing tests is "proof by example". But so is QuickCheck, and to a limited extent type systems. Anything that's not straight-up formal verification is going to be limited in what it tells you about your code. Instead, you have to think in terms of the relative merit of approaches.

Generative testing, like QuickCheck, is really good for sweeping a wide space of inputs. It's also a lot better for tackling edge cases than manual tests: generative testing libraries are going to be more experienced with this than you are. On the other hand, they only tell you about invariants, not specific outputs. So to validate your program is getting the correct results, you still need some manual tests to verify that, in fact, foo(bar) = baz.

Hovercouch
  • 550
  • 3
  • 8