Looping Exhaustive Test Cases - A Good Practice?

Question

Sometimes we have code like this:

private static final List<Goat> INEDIBLE_GOATS = [GOAT_A, GOAT_B, GOAT_C] 

void eatGoat(Goat goat) {
   if(INEDIBLE_GOATS.contains(goat)) {
       throw new CantEatGoatException();
   }
   ...
}

Should it best tested (along with whatever tests for the actual eating) as:

for(Goat goat: INEDIBLE_GOATS) {
   // forgive abuse of syntax
   assertThrows(CantEatGoatException, eatGoat(goat));
}

Or more like?

assertThrows(CantEatGoatException, eatGoat(GOAT_A));

Should tests verify each and every sort of inedible goat? Or should they test a representative goat from the list and trust that it works for the other goats?

ROFL. Why do I get visions of farmers' sons doing unthinkable things with the livestock when nobody's looking? — Robert Harvey, May 12 '14 at 23:42
In other news, what coding convention is this? Makes my eyes burn. — Robert Harvey, May 12 '14 at 23:43
Shouldn't your test include both the positive and negative results? ``assertTrue(isFoobarableGoat(new Goat())`` and ``assertTrue(isFoobarableGoat(new FoobarableGoat())`` — Adam Zuckerman, May 12 '14 at 23:55
@AdamZuckerman, obviously. But I didn't include them to reduce TLDR factor. — Winston Ewert, May 12 '14 at 23:59

score 5 · Answer 1 · answered May 13 '14 at 04:37

A major reason to have tests is to build your confidence in your code. Tested code can be proved to have a specific functionality because the tests say they do. Confidence grows from the proof. The question is, by looping through all the INEDIBLE_GOATS, do you gain any more confidence in your code? Either way, with both positive and negative tests, you cover the relevant code. With the loop, though, you are basically exercising these lines over and over. Essentially, you are testing that List<Goat>#contains() does what it says. Do you gain anything from that? If you trust that a List works as it should, then no, you don't get anything additional. If, on the other hand, you are not confident that a List works as it should, then you should have a separate test of List#contains() to give you that confidence.

You also have to balance the number of tests with the performance of your test suite. Looping through all the INEDIBLE_GOATS may give you a little more confidence, but if there are, say, 1.8 million of them, and running the tests takes 30 seconds longer, does that tiny additional confidence outweigh the lost time (and ensuing lost productivity).?

Finally, while INEDIBLE_GOATS may only have a small number of elements now, theoretically, the number of inedible Goats may actually be infinite. It is impossible to test every potential Goat. While in this simplified example, you may say that there won't really be infinite Goats, in general, we must account for infinite and near-infinite possible inputs and accept that we cannot practically test everything. We must also realize that this is a good thing. Not all inputs are equal, and trying to test everything wouldn't necessarily get us any more confidence. It is optimizing tests for confidence that makes them practical and profitable.

As an aside, I tend to test code that uses complex regexes pretty thoroughly as they are fragile and hard to read. The cost of thorough testing is worthwhile because small, hard-to-notice-or-decipher changes can break a regex, and knowing that as soon as possible helps to speed fixing the bug. — cbojar, May 13 '14 at 04:41

score 0 · Answer 2 · answered May 12 '14 at 23:45

0

The second test seems more accurate to me. If you put a Goat within your List, you expect that Goat to be a foobarable Goat (that is, you expect to get true from your method).

answered May 12 '14 at 23:45

b3by

1
1

What does "more accurate" mean? – Robert Harvey May 12 '14 at 23:46
I assume the test he's talking about deals with the method `boolean isFoobarableGoat(Goat goat)`. So, if he calls that method for a bunch of goats already contained within the list, he cannot be sure the method works the other way as well. Maybe he should add to his test something like `assertFalse(isFoobarableGoat(outerGoat));`. – b3by May 12 '14 at 23:50

score 0 · Answer 3 · answered May 12 '14 at 23:46

0

Seems like you're just pushing the logic around. How about:

assertTrue(FOOBARABLE_GOATS.contains(goat));

answered May 12 '14 at 23:46

Robert Harvey

198,589
55
464
673

1

The whole point is to test isFoobarableGoat. So its kinda not helpful to not call that function. – Winston Ewert May 12 '14 at 23:49
2

Yeah, this is what happens when your examples are too abstract. – Robert Harvey May 12 '14 at 23:50
I've tried to deabstract it a bit. – Winston Ewert May 12 '14 at 23:58
Does the collection only have to contain one FoobarableGoat, more than one, or does the entire collection need to contain FoobarableGoats? – Robert Harvey May 13 '14 at 00:00
There is only one collection and its static final. Its part of the code and always has sone exact number of arguments. – Winston Ewert May 13 '14 at 00:03

score 0 · Answer 4 · edited Apr 12 '17 at 07:31

Your question is an important one. To restate more generically: should you test all possible inputs or just selected ones? As @cbojar points out, the former is typically either (a) not practical or (b) not useful.

The question then becomes how to whittle the list of all possible test inputs into a set of optimal test inputs. To do this, divide the set of all possible inputs into groups based on expected outcomes. Any input from one group should yield equivalent (though not identical) results. Such groups are called equivalence classes. and this process of identifying such classes is equivalence partitioning.

As a simple example, to test a square root function that takes an integer input, a good starting point would be these equivalence classes:

 1   -- all positive integers
-1   -- all negative integers
 0   -- zero

You might refine that list further:

 1   -- all positive integers that are not perfect squares
 4   -- all positive integers that are perfect squares
-1   -- all negative integers
 0   -- zero

And if you are a real stickler, you might add maxint and minint for your integer type. But with any of these you have infinitely less test cases than the set of all possible inputs :-) -- a pretty good savings!

(For more, see my answer to a related question, What should you test with unit tests?)

Looping Exhaustive Test Cases - A Good Practice?

4 Answers4