To what extent should I test what should not happen

Question

In the case of unit testing, I'm testing as many cases as I can that when I do X, Y happens.

I'm also concerned in some cases that when I do X, Z does not happen.

But for the sake of protection, I could also test that A, B, C ... do not happen either. And going that direction I could basically test a billion things that should not happen, and, in many cases, won't ever happen.

My question is, at what point do you think you're being too defensive, and when is it still considered useful ?

Let's use this bit of pseudo code as an example

void MyFunction()
{
    if (X)
    {
        A();
    }
    else
    {
        B();
    }

    C();
}

void AnotherFunction()
{
   //Irrelevant code
}

Here are the tests I can have :

A happens when X is true
A does not happen when X is false
B does not happen when X is true
B happens when X is false
C happens when X is true
C happens when X is false

This is what I would do. But following that reasoning I could also test that AnotherFunction is not called when X is true and false, just like I'm testing that C is called when X is true and false or that A/B are not called when X is false/true.

And going down that road I could literally test a million things that shouldn't happen for every test case I can find. Just to make sure next developers won't break the code.

In my example you might say "this is too obvious to be tested" but I'm sure you can imagine a point where code could be so fragile / complex that testing many different things that shouldn't happen could be done. My question is : should it be tested, and how far ?

Is there some kind of rule ? Am I missing something in the unit testing knowledge ?

On one side I feel that this is safer and offers a good defense against pretty much anything that I can think of. On the other hand this is far too defensive and abominable to maintain, while time consuming since you get to code 'anything that you could think of'.

Note that I applied this to function calls, but it extends to anything that follows the line of "testing things that should not happen".

Where should one stop testing in that direction?

Possible duplicate of [Where is the line between unit testing application logic and distrusting language constructs?](http://softwareengineering.stackexchange.com/questions/322909/where-is-the-line-between-unit-testing-application-logic-and-distrusting-languag) — gnat, Apr 06 '17 at 13:33
Test-first (or even thinking test-first even if the code is already there) tends to help with these kinds of doubts. If you were to write your tests first, you might start by saying "A should be called if X" and then just write `Method() { A(); }` - you'd then need to write a test that says "A should not be called if not X" to drive out `Method() { if(X) A(); }` ... nothing about this would ever lead you to feel you needed to write a test called `AnotherFunctionShouldNotBeCalled`. — Ant P, Apr 06 '17 at 14:26
What kind of application? "Being too defensive" is going to be different whether you're writing dentist appointment scheduling or the flight controller for the space shuttle. — whatsisname, Apr 06 '17 at 16:14
Somewhere in between a space shuttle and a dentist app. It's a banking app (yet another one) that is expected to be widely used and stability is my top concern. My main task is to go behind everyone, triple check everything, refactor and add missing tests. In many cases this means writing the very first tests. — Gil Sand, Apr 07 '17 at 09:32
If you are asserting preconditions, post-conditions, and invariants in at least one type of debug build, you get more testing value for your money, as you get more testing with each test. This eliminates some of the bulk of your test code. Figuring out those specifications also lets you create informal proofs of behavior that are generally more valuable than tests, as those proofs work with all possible inputs, and can help you reduce the state space of your program. — Frank Hileman, Apr 09 '17 at 00:29

Becuzz · Accepted Answer · 2017-04-06T14:15:33.953

Tests serve a few purposes.

Make sure requirements are met.
Make sure code behaves as expected.
Describes the expected behavior of the code.

Also, the tests you write need to give you value. Value comes from fulfilling one of those purposes.

For example, writing a test to make sure that 2 + 2 actually yields 4 is something you could do, you know, to make sure the compiler and your CPU work as designed. But that doesn't give you a lot of value and doesn't justify writing the test. (Nor does it really fulfill a purpose of a test.)

It's possible to write tests to make sure that RandomOtherFunction isn't called, but that isn't likely to give you value. It isn't (typically) fulfilling one of those purposes. It's describing what the code isn't. And that would be impossible to write everything the code isn't. It also isn't valuable to you to know what the code isn't and that no solar rays or gramlins have changed the code into that.

Tests that specify too much of what your program isn't also makes it hard to change. What if you, for some reason in the future, have to start using RandomOtherFunction in a place you didn't used to? Now you have to refactor lots of tests on top of doing the work. The tests didn't help you make the change or make it safely, they just made it harder.

You stop writing tests when they don't give you value. How much value justifies writing a test is up to you and your organization. If your company really, really wants to pay you to write a million tests about what the software isn't, fine. Let them pay you for that "value". (You also may want to look for a new job because I can't imagine a company that pays for that kind of value lasting very long.)

Haha, thanks a lot. No, nobody is forcing me, I'm questioning myself because I see my tests slowly deviate towards testing what the code isn't and it felt more wrong every second. — Gil Sand, Apr 06 '17 at 13:54
It's worth also mentioning that overspecification in tests can be an enormous hindrance to change. — Ant P, Apr 06 '17 at 13:55

To what extent should I test what should not happen

1 Answers1