How to decide what forms the System Under Test

Question

Ok, so we started with the initial version of "The Art Of Unit Testing" and we do unit tests for classes where every test covers one aspect of one method.

This has the drawback of high maintenance costs and brittle tests and seems that the second version of "The Art Of Unit Testing" points to take bigger units when testing.

Which is the right size (the one that you use) when doing unit tests? I've heard of testing only the public interface provided by a module but in the project we are working we have a relative small public surface with a very big back office so I'm not sure if that would work as the Arrange of the tests would be really big and probably difficult to follow.

So, again, how to you set the boundaries of your SUT or how do you decide what forms your SUT?

Answer 1

Ideally things would vary according to the rule of common sense. Martin Fowler talks about a unit as a class (plus related classes in some cases) and IMHO a unit should be a useful lump of code, this may be a single method where a method is quite complex, or a whole class as classes are the means of dividing a system into manageable components.

A single method is generally not a unit, as they tend to be too granular to be of any practical use.

Perhaps a rule of thumb is to think of the test as documentation - you can use your tests as example code, a how-to-use a piece of functionality. If you think that a test is meaningless to anyone else, as the tests are too small or trivial, then its a smell that the unit being tested is too small too, and should be enclosed in a wider unit. So a complex method may require a lot of possible inputs, and your test would demonstrate them, but a network class (say) would have a test that describes the entire class by constructing it, setting hostname and port, and then calling send or receive methods on it - those methods by themselves being useless by themselves.

Only testing public interface makes sense - if the code cannot be accessed by the public interface, then there's no real reason to have it! However, you say you have a large back-office functionality, the API to access the component parts of the back-office is still a public interface (ie its public to a computer client or class) and should be considered for a unit test in that regard.

Answer 2

There is no "one size fits all" answer to where you should put the boundaries of your SUT.

The optimal size of a SUT for unit-testing depends on a lot of factors, including

• The language and toolchain you are working with (how many errors can be detected by static analysis/compiling the code)
• The complexity of the product you are making
• How familiar the team is with creating such a product

A team writing their tenth low-complexity website might use much larger SUTs in their unit-tests than a team that is new to the job and/or is tasked with a very complex site.

Choosing the size of your SUT is a balancing act. If it is too small, then the tests start to depend on implementation choices, which causes rework of the tests if the implementation changes.
If it is too large, then there is no easy correlation between "Test A fails" and "There is a problem in function/area B".

Some general guidelines are:

1. The tests are a client of a class, just like the other modules that will eventually interface with it. Testcases should not have access to internals of a class unless that is absolutely necessary to verify that a particular requirement has been met. And even then you should first look if that requirement can be verified using only the public members of the class.

2. Each test should verify only one requirement. Often this is misrepresented as having only a single assert in a test. Quite often requirements can't be verified with only a single assert statement, leading to people writing multiple testcases where each test covers only part of the verification. If you need several closely related assert statements to verify a requirement, then it is perfectly fine to have it all in one test.

3. You should mock away all dependencies of your SUT, unless they are known to be fast and well-tested (like standard or third-party classes). When writing expectations for these mocks, try not to rely on the order in which calls are made, unless the order of the calls is an explicit requirement.

4. Focus your testing effort on the areas with high complexity and/or uncertainty. There is very little value in writing a test for a one-line getter. For those functions, it is very easy to see that they are correct and there is little risk that a future change will break them.