Should test code be treated as normal code?

Question

I remember reading on someone else's post that test code should not be treated as normal code but as small whole programs and that is an idea I really like and would like to refer to when discussing with other engineer colleagues, but is not unanimously shared.

Specially I was referring to it when discussing if test should be refactored as normal code or not, using private functions for example. Some colleagues were encouraging developers to transform test like these:

public test1() {
    variable1 = 'value1'
    variable2 = 'value2'
    variable3 = 'value2'
    assert_something1()
    assert_something2()
    assert_something3()
}

public test2() {
    variable1 = 'value1'
    variable2 = 'value2'
    different_variable3 = 'different_value3'
    assert_something1()
    assert_something2()
    assert_something_different3()
}

Into this:

public test1() {
    set_common_variables()
    do_common_assertions()
}

public test2() {
    set_common_variables()
    different_variable3 = 'different_value3'
    do_common_assertions()
    assert_something_different3()
}

private set_common_variables()
{
    variable1 = 'value1'
    variable2 = 'value2'
}

private do_common_assertions()
{
    assert_something1()
    assert_something2()
}

Arguing they were just encouraging basic refactoring rules for a better code readability by avoiding repetition, but I personally think these do not apply here for these reasons:

• Test should be meant to be read separately and short, so I think premises and assertions should be together, event if this means repearing code.

• The example above would point a refactor need but not in the test itself but in the code, by refactoring the test we may be occluding what is going on.

What are your experiences on this? Do you think tests should be generally treated as normal code or not?

Answer 1

I remember reading on someone else's post that test code should not be treated as normal code but as small whole programs and that is an idea I really like and would like to refer to when discussing with other engineer colleagues, but is not unanimously shared.

Working Effectively with Unit Tests, by Jay Fields, covers this topic and may offer useful arguments supporting your position.

You may also want to look into DAMP not DRY.

In contexts like Test Driven Development, where your automated checks are expected to provide developer support that you would otherwise get from detailed design documents, reading a test is supposed to evoke a similar experience to reading a specification. I, myself, don't particularly enjoy reading specifications where a lot of navigation is required.

Another spelling of the same idea: since the tests themselves don't have dedicated tests, you want a design that obviously has no mistakes.

That doesn't mean no shared code in the checks, but the shared code is usually centered on sharing intent, rather than avoiding duplicated lines. You might have, for instance, routines that return test data, with spellings chosen so that you can easily tell which data is significant for your test, and which is incidental.

// Verify that the payroll code does the right thing
// in a state with no income tax
employee = AnyEmployee.withHomeState("TX")

...

Answer 2

Definitively, the testing code worth be treated with so much care as the production code for the next reasons

• Testing code is an important documentation resource. Testing code speaks about the system we are working on. It tells stories about the business or the domain of the system. How components should behave, how the system behaves when components interact with each other. What's acceptable and what's not. This information should be easy to get by looking at the tests but I would find it hard with testing code hard to read or code hard to reason about. Readability matters no matter the code.

• Our confidence in the good behaviour of the system lays upon this code. No spaghetti code can be reliable and is worthy of our confidence for a long time.

• Testing code harder to maintain affects our capacity to write new reliable production code or evolve the existing one. It eventually affects productivity which in turns affects our capacity to adapt the code the changing demands of the market. Having a testing specific domain language can make a difference between systems easy (and cheap) to evolve than those that are not. For companies, productivity and time-to-market matters.

• Testing code harder to maintain also discourages developers from writing more tests or maintaining the existing ones. The consequences of this are suffered by the production code which starts to decay progressively. If we can not evolve the production code with a high grade of certainty and confidence, the whole system starts dying.

Bear in mind that the testing code will be maintained throughout the whole system life. Probably we will start writing production code from the testing code and at this point, we will wish to have testing code reasonably well structured and reusable.

Why do I say this? Because of Test should be meant to be read separately and short, so I think premises and assertions should be together, even if this means repeating code.

public test1() {
    set_common_variables()
    do_common_assertions()
}

public test2() {
    set_common_variables()
    different_variable3 = 'different_value3'
    do_common_assertions()
    assert_something_different3()
}

The snippet above is fairly short and easy to read¹.

Moreover, encapsulating do_common_assertions helps us out to abstract us from what do_common_assertions means. If do_common_assertions validate rules prone to change, worth having these validations in a single place so that other tests can take advantage of it. Otherwise, we expose the testing code to unexpected false positives, those caused by developers forgetting to update tests with the new rules.

Finally, I have come to realise that verbosity kills readability as much as too many nested ìf/else or try/catch blocks. After several lines that looks 90% the same, we stop reading and we start losing the focus.

---

^{1: Assuming it has meaningful methods names and they evoke the domain's ubiquitous language}

Answer 3

I agree with your colleagues. DRY should be applied to tests as well as production code. The authors of test frameworks also agree by providing mechanisms (annotations, function decorators, etc) to execute a common setup method before each test method runs.

Answer 4

I think it depends on what you want to achieve. If its only a safety net to prevent regression, especially if it's solely from the outside (i.e. its treated as a closed black box that answers "Is it broken?") then the requirements are much the same as production code. Indeed there are production codes that black box test devices. If this is the case then the test code should probably be treated as such. If the aim is to be documentation that happens to run I am inclined to agree with OP sometimes more explicit it clearer:

bool test_foo_completes_for_negatives(){
  foo(-1);
  foo(-2);
  foo(-102);

  return true;
}

Reads a lot easier in terms of what's tested

template <typename I>
bool test_foo_completes_for_negatives(){
  std::vector<I> negatives = get_negative_list();
  for (auto& test_val : negatives) {
    foo(test_val);
  }

  return true;
}

Even if the later is more flexible and reads easier in terms of the intent of the test. It doesn't make one better or worse as always its a trade off.

Though it might be worth noting test code is very rarely treated exactly the same as production code. For example test code rarely has its own (meta) tests.