22

I'm testing that a function does what expected on a list. So I want to test

f(null) -> null
f(empty) -> empty
f(list with one element) -> list with one element
f(list with 2+ elements) -> list with the same number of elements, doing what expected

In order to do so, What is the best approach?

  • Testing all the cases in the same (method) test, under the name "WorksAsExpected"
  • Placing one test for each case, thus having
    • "WorksAsExpectedWhenNull"
    • "WorksAsExpectedWhenEmpty"
    • "WorksAsExpectedWhenSingleElement"
    • "WorksAsExpectedWhenMoreElements"
  • Another choice I wasn't thinking of :-)
malarres
  • 331
  • 3
  • 6
  • 2
    Possible duplicate of [Is this method of writing Unit Tests correct?](https://softwareengineering.stackexchange.com/questions/156417/is-this-method-of-writing-unit-tests-correct) – gnat Apr 18 '18 at 10:35
  • 2
    I would write those as separate test cases. You could use parameterised tests if your test system supports that. – jonrsharpe Apr 18 '18 at 10:38
  • 5
    If you write your tests in a Given...When...Then style then it becomes self-evident that they should really be tested separately... – Robbie Dee Apr 18 '18 at 10:40
  • All your tests will start with "WorksAsExpected..." I'd follow @RobbieDee's advice and use that format – Mart10 Apr 18 '18 at 13:38
  • 1
    I'd just like to add: IMO, it's good to separate out edge cases (like null and empty) into separate tests, because these often involve special case logic across different possible implementations, and if these tests fail, they will indicate clearly in what way the code under test fails (you won't have to dig deeper, or debug the test case to figure out what's going on). – Filip Milovanović Apr 18 '18 at 15:23
  • 1
    List with duplicate elements ? – atakanyenel Apr 18 '18 at 18:00
  • @TheDuplicateQuestionVoters: I don't think bulk operations on a list is necessarily as trivial as "a car has the engine I created I with" Not a duplicate IMHO. – Greg Burghardt Apr 18 '18 at 22:17
  • 1
    I think the example is too generic. If "works as expected" means, for example, that the function is idempotent, then sure these can all be under one `isIdempotent` test. If it means, for example, that calculations are correct + appropriate exceptions raised + emails sent + missiles fired + ... then there should be many more tests than just null + empty + singleton + multiple. – Warbo Apr 19 '18 at 08:19
  • @Warbo yes you're right. I thought that "testing on lists" was narrow enough...but there are zillion of things you can do on lists ;) In my specific case I was sorting the list given some attributes of the objects inside the list (and not others, so I ended up needing a specific comparer) so sorting (null,empty,single) is pretty obvious. But the case with missiles fired would need for sure some extra actions :D – malarres Apr 19 '18 at 10:27
  • @gnat this is not a duplicate of the linked Q – BЈовић Apr 19 '18 at 11:38

5 Answers5

30

The simple rule of thumb I use for whether to perform a set of tests in one test case, or many, is: does it involve just one setup?

So if I were testing that, for multiple elements, it both processed all of them and derived the correct result, I may have two or more asserts, but I only have to set up the list once. So one test case is fine.

In your case though, I'd have to set up a null list, an empty list etc. That's multiple setups. So I'd definitely create multiple tests in this case.

As others have mentioned, those "multiple tests" might be able to exist as a single parameterised test case; ie the same test case is run against a variety of setup data. The key to knowing wether this is a viable solution lies in the other parts of the test: "action" and "assert". If you can perform the same actions and asserts on each data set, then use this approach. If you find yourself adding if's for example to run different code against different parts of that data, then this is not the solution. Use individual test cases in that latter case.

David Arno
  • 38,972
  • 9
  • 88
  • 121
16

There's a trade-off. The more you pack in one test, the more likely you'll have an onion effect trying to get it to pass. In other words, the very first failure stops that test. You won't know about the other assertions until you fix the first failure. That said, having a bunch of unit tests that are mostly similar except for the set up code is a lot of busy work just to find out that some work as written and others don't.

Possible tools, based on your framework:

  • Theories. A theory lets you test a series of facts about a set of data. The framework will then feed your tests with multiple test data scenarios--either by a field or by a static method that generates the data. If some of your facts apply based on some preconditions and others don't these frameworks introduce the concept of an assumption. Your Assume.that() simply skips the test for the data if it fails the precondition. This lets you define "Works as expected" and then simply feed it a lot of data. When you view the results, you have an entry for the parent tests and then a sub-entry for each piece of data.
  • Parameterized Tests. Parameterized tests were a precursor to theories, so there may not be that precondition checking that you can have with theories. The end result is the same. You you view the results, you have a parent entry for the test itself, and then a specific entry for each data point.
  • One test with multiple asserts. It takes less time to do the set up, but you end up uncovering problems a little at a time. If the test is too long and there are too many different scenarios tested, there's two big risks: it will take to long to run, and your team will get fed up with it and turn off the test.
  • Multiple tests with similar implementation. It's important to note that if the assertions are different they tests aren't overlapping. However, this would be the conventional wisdom of a TDD focused team.

I'm not of the strict mindset that there can only be one assert statement in your test, but I do put the restrictions that all of the asserts should test the post-conditions of an single action. If the only difference between the tests is data, I'm of the mindset to use the more advanced data driven test features like parameterized tests or theories.

Weigh your options to decide what the best result is. I will say that "WorksAsExpectedWhenNull" is fundamentally different than any of the cases where you are dealing with a collection that has varying numbers of elements.

Berin Loritsch
  • 45,784
  • 7
  • 87
  • 160
5

Those are different test cases, but the code for the test is the same. Using parameterized tests is therefore the best solution. If your testing framework does not support parameterization, extract the shared code into a helper function, and call it from individual test cases.

Try to avoid parametrization via a loop within one test case, because that makes it difficult to determine which data set caused the error.

In your TDD red–green–refactor cycle, you should add one example data set at a time. Combining multiple test cases into a parametrized test would be part of the refactoring step.

A rather different approach is property testing. You would create various (parametrized) tests that assert various properties of your function, without specifying concrete input data. E.g. a property could be: for all lists xs, the list ys = f(xs) has the same length as xs. The testing framework would then generate interesting lists and random lists, and assert that your properties hold for all of them. This moves away from manually specifying examples, as manually choosing examples could miss interesting edge cases.

amon
  • 132,749
  • 27
  • 279
  • 375
3

Having one test for each case is appropriate because testing a single concept in each test is a good guideline that is often recommended.

See this post: Is it OK to have multiple asserts in a single unit test?. There's a relevant and detailed discussion there as well:

My guideline is usually that you test one logical CONCEPT per test. you can have multiple asserts on the same object. they will usually be the same concept being tested. Source - Roy Osherove

[...]

Tests should fail for one reason only, but that doesn't always mean that there should be only one Assert statement. IMHO it is more important to hold to the "Arrange, Act, Assert" pattern.

The key is that you have only one action, and then you inspect the results of that action using asserts. But it is "Arrange, Act, Assert, End of test". If you are tempted to continue testing by performing another action and more asserts afterwards, make that a separate test instead. Source

sonnyb
  • 139
  • 1
  • 6
0

In my think, it depends on test condition.

  • If your test has only 1 condition to setup the test, but many side effects. multi-assert is acceptable.
  • But when you have multiple conditions, means you have multiple test cases, each should be covered by 1 unit test only.
HungDL
  • 139
  • 2