15

To give a slightly contrived example, let's say I want to test that a function returns two numbers, and that the first one is smaller than the second one:

def test_length():
    result = my_function()
    assert len(result) == 2

def test_order()
    a, b = my_function()
    assert a < b

Here, if test_length fails, then test_order will fail too. Is it a best practice to write test_length, or to skip it?

EDIT: note that in this situation, both tests are mostly independent from each other, each one can be run in isolation, or they could be run in reverse order, this does not matter. So none of these former questions

is a duplicate of the one above.

Community
  • 1
Mihai
  • 261
  • 1
  • 7
  • 2
    possible duplicate of [Unit testing - one test is partly a "superset" of another, is this wrong?](http://programmers.stackexchange.com/questions/245061/unit-testing-one-test-is-partly-a-superset-of-another-is-this-wrong) –  Jan 12 '15 at 15:41
  • 2
    @GlenH7 - this seems like a different question. The other is "if you have functions where `A` calls `B` and returns the same result, should you test both `A` and `B`". This is more about the _tests_ being overlapping rather than the function(s) under test. (Though it's confusing as they're currently named). – Telastyn Jan 12 '15 at 15:46
  • possible duplicate of [How should I test the functionality of a function that uses other functions in it?](http://programmers.stackexchange.com/questions/225323/how-should-i-test-the-functionality-of-a-function-that-uses-other-functions-in-i) – gnat Jan 12 '15 at 17:03
  • @gnat: that other question is IMHO written in a way that it is hard to tell if it is a duplicate or not. And it is already marked as a duplicate of two other questions which are **clearly** no dupes of the current one. So I don't see a reason to change the question above to make it more clear it is not a dupe the other one (you can edit **the other question** if you like to, I won't). – Doc Brown Jan 12 '15 at 17:32
  • @DocBrown per my reading, "a function that uses other functions in it" is just what is asked about here. As for the dupes there, one about [another test's setup](http://programmers.stackexchange.com/q/221766/31260) looks like a fit here too. FWIW the latter question is linked to another possible dup, [How to manage success dependency between unit tests](http://programmers.stackexchange.com/q/224829/31260) – gnat Jan 12 '15 at 17:38
  • @gnat: what I said, the question title of the other question is misleading - that one should be changed. And in the question here, neither test 1 is a setup for test 2, nor the other way round. Technically, the two tests above are independent from each other, they could be run in isolation or in arbitrary order. So your new "dupe" is clearly none. – Doc Brown Jan 12 '15 at 17:45
  • @DocBrown _if test_length fails, then test_order will fail too_ -- that reads as a success dependency to me – gnat Jan 12 '15 at 17:46
  • @gnat: see my edit to the question above. – Doc Brown Jan 12 '15 at 17:55
  • 1
    Stricly speaking these tests aren't really overlapping (they don't have success dependency). A function `lambda: type('', (), {'__len__': lambda self: 2})()` will pass the first, but not the second. – liori Jan 12 '15 at 19:40
  • @DocBrown - Good edit, and I think you have made this question sufficiently distinct from the other previously asked questions. –  Jan 13 '15 at 15:58
  • 1
    @GlenH7: FWIW, I did not really change the question, just tried to make it gnat-safe ;-) (@gnat: no offense, just joking!) – Doc Brown Jan 13 '15 at 16:02

4 Answers4

27

There can be value, but this is a bit of a smell. Either your tests aren't well isolated (since test_order really tests two things) or you're being too dogmatic in your testing (making two tests testing the same logical thing).

In the example, I would merge the two tests together. Yes, it means you have multiple asserts. Too bad. You're still testing a single thing - the result of the function. Sometimes in the real world that means doing two checks.

Telastyn
  • 108,850
  • 29
  • 239
  • 365
  • I upvoted your answer though it misses one minor point. In Python, the second test will also fail when `my_function` does not return exactly two values, without any assert - because the assignment will result in an exception. So there is actually no need to use multiple asserts and two checks to test the same thing. – Doc Brown Jan 12 '15 at 15:58
  • @DocBrown - I assumed as much, but I'm not familiar with Python's error messages to know if there may be value in having an assert failure instead of random exception/error for informative reasons. – Telastyn Jan 12 '15 at 16:00
  • I think in the current situation the error message would probably be informative enough. But in general, that might not be the case. That's why I agree to you that the general approach for a case like this should be to think of a "merge", using two asserts. If it turns out that the test can be further simplified by leaving out one of the assert, fine. – Doc Brown Jan 12 '15 at 16:06
  • 5
    @DocBrown The other value in explicitly checking with an assert is that it becomes clear to anyone examining the test that this is a failure of the code under test, and not the test itself. When my tests encounter unexpected exceptions, I always have to spend some time figuring out which one is actually wrong. – Chris Hayes Jan 12 '15 at 21:49
  • @ChrisHayes: I wrote "there is no need", not "there is no value" in it. – Doc Brown Jan 12 '15 at 22:09
  • @DocBrown I never said you claimed that. Just trying to add to the conversation here. – Chris Hayes Jan 13 '15 at 01:50
5

Your tests should be explicit. It's not completely inferred that if text_length fails test_order fails.

I'm not sure how it goes in Python which you've posted, but if len(result) is 3 then the first will fail but the second may pass (and if not in Python, then in languages like JavaScript for sure).

Like you said, you want to test that the function return two numbers and that they are in order. Two tests it is.

Madara's Ghost
  • 8,787
  • 9
  • 25
  • 33
  • 1
    `It's not completely inferred that if text_length fails test_order fails` - in Python, it is, will give you an exception. – Doc Brown Jan 12 '15 at 15:59
  • I think the question is about the case where failure of `test_length` implies failure of `test_order`. The fact that a similar pair of tests written in Javascript wouldn't behave the same as these two python tests is kind of irrelevant. – Dawood ibn Kareem Jan 12 '15 at 19:36
  • 2
    @DavidWallace: remember, the question was "Is it a best practice to write test_length, or to skip it"? In Javascript, you *need* both tests to make sure you did not miss an issue (when you don't merge the two tests into one). In Python, you can safely omit the first (which is denied in the first sentence of this answer, and declared as "I am not sure" in the second). Honestly, a good answer looks different. However, I did not downvote this answer since the good part actually is the mentioniong of JavaScript. – Doc Brown Jan 12 '15 at 20:29
  • @DocBrown No, the question was "Is there any value in writing a unit test that is a subset of another test?" and the two particular tests shown were a Python example. The fact that similar tests would behave differently in Javascript is off-topic for the question. The very mention of Javascript is a red herring. (BTW I didn't downvote this). – Dawood ibn Kareem Jan 12 '15 at 22:04
  • 4
    @DavidWallace: since we are here an programmers.com, not on stackoverflow.com, IMHO giving an answer which could be applied to more than one language is better than an answer only for a particular language. But when referring to a specific language, the answer should be correct about the language, and not mix up some properties. – Doc Brown Jan 12 '15 at 22:25
  • 1
    Thing is, the question is "is it worth writing a test that's a subset of another test", and this answer says, "in some languages, neither of your tests is a subset of the other" and then reaches the conclusion both tests are therefore necessary. It reads to me as if someone said, "your code doesn't compile at all in C++, and besides, in C++ a function can only return one value so you don't need to test it returns two. Only write one test" ;-) – Steve Jessop Jan 13 '15 at 09:44
  • I do agree with the other two paragraphs, though. If `test_length` is just removed, then there's nothing to say whether `test_order` is supposed to test that exactly two values are returned in order, or merely to test that all values returned are in order and the test has erroneously assumed there will only be two. So the single test (absent some comments, anyway) doesn't do as good a job of explicitly documenting `my_function()`. – Steve Jessop Jan 13 '15 at 09:50
2

The only value of test_length here is that, if everything passes, its very presence indicates that "length" has been tested.

So it's really unnecessary to have both these tests. Keep only test_order but consider renaming it test_length_and_order.

Incidentally, I find the use of names that begin with test a bit clumsy. I'm a strong advocate of test names that actually describe the condition that you're asserting.

Dawood ibn Kareem
  • 1,852
  • 12
  • 14
  • 1
    The `test` wart is meaningful to Python's test framework. Which of course needn't change whether you're a fan of it, but does change the weight of argument needed to stop someone using it ;-) – Steve Jessop Jan 12 '15 at 21:04
  • @SteveJessop Yeah, I keep forgetting that it's needed. When I was writing Python tests a while ago, I got into the habit of beginning them all with `test_that_`, like `test_that_return_values_are_in_order` and so on. Maybe a future version of the test framework will work around this requirement somehow. – Dawood ibn Kareem Jan 12 '15 at 22:29
  • @DavidWallace: you can [change the prefix](https://docs.python.org/3.2/library/unittest.html#unittest.TestLoader.testMethodPrefix) if you want to. – Lie Ryan Jan 13 '15 at 00:40
1

I'd leave these tests separate if that is how they've evolved to be. Yes test_order is likely to fail whenever test_length does, but you definitely know why.

I'd also agree that if the test_order appeared first and you found a testable failure was that the result was possibly not two values, adding that check as an assert and renaming the test to test_length_and_order makes sense.

If you had to also check the types of the values returned were integers, I'd include that in this "omnibus" results test.

But note now you have a battery of tests for the result of my_function(). If there are multiple contexts (or, more likely, multiple parameters) to test this can now be a subroutine to test all results from my_function().

However, when I write unit tests, I normally test the edge and bad cases separately from good input and normal output (partly because most of my "unit" tests are often mini integration tests), and often with multiple asserts, which are only broken into separate tests if they fail in ways where I find I want more information without debugging.

So I would probably start with your separate tests and expand test_length to test_length_and_types and leave test_order separate, assuming the latter is seen to be "normal processing".

Mark Hurd
  • 343
  • 1
  • 3
  • 12