Is this a test smell or is it even worse?

Question

I have recently been looking at some testscripts which looks a bit like

...
try {
  receiveSomething();
  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

The problem I have with these types of tests is

They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.
Its very hard to simultaneously test something else, which might actually, in this case, send something.

My thoughts are one the one side, how can this types of tests be designed better, and secondly can this be an indication of a bigger design smell of the software that is being tested?

EDIT

To clairify, these test scripts are used for black box-testing of event-based, complex software, not unit testing, and my feeling is that the 'doing nothing' event is a very ambiguous one. :)

as soon as test and code allow for a proper **[tag:mocking]** of the data source used in `receiveSomething()`, there would be nothing wrong with this snippet of test — gnat, Feb 14 '13 at 12:14
@Theodor: you may edit your question to make more clear what you really meant - seems most of the other folks misunderstood that. — Doc Brown, Feb 14 '13 at 14:55

pdr · Answer 1 · 2013-02-14T12:09:20.870

12

They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.

It doesn't matter, in the context of this test. All that matters is that, under the preconditions defined, nothing is received. Other tests can simulate other preconditions and check their results are as-expected.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

Good! You shouldn't try to plug other tests into this test. This test is doing exactly what it should be doing. One set of preconditions => One result.

This is the very definition of determinism.

edited Feb 14 '13 at 12:09

answered Feb 14 '13 at 11:52

pdr

53,387
14
137
224

I see your points here. Ok, so testing a feature like this might be ok, if it was designed this way. But would a more robust design be to _explicitly_ and not _implicitly_ communicating that nothing was to be expected, e.g. send a `State.PASSIVE` message? – Theodor Feb 14 '13 at 12:17
Maybe. That's a code decision rather than a test decision. Does it make your CODE more robust? If so, do it and test it accordingly. If not then test what you have. – pdr Feb 14 '13 at 12:30
@pdr: I guess the situation of the OP is a little bit different, see my answer below. – Doc Brown Feb 14 '13 at 13:32

Doc Brown · Accepted Answer · 2013-02-14T14:52:02.480

From what you wrote, I guess your situation is more like this

startAsynchronousComplexOperation();
try {
  receiveSomething(); // listen for events from that async operation above

  // something was received even though it shouldn't
  failTest();
} catch (TimeoutException e) {
  // nothing should be received
  succedTest();
}

and the object under test is not "receiveSomething", but the asynchronous operation. Thus the outcome of that test depends on the speed of the test machine, if a lot of other processes run on the test machine etc., and this is non-deterministic.

IMHO, this is indeed a design problem, and not just a "test smell". If your "complex operation" crashes, or just "takes too long", not only your test will give a "false negative". Think about how to deal with that situation in production. Is there no requirement to determine for sure that your "complex operation" is still alive and did not crash? So you may think about adding some kind of "watchdog" or "keep-alive-packet-sending" mechanic to your program to get control over that. And once you have such a mechanic available, you can use it in your tests like this:

startAsynchronousComplexOperation();
while(isComplexOperationStillAlive())
{
    try {
       receiveSomething(); // listen for events from that async operation above
       // something was received even though it shouldn't
       failTest();
   } catch (TimeoutException e) {
   }
}
if(didComplexOperationCrash())
    failTest();
else
    succedTest();

Interesting leap and *could* be spot on, but it doesn't answer the question, as they are both currently written. — pdr, Feb 14 '13 at 13:40
@pdr: I am pretty sure it answers the question as it was meant. "receiveSomething" obviously waits for an event - from where, if not from a part of the code the OP forgot to add to the example? And why else does he think the test is non-deterministic? — Doc Brown, Feb 14 '13 at 14:17
Exactly. These scripts are testing event-based, complex software :) The 'doing nothing' event is a very ambiguous one. — Theodor, Feb 14 '13 at 14:32
Upvoting because of correct assumptions about the question :) — Silver Quettier, Feb 14 '13 at 15:09
+1 for good answer to question he meant to ask and eventually did — psr, Feb 14 '13 at 20:01

score 1 · Answer 3 · answered Feb 14 '13 at 10:24

Very hard to answer this without some context.

Were the test case to be for the function point ("System Will not respond to any request after 'Quiesce' command issued") then yes this would be a perfectly valid test case.

Although I would have specified that a valid response was received immediately before the shutdown request - but these type of "operational" tests are very difficult to automate as its unlikely you would be able to issue such system commands from an API.

score 1 · Answer 4 · answered Feb 14 '13 at 12:02

They are not deterministic. You don't know if nothing was sent on purpose or if everything has crashed.

A deterministic algorithm as defined in Wikipedia is as such:

is an algorithm which, given a particular input, will always produce the same output

But the algorithms to build the test hardly matter, pretty much because it is supposed to test the proper state after invoking a specific behavior. In other words, the only thing that needs to be deterministic is the code module that you are testing. The test itself is actually definining what the determinination should be.

Its very hard to simultaneously test something else, which might actually, in this case, send something.

As others pointed out, that is a GOOD THING. If I have a test for module A, that has dependencies on Modules B and C, then if TestModuleA fails, how do I know it is ACTUALLY an issue in module A?

Your test is supposed to just verify that Module A provides the correct state for invoking an operation of A. You should have seperate tests for the dependencies to verify their correct behavior too.

Is this a test smell or is it even worse?

4 Answers4