90

I am writing tests for a project that consists of multiple submodules. Each test case that I have written runs independent of each other and I clear all data between tests.

Even though the tests run independently, I am considering enforcing an execution order, as some cases require more than one submodule. For example, a submodule is generating data, and another one is running queries on the data. If the submodule generating the data contains an error, the test for the query submodule will also fail, even if the submodule itself works fine.

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

In this case, is it OK to enforce an execution order for the tests or is it bad practice? I feel like there is a smell in this setup, but I can not find a better way around.

edit: the question is from How to structure tests where one test is another test's setup? as the "previous" test is not a setup, but tests the code which performs the setup.

Ali Rasim Kocal
  • 992
  • 1
  • 6
  • 11
  • 3
    Possible duplicate of [How to structure tests where one test is another test's setup?](https://softwareengineering.stackexchange.com/questions/221766/how-to-structure-tests-where-one-test-is-another-tests-setup) – gnat Apr 03 '18 at 13:33
  • 124
    If you're testing the connection to a remote server, then they're _by definition_ not unit tests. – Telastyn Apr 03 '18 at 13:34
  • 9
    The first answer confused me here because in your title you said "Is it bad practice?" and in the summary of your question you wrote "is it ok?" Anyone answering either yes or no is going to confuse one of those! – Liath Apr 03 '18 at 13:53
  • 8
    Sounds like you are creating set of integration tests. Even for that one test shouldn't rely on other tests. – Low Flying Pelican Apr 03 '18 at 14:03
  • 17
    If the order matters then you are probably doing it wrong. – Mark Rogers Apr 03 '18 at 15:39
  • 3
    To clarify, are you saying that the tests have no dependencies between themselves, but some of them depend on a particular piece of common code (making them not-quite-unit tests), if the test for that code fails, there is no point running the other tests? – James_pic Apr 03 '18 at 16:34
  • 3
    Normally you'd mock the black-box remote server for this. Doing real comms in a unit test causes all sorts of problems: should the tests fail if the Internet is down? – pjc50 Apr 04 '18 at 09:14
  • 1
    I believe that unit tests independent. I also like them all to start/end in a known "idle" state. In addition to running each test once, I run a 1,000 or so loop invoking random tests. This catches cases where one functionality unknowingly sets up data for the next, when it ought not to, or one piece of code relies on "hangover" data from another (E.g reading data after the pointer to has been freed, but not NULLed) – Mawg says reinstate Monica Apr 04 '18 at 12:49
  • 2
    Your builds will be faster if you can parallelize. Faster builds are generally favourable. So basically: no, don't enforce an order. And don't let tests fail just because others failed – BlueWizard Apr 04 '18 at 18:57
  • 1
    If there's more than one submodule involved it's an integration test, which you can run in addition to your unit test suite. Integration tests are a _good_ thing, especially if you can code in communication failures and test the system's response to those. See Paddy's answer below : his refactoring could help with both testing the data generation and tests that use that generated data. – WillC Apr 04 '18 at 23:42
  • If there is an error in your individual tests (or in the code) which violates the independence of execution order, then a random execution order can lead to random, not reproducible failures. A typical case would be that someone adds caching in the code and one of the tests will pick up cached stuff when run late in the execution chain. Hint: if "Independence of execution order" is a *requirement*, then you need your test needs to test *all* possible execution orders. That's easy to implement, but boring to wait for... – Klaws Apr 06 '18 at 13:22
  • It's actually a good practice to randomize the order of your unit tests to prove that you didn't accidentally let an order requirement slip in, so yes it's a bad practice. – Bill K Apr 06 '18 at 21:13
  • Possible duplicate of [Should each unit test be able to be run independently of other tests?](https://softwareengineering.stackexchange.com/questions/64306/should-each-unit-test-be-able-to-be-run-independently-of-other-tests) – Steve Apr 06 '18 at 21:52
  • At the risk of sounding like an ignorant yokel, but if the order of execution affects the outcome of unit testing, doesn't that directly imply that it's failing the test? – Shadur Apr 07 '18 at 18:02

6 Answers6

241

I can not work with dummy data, as the main functionality I am testing is the connection to a black box remote server, which only gets the data from the first submodule.

This is the key part for me. You can talk about "unit tests" and them "running independently of each other", but they all sound like they are reliant on this remote server and reliant on the "first sub module". So everything sounds tightly coupled and dependent on external state. As such, you are in fact writing integration tests. Having those tests run in a specific order is quite normal as they are highly dependent on external factors. An ordered test run, with the option of an early quit out of the test run if things goes wrong is perfectly acceptable for integration tests.

But, it would also be worth taking a fresh look at the structure of your app. Being able to mock out the first submodule and external server would then potentially allow you to write true unit tests for all the other submodules.

Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
David Arno
  • 38,972
  • 9
  • 88
  • 121
  • 14
    Not to mention that some test has to specifically check that the expected behaviour occurs when the remote server is unavailable. – Alexander Apr 04 '18 at 08:56
  • 3
    Or, perhaps, you are in fact intending to write integration tests, and thus mocking out the data is not going to achieve what you are trying to accomplish with these tests. – Guy Schalnat Apr 04 '18 at 11:51
  • 11
    Problem is most likely that Junit has "unit" in its name. – Thorbjørn Ravn Andersen Apr 05 '18 at 08:48
  • 8
    @ThorbjørnRavnAndersen Exactly. People naturally write integration tests, rather than unit tests, because integration tests are both far more useful and far less difficult to write than "real" unit tests. But because the popular testing frameworks were named after the concept of unit tests, the term has been co-opted and come to mean "any automated testing" in modern parlance. – Mason Wheeler Apr 05 '18 at 21:24
  • 1
    @MasonWheeler Or even co-opted by non technical managers to mean acceptance testing. – StackOverthrow Apr 06 '18 at 00:02
  • @MasonWheeler I'd contest that integration tests are more useful and easier to write. You normally cover less of the various scenarios, and its more difficult to write due to needing to get all your ducks in a row first (which is why coverage is often less). – Andy Aug 09 '18 at 01:49
33

Yes, it's a bad practice.

Generally, a unit test is intended to test a single unit of code (e.g. a single function based on a known state).

When you want to test a chain of events that might happen in the wild, you want a different test style, such as an integration test. This is even more true if you're depending on a third party service.

To unit test things like this, you need to figure out a way to inject the dummy data, for example implementing a data service interface that mirrors the web request but returns known data from a local dummy data file.

Paul
  • 3,277
  • 1
  • 17
  • 16
  • 9
    Agreed. I think this confusion stems from the fact that that many people have an idea that integration tests have to be end-to-end, and use "unit test" to refer to any test that only tests one *layer*. – autophage Apr 03 '18 at 13:56
  • 8
    @autophage: Definitely agree with this. In fact I agree with it so much that I regularly find myself falling into the same trap _despite agreeing that it is a trap_ – Lightness Races in Orbit Apr 03 '18 at 16:16
16

The enforced execution order you propose only makes sense if you also abort the test run after the first failure.

Aborting the test run on the first failure means that each test run can uncover only a single problem and it can't find new problems until all preceding problems have been fixed. If the first test to run finds a problem that takes a month to fix, then during that month effectively no tests will be executed.

If you don't abort the test run on the first failure, then the enforced execution order doesn't buy you anything because each failed test needs to be investigated anyway. Even if only to confirm that the test on the query submodule is failing because of the failure that was also identified on the data generating submodule.

The best advice I can give is to write the tests in such a way that it is easy to identify when a failure in a dependency is causing the test to fail.

Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
Bart van Ingen Schenau
  • 71,712
  • 20
  • 110
  • 179
7

The smell you're referring to is the application of the wrong set of constraints and rules to your tests.

Unit Tests often get confused with "automated testing", or "automated testing by a programmer".

Unit Tests must be small, independent, and fast.

Some people incorrectly read this as "automated tests written by a programmer must be small independent and fast". But it simply means if your tests are not small, independent, and fast, they are not Unit Tests, and therefore some of the rules for Unit Tests should not, cannot, or must not apply for your tests. A trivial example: you should run your Unit Tests after every build, which you must not do for automated tests that are not fast.

While your tests not being Unit Tests means you can skip one rule and are allowed to have some interdependence between tests, you also found out that there are other rules which you may have missed and will need to reintroduce - something for the scope of another question.

Peter
  • 3,718
  • 1
  • 12
  • 20
6

As noted above, what you are running seems to be an integration test, however you state that:

For example, a submodule is generating data, and another one is running queries on the data. If the submodule generating the data contains an error, the test for the query submodule will also fail, even if the submodule itself works fine.

And this may be a good place to start refactoring. The module that runs queries on the data should not be dependent on a concrete implementation of the first (data generating) module. Instead it would be better to inject an interface containing the methods to get to that data and this can then be mocked out for testing the queries.

e.g.

If you have:

class Queries {

    int GetTheNumber() {
        var dataModule = new Submodule1();
        var data = dataModule.GetData();
        return ... run some query on data
    }
}

Instead prefer:

interface DataModule {
    Data GetData();
}


class Queries {

    IDataModule _dataModule;

    ctor(IDataModule dataModule) {
       _dataModule = dataModule;
    }

    int GetTheNumber() {
        var data = _dataModule.GetData();
        return ... run some query on data
    }
}

This removes the dependency from queries on your data source and allows you to set up easily repeatable unit tests for particular scenarios.

Paddy
  • 2,623
  • 16
  • 17
6

The other answers mention that ordering tests is bad (which is true most of the time), but there is one good reason for enforcing order on test execution: making sure your slow tests (i.e., integration tests) run after your faster tests (tests that don't rely on other outside resources). This ensures that you execute more tests faster, which can speed up the feedback loop.

Mike Holler
  • 169
  • 2
  • 2
    I'd be more inclined to investigate why a certain unit test is running slowly rather than enforce an order. Unit tests are supposed to be fast. – Robbie Dee Apr 04 '18 at 14:27
  • The OP says he cannot work with dummy data for some of these tests. That means a database hit of some sort, slowing down all tests (even some true unit tests that should run fast naturally). If he has other tests that do not require database hits, they will run an order of magnitude faster than anything requiring disk or network hits. – Mike Holler Apr 04 '18 at 15:46
  • 2
    You're both right, I think; Robbie is right that _unit tests_ should be small and fast and isolated from dependencies so order shouldn't matter and random-ordering often encourages better design by enforcing that independence; and Mike is right that running faster tests first is very, very good for _integration tests_. As in answers above, part of the problem is terminology of unit vs. integration tests. – WillC Apr 04 '18 at 23:39
  • @MikeHoller Then they're not unit tests. There really should be no confusion as to what [unit tests are](https://en.wikipedia.org/wiki/Unit_testing). – Robbie Dee Apr 05 '18 at 07:35
  • @RobbieDee I was merely using the terminology the OP was using. I understand that these are not true unit tests. If you want to fight about terminology, bring it up with OP. (hence why I clarified with "true unit tests" in my earlier comment") – Mike Holler Apr 05 '18 at 15:31
  • I've also edited my original post to remove "unit test" completely. – Mike Holler Apr 05 '18 at 15:33
  • @MikeHoller There are still better ways of doing this rather than ordering e.g. test attributes and various schedules for CI/CD environments, but yes, it is a misunderstanding on the part of the OP calling these unit tests. – Robbie Dee Apr 05 '18 at 15:48