15

Do you know of any xUnit framework that allows to run tests in parallel, to make use of multiple cores in today's machine?

If none (or so few) of them does it, maybe there is a reason... Is it that tests are usually so quick that people simply don't feel the need to paralellize them?

Is there something deeper that precludes distributing (at least some of) the tests over multiple threads?

gnat
  • 21,442
  • 29
  • 112
  • 288
Xavier Nodet
  • 3,684
  • 2
  • 26
  • 33
  • Unit testing is definitely slow. Even if each test by itself is fast, they'd cumulate as people literally have millions of test cases. – Pacerier Aug 31 '15 at 08:51

6 Answers6

10

To answer the second part of your question: Is there something deeper that precludes distributing (at least some of) the tests over multiple threads?

A great deal of code only works when run single threaded. It's trivial to accidentally produce resource contention and deadlocks when writing programs on the assumption that they will be run single threaded. And this works fine because most programs do actually run single threaded. Parallelism is gained by running multiple copies, or different programs at the same time (web scripts being one common example - lots of users accessing a single page means lots of copies of the scripts for that page running at the same time).

Imagine a simple "log to file" class. When you create an instance it opens the file for writing, when you free the instance it closes the file. So, the first test creates an instance and starts running a test. The second test does the same thing in a second thread. And fails, because the second instance can't get write access to the file. But if run one at a time all the tests would pass.

All of this can be coded around, and the simple example could be tweaked to work. But doing that is probably unnecessary for the original program. Having to write thread-safe code just so you can run unit tests is unreasonable for many people. So multi-threaded unit tests should remain an optional extra.

6

NUnit 2.5 bundled pNUnit which allows the running of tests in parallel.

This release includes pNUnit, an extended NUnit runner for distributed parallel tests. The pNUnit program was developed at Codice Software for use in testing the Plastic SCM and has been contributed to NUnit. For more info about using pNUnit see the pNUnit site.

The JUnit side has parallel-junit as well as amino.

Aaron McIver
  • 3,262
  • 16
  • 19
4

If tests need to setup and query a database, tests running in parallel would interfere with each other unless unless there's a separate database for each test running in parallel.

Clint Miller
  • 181
  • 3
  • That's not for the testing platform (xUnit) to care about; that's an implementation detail. – Aaron McIver Mar 08 '11 at 20:57
  • And not all tests written in a unit testing framework are unit tests, just like the one that accesses the database is not really a unit test, more like an integration test. – c_maker Mar 09 '11 at 01:57
  • Just because a test touches a database doesn't mean it's an integration test. A method that runs partially in C# and partially in a database sproc for example is still conceptually a unit test, as long as no pre-configuration is expected (i.e. the data schema is present, but no data is present). Such tests may create data for an individual run, but should reset to a blank state when finished). This is probably a controversial opinion, but such tests cannot be considered integration tests, because they're explicitly zero-config, zero-deploy tests that test small code units. – Triynko Dec 13 '19 at 16:26
2

While JUnit per se may not allow it (I am not intimately familiar with its latest versions though), Maven with its Surefire plugin does have an option to run tests in parallel. I haven't tried it yet though.

I am not strongly pressed to investigate this option, as we have only a bit over a thousand tests and they run fast enough. However, I do know that some of the test fixtures have implicit dependencies between (we have found some such dependencies when some tests broke unexpectedly in the past), so there is a risk in that parallelizing the tests will make some of them fail unpredictably. You may say this is fine as it makes the problem explicit. However, we are dealing with a legacy system, and we have many more important issues to deal with - time is a scarce resource (as usual).

Péter Török
  • 46,427
  • 16
  • 160
  • 185
0

MBUnit is able to run tests in parallel simply by specifying some assembly level attributes.

[assembly: DegreeOfParallelism(6)]
[assembly: Parallelizable(TestScope.All)]

I've been using that project to run selenium tests in parallel quite successfully for some time. Unfortunately the project is not very much alive anymore.

xUnit 2.0 should also support parallel unit testsing but i haven't tried it out yet.

Ivo Grootjes
  • 101
  • 1
0

Multi-threaded coding is not trivial. Even when done by people who know what they are doing, timing dependent bugs can occur. They are difficult to fix. Having dealt with the one a few thousand case type bugs multi-treading can produce, I would prefer not to have them in my testing framework. The first fix I got appeared to work, but on further testing it was found that it had just become a one in tens of thousands bug.

The techniques to do multi-threading on multi processors are getting better with the advent of multi-processor PCs. However, it will take time before they are in wide use.

Some test suites have dependencies between tests that do not need to be explicitly stated when the tests are run in a single stream. However, on a multi-steam engine, they would need to be explicitly stated. (Where such dependencies should exist is a different question.)

From another standpoint, some things just don't need to be run in parallel. It the process runs adequately fast, it may better to focus efforts on things other than implementing multi-threading.

BillThor
  • 6,232
  • 17
  • 17