How should C++ Unit Test code be organized for maximum Unit Test efficiency?

Question

This question is not about Unit Testing Frameworks.
This question is not about writing Unit Tests.
This question is about where to put the UT code written and how/when/where to compile and run it.

In Working Effectively with Legacy Code, Michael Feathers asserts that

good unit tests ... run fast

and that

A unit test that takes 1/10th of a second to run is a slow unit test.

I think these definitions do make sense. _{I also think that they imply that you have to keep a set of Unit Tests and a set of Those Code Tests That Take Longer separately, but I guess that's the price you pay for only calling something a Unit Test if it runs (very) fast.}

Obviously the problem in C++ is that to "run" your Unit Test(s), you have to:

Edit your code (production or Unit Test, depending on which "cycle" you're in)
Compile
Link
Start Unit Test Executable(s)

Edit _{(after weird close vote)}: Before going into the details, I'll try summarize the point here:

How can C++ Unit Test code be effectively organized, so that it's both efficient to edit the (test) code and to run the test code?

The first problem then is to decide where to put the Unit Test code so that:

it's "natural" to edit and view it in combination with the associated production code.
it's easy/quick to start the compilation cycle for the unit your currently changing

The second, related, problem then is what to compile so that the feedback is instantaneous.

Extreme options:

Each Unit-Test-Test-Unit lives in a separate cpp file and this cpp file is compiled+linked separately (together with the source code unit file it tests) to a single executable which then runs this one Unit Test.
- (+) This minimizes startup (compile+link!) time for the single Test Unit.
- (+) The test runs super fast, because it only tests one unit.
- (-) Executing the whole suite will need to start a bazillion of processes. Can be a problem to manage.
- (-) The overhead of process starts will become visible
The other side would be to have -- still -- one cpp file per test, but all the test cpp files (together with the code they test!) are linked into one executable (per module / per project / pick your choice).
- (+) Compile time still would be OK, as only changed code will compile.
- (+) Executing the whole suite is easy, as there's just one exe to run.
- (-) The suite will take ages to link, as each recompilation of any object will trigger a re-link.
- (-)(?) The suit will take longer to run, although if all Unit Tests are fast, the time should be OK.

So, how are real world C++ Unit Tests handled? If I only run that stuff nightly/hourly, the second part doesn't really matter, but the first part, namely how to "couple" the UT code to the production code, so that it's "natural" for developers to keep both in focus always matters I think. (And if developers have the UT code in focus, they'll want to run it, which brings us back to part two.)

Real world stories and experience appreciated!

_Notes:

_{This question intentionally leaves unspecified platform and make/project system.}
_{Questions Tagged UT & C++ is a great place to start, but unfortunately too many questions, and especially answers, are too heavily focused on the details or on specific frameworks.}
_{A while ago, I answered a similar question on structure for boost unit tests. I find this structure to be lacking for "real", fast Unit Tests. And I find the other question too narrow, hence this new question.}

A further complication is introduced by the C++ idiom of moving as many errors as possible to compile time - a good suite of unit tests often needs to be able to test that certain usage fails to compile. — , Aug 31 '11 at 11:41
@Closers: Could you please provide the arguments that led you to close this question? — Luc Touraille, Aug 31 '11 at 11:55
I don't quite see why this had to be closed. `:-(` Where are you suppose to look for answers to such questions if not in this forum? — , Aug 31 '11 at 13:50
It was closed because StackOverflow isn't the place for extended discussions. But I know of somewhere it is. — Ripped Off, Aug 31 '11 at 15:03
@Joe: Why do I need a unit test to find if something will compile when the compiler will tell me that anyway? — David Thornley, Aug 31 '11 at 15:21
@Martin: "This question is not a good fit to our Q&A format. We expect answers to generally involve facts, references, or specific expertise; this question will likely solicit opinion, debate, arguments, polling, or extended discussion." This means it is **not** on topic at StackOverflow. Its a better fit here (*crosses fingers*), as PSE is more appropriate for discussions about programming. — Ripped Off, Aug 31 '11 at 16:34
@David: Because you want to make sure that it **does not compile**. Quick example would be a pipeline object that accepts an input and and produces output `Pipeline.connect(Pipeline)` should compile whereas `Pipeline.connect(Pipeline)` should not compile: The output type of the first stage is incompatible with the input type of the second stage. — sebastiangeiger, Sep 01 '11 at 06:56
I find this question does ask something similar: http://stackoverflow.com/questions/5484335/tdd-and-housekeeping-overhead-with-c — Martin Ba, Sep 05 '11 at 11:43
ANother "duplicate": http://stackoverflow.com/questions/1156816/how-to-go-about-organizing-a-unit-test-harness-for-legacy-visual-c-code — Martin Ba, Sep 05 '11 at 11:52
And another question asking the right thing in the title and then just discussing the framework: http://stackoverflow.com/questions/5307627/test-driven-development-with-c — Martin Ba, Sep 05 '11 at 12:27
I am trying to find this _exact_ same answer, and I came across your question. Did you ever find good information on this subject? — Dan Nestor, Nov 16 '14 at 07:44
@DanNestor - Sorry to say that I never found anything good. Too much Dogma, Simple-Examplism and focus on non-C++ languages in the UT/TDD world, so you'll have to use your own guidance. — Martin Ba, Nov 17 '14 at 10:36

score 7 · Answer 1 · answered Aug 31 '11 at 11:38

We have all unit tests (for a module) in one executable. The tests are put into groups. I can execute a single test (or some tests) or a group of tests by specifying a (test/group) name on the command line of the test runner. The build system can run group "Build", the test department can run "All". The developer can put some tests into a group like "BUG1234" with 1234 being the issue tracker number of the case he's working on.

score 7 · Answer 2 · answered Aug 31 '11 at 16:43

First, I disagree with "1) Edit your (production) code and your Unit Test". You should modify only one at a time, otherwise if the result changes you won't know which one caused it.

I like to put unit tests in a directory tree that shadows the main tree. If I have /sources/componentA/alpha/foo.cc and /objects/componentA/beta/foo.o, then I want something like /UTest_sources/componentA/alpha/test_foo.cc and /UTest_objects/componentA/beta/test_foo.o. I use the same shadow tree for stub/mock objects and whatever other sources the tests need. There will be some edge cases, but this scheme simplifies things a lot. A good editor macro can pull up the test source alongside the subject source effortlessly. A good build system (e.g. GNUMake) can compile both and run the test with one command (e.g. make test_foo), and can manage a bazillion such processes -- just the ones whose sources have changed since they were last tested -- quite easily (I have never found the overhead of starting these processes to be a problem, it's O(N)).

In the same framework, you can have larger-scale tests (no longer unit tests) which link many objects together and run many tests. The trick is to sort these tests by how long they take to build/run, and work them into your daily schedule accordingly. Run the one-second-or-less test whenever you feel like it; start the ten-second test and stretch; five-minute test and take a break; half-hour test and go for lunch; six-hour test and go home. If you find that you're wasting a lot of time e.g. relinking a huge test after changing only one small file, you're doing it wrong-- even if the linking were instantaneous, you'd still be running a long test when it wasn't called for.

ad (1) - yes that was phrased sloppily – Martin Ba Sep 01 '11 at 06:47 — Martin Ba, Sep 01 '11 at 06:47

How should C++ Unit Test code be organized for maximum Unit Test efficiency?

2 Answers2

Linked