5

Is the execution time of a unit test useful for anything? Is a unit test an appropriate place for any sort of code profiling? Why or why not?

Sample Use Case:

  • Every unit test lists name, pass/fail, and execution time.
  • Time is also calculated per unit test class and total test suite execution time.

How could I use this info?

Curious Query
  • 53
  • 1
  • 4

6 Answers6

10

One of the most important rules about unit tests is they should run fast.

Developers should be able to run the whole suite of unit tests in seconds, and definitely not in minutes and minutes. Developers should be able to quickly run them after changing the code in anyway. If it takes too long, they won't bother running them and you lose one of the main benefits of the tests. We currently have about 800 tests that run in around 30 seconds.

So if any tests do start taking too long, you can see which test is taking too long and do something about it.

Our unit test framework tells us how long it took for each test to run. Most of them take 0- 5ms (if I remember correctly), we have one or two that take about 3 seconds.

Hugo
  • 3,669
  • 2
  • 25
  • 40
  • 2
    Do your unit tests need to get data from the database? I find that usually slows them down a little bit beyond what you would consider acceptable. – FrustratedWithFormsDesigner Sep 15 '11 at 21:40
  • 1
    Also consider the use case where a developer uses the tests to debug a failure in the tests. They may have to run them repeatedly to fix their problem. Unit tests should make debugging problems easier not more cumbersome. – Doug T. Sep 15 '11 at 21:47
  • 2
    @FrustratedWithFormsDesigner We're lucky in that we don't use any databases on this project but I've read they are a kind of special case and can be treated as an external dependency, so you test the interactions with the database and not the data itself, especially as tests should be repeatable and not depend on any previous state. Here's some tips: http://osherove.com/blog/2004/6/19/simplified-database-unit-testing-using-enterprise-services.html – Hugo Sep 15 '11 at 21:50
  • @Doug usually a failure affects one or several tests, not the entire suite. you shouldn't need to run the entire suite to debug a specific test – Eran Galperin Sep 15 '11 at 22:31
  • @Eran I guess it depends how smart your testing is and if you can easily rearrange the order they get executed in. – Doug T. Sep 16 '11 at 00:26
  • @Doug usually no rearrangement would be necessary, you just run the tests that failed instead of the entire suite while debugging – Eran Galperin Sep 16 '11 at 01:06
  • 1
    With the exception of unit tests which involve the database, if an individual unit test takes more than a second then your unit test may be doing too much. The vast majority of tests should run in the order of milliseconds. – RichardM Sep 16 '11 at 19:02
1

Is the execution time of a unit test useful for anything?

Yes.

Is a unit test an appropriate place for any sort of code profiling?

Yes.

Why?

Because you need facts and measurements. If a test suite is slow, it's important to know where so you can focus -- separately -- on why.

Code profiling is only done when code is proven to be slow.

Unit tests are an easy, cheap, obvious, painless way to locate (1) slow code and (2) sudden changes in performance.

S.Lott
  • 45,264
  • 6
  • 90
  • 154
1

Consider the development of your non-functional requirements:

Functional Requirement: Once the filters have been selected, the report should be generated. Non-Functional: The report should generate in 2 seconds.

If your test's execution time is > 2 seconds, you could even fail the test as a "usability" test.

Babak Naffas
  • 496
  • 4
  • 5
0

It could be useful if you are testing a procedure that is known to be slow. You could write unit tests that test the procedure with different sized inputs and you could monitor the performance of just that piece of code. Of course, whether or not this is applicable in your project really depends on your project. I use when it is easy to determine which function has a bottle-neck, and we want to isolate and test just that function.

FrustratedWithFormsDesigner
  • 46,105
  • 7
  • 126
  • 176
0

I can only think of timing out tests if something like an endless loop could occur in the tested code.

If you run unit tests as a part of post-commit checks or a build, the machine handling it may have all kinds of loads that will make precisely timed tests useless. Run such performance tests (not unit tests) in a controlled environment.

It can be useful to collect stats of test execution times to predict running times of large test suites (integration servers, like TeamCity, do this).

9000
  • 24,162
  • 4
  • 51
  • 79
0

It could be valuable if you were to keep track of average performance for a given test across all test runs. A sudden increase in execution time might signal the introduction of a new bottleneck into the code.

Also in general it is desirable to keep the execution for an entire set of unit tests as fast as possible in order to encourage running the tests frequently. Knowing the execution of individual tests will help identify which tests are taking the most time across all tests.

Ken Liu
  • 327
  • 1
  • 9