32

In a company I used to work for, executives insisted that the code coverage with unit tests must be 99% or more. This resulted in writing more tests than code. It took us literally 3 days to write tests for a single class that took a day to implement.

As a result, however, I learnt a lot about TDD, testing tools, practices etc.

In the company I worked for afterwards, unit testing was an unknown thing. It was something someone maybe heard before. I struggled to introduce them to concept of unit testing, but without effect.

Now, as a self-employed, I wonder - how much time is really necessary to spend on unit testing? Being mostly iPhone/Android developer, which parts of the code should be covered in tests?

Arseni Mourzenko
  • 134,780
  • 31
  • 343
  • 513
Maggie
  • 816
  • 1
  • 8
  • 11
  • In my previous company, it was Java EE, stripes and struts framework. As I stated, now it is mostly iPhone and Android development. – Maggie Aug 08 '11 at 11:59
  • TDD mean that you write the tests before the code. It sounds like this was "after the fact" testing, which is something else. –  Aug 08 '11 at 12:09
  • managers didn't care for the technique, my team leader insisted on TDD. it ended up as a combination of both :) – Maggie Aug 08 '11 at 12:38
  • Maggie, you might find this article interesting: http://www.fastcompany.com/magazine/06/writestuff.html –  Aug 08 '11 at 13:32
  • Is there some metrcis to estimate time? Tools for JS/.NET code? – hellboy Sep 21 '15 at 12:44
  • It's normal to have more tests than product code. 100% coverage (which is a natural consequence of doing TDD) produces test code that is about double or triple the amount of product code. Of course, test code is simpler (more repetitive, less algorithmicaly complex) than product code. – Jonathan Hartley Jun 14 '16 at 17:17
  • Taking a reeeaally long time to write tests is a sign that the tests are overly difficult to write, which is usually caused by the design of the code-under-test not being very easy to test. TDD (done right) is one discipline that helps fix this: If you write the test first, it is unconstrained by the design of the code-under-test, so you are free to write the test in the simplest, most straightforward way imaginable. As a happy side effect, product code which makes these simple tests pass will tend to be better designed than code that makes your old hard-to-write tests pass. – Jonathan Hartley Jun 14 '16 at 17:20
  • possible duplicate of [How long should we generally spend writing unit tests for a new feature or bug fixing?](http://programmers.stackexchange.com/questions/84491/how-long-should-we-generally-spend-writing-unit-tests-for-a-new-feature-or-bug-f) – gnat Jun 20 '16 at 08:25
  • Possible duplicate of [Is it normal to spend as much, if not more, time writing tests than actual code?](http://softwareengineering.stackexchange.com/questions/299796/is-it-normal-to-spend-as-much-if-not-more-time-writing-tests-than-actual-code) – gnat Apr 01 '17 at 10:09

6 Answers6

19

The amount of unit testing that is needed depends on several factors:

  • Product Size (The larger the project, the greater the need to include at least some unit testing)
  • Required Quality Level (If you are quickly putting software together that needs to be out as quick as possible and some minor bugs are acceptable, then you might be forced to skip some testing like unit testing)
  • Type of Product (UI's can be unit tested, but sometimes it is easier to skip unit testing on heavy GUI sections of a project and instead test manually)
  • Your Coding Ability/History (What type of bugs do you normally create? Are they things that Unit Testing normally catches or things that another type of testing normally finds. Knowing this might push you to do more or less unit testing)
jzd
  • 4,166
  • 25
  • 28
  • I think he is asking for "statistical numbers". Which in my opinion is around 50% effort on average. – Pwnstar Oct 13 '22 at 07:05
12

Unit tests pay off at maintenance time. If you plan to have a long living application you will spend more time maintaining than you think you will now (if you have not yet tried this, you will be surprised how long a successful project may live)

What you want, is that if you accidentally change your functionality your tests break so you find these things as fast as possible. Customers strongly dislike when functionality changes unexpectedly.

  • 3
    And you look bad when you fix bug A only to introduce bugs B & C. – JeffO Aug 08 '11 at 12:17
  • depends if B and C are caught by tests or not –  Aug 08 '11 at 12:37
  • "you will spend more time maintaining than you think now" - indeed, especially taking into consideration that SW products typically outlive their planned lifetime, often by far. – Péter Török Aug 08 '11 at 12:40
  • You can't really "plan" to have a long living application since you can't tell in advance whether the project will be successful or not. You should always consider that when budgeting time for testing – Eran Galperin Aug 08 '11 at 13:10
  • Sorry, I was refering to customer's dislikes. – JeffO Aug 08 '11 at 13:26
  • 1
    @Eran, so you should plan for failure so you do not have to budget for tests? –  Aug 08 '11 at 13:29
  • @Thorbjorn Not plan for failure, but consider it as a possibility. There are also degrees of success - some projects might finish but will die off over the course of a few months, for non technical reasons (marketing, distribution, design, etc). Planning for long term maintenance before the project has been successful is something that should done wisely. – Eran Galperin Aug 10 '11 at 09:52
  • @Eran: I've seen a large project fail precisely *because* it didn't have unit tests -- testing was so expensive that changes to working code were discourage. New features were developed by copying large sections of code from existing features. Eventually the pile of WTF got so big that it cost more to maintain than it was worth. – kevin cline Jul 28 '12 at 17:36
11

In our product group we target 50-70% code coverage from unit tests and 90%+ coverage from unit tests and test automation combined. Typical time budgeted on writing unit tests is about 1 day for every feature that takes 3-4 days of heads down coding. But that can vary with a lot of factors.

99% code coverage is great. Unit tests are great. But 99% code coverage from unit testing alone? I find that hard to believe that you can get that much coverage from unit testing alone.

For the case where you spent 3 days writing tests for a class that otherwise took 1 day to implement. You didn't elaborate as to why it took this long or share any code. From speculation, I am guessing you weren't really writing a true unit test for your class, but were actually writing test automation. And there's actually nothing wrong with that - as long as your recognize the difference between the two different types of tests.

But you said the three days of test writing was only for a single class. Perhaps the class itself was not designed for unit testing. Does the class implement UI? Networking? File I/O? If so, you might have ended up writing more code to test the Java runtime than your business logic that interacts with the runtime.

TDD gets you thinking in terms of interfaces and interfaces to dependencies. That single class that implements UI, networking, and file/io for a single feature might be better served split into multiple classes - one for networking, one for file/io, and the UI broken into a model-viewer-controller design. Then you can implement appropriate tests for each with simple mock objects for the dependencies. Of course, all of this takes up more time. So rather than 1 day to code and 3 days to write tests, this type of design may require 3 days of coding and 1 day of writing tests. But the code will be far better maintainable and reusable.

selbie
  • 633
  • 1
  • 5
  • 12
  • 1
    Great answer. Most of it was too complicated for testing, you are right about that. And to split the class into several smaller units just to suit better unit tests seems a bit - overkill. I'd love to attach the code for the class and the acompanying tests, and I'd love to hear your opinion, but I'm not sure if I am allowed to. The code is not strictly mine, and I don't work for that company anymore, so I would like to avoid trouble. – Maggie Aug 08 '11 at 12:52
  • 2
    That is why you should write the tests first. Then you can find the natural places where the logic glues together. –  Aug 08 '11 at 13:31
4

If you're doing TDD, you'll be writing the tests at the same time as the code, switching between them every few minutes (or less.) There won't be any distinct time spent for tests. Using TDD makes it much easier to know that you have solid test coverage.

If you are Unit testing after the fact, you need to write the tests that will tell you if the code is broken due to changes. I wouldn't rely on coverage metrics here, but would go based on use cases and parameters to public interfaces. This will ultimately be based on your good taste and experience.

Sean McMillan
  • 5,075
  • 25
  • 26
  • Yep, for me this is true (in fact I usually cycle between working on tests and working on product code every few *seconds*, not minutes.) So I'm arguably working on tests 100% of the time. – Jonathan Hartley Jun 14 '16 at 17:14
3

If you will not spend time on tests, you will spend even more time to debug in live code.
So spend as many time as it needed to tests, to cover all (or 99% of code).

OZ_
  • 307
  • 1
  • 6
  • 4
    Why do people have such paranoia of debugging? If you know the tools, you rarely run into a problem that takes more than 5 minutes to debug. Problems that are hard to debug are mostly related to threading, but unit tests are useless there anyway. – Coder Aug 08 '11 at 13:34
  • 3
    @Coder, because people have experience and know that tests are much more useful thing than blind debugging. – OZ_ Aug 08 '11 at 13:36
  • 2
    Depends. Unit-tests are often counter-productive and add false sense of security. And in well structured code you won't get into "blind debugging" problem. You get a problem, you know where to look. If you don't, do a complete code/design review. Also see: http://programmers.stackexchange.com/questions/86636/tdd-vs-productivity/87671#87671 – Coder Aug 08 '11 at 13:43
  • @Coder I have experience both with "debugging" style and with tests. So don't tell me nothing about "debugging" style, I know it's past stage of evolution. – OZ_ Aug 08 '11 at 13:44
  • 4
    That's the problem with many TDD proponents, they have a hostile stance against debugging and design, while relying on tests to find all bugs. Then, in the production code, applications leak memory, handles, and crash on multi-cores, and they are like ?WTH?. TDD is just a tool, and depending on task at hand can be either very productive, or very counterproductive. Try to write unit-tests for all the hard cases in the linked post, and you'll never ship the product. – Coder Aug 08 '11 at 14:04
  • @Coder try to write at least one test. – OZ_ Aug 08 '11 at 14:06
  • 1
    "If you know the tools, you rarely run into a problem that takes more than 5 minutes to debug." - @Coder I wonder what type of applications you're looking at. – Kirk Broadhurst Jun 05 '15 at 15:09
  • @coder I find dealing with bugs to be counter productive. When I’ve got a good unit test suite, I’ve few to zero bugs per release. I’m sorry you’ve only experienced “bad” unit testing. If we swapped our experiences, we’d be arguing on opposite sides of this issue than we are now. Without a mind meld the only thing I can suggest is to find a “Vanilla Pop” to pair program with, someone who succeeds at TDD otherwise you’ll keep experiencing bad examples: http://agilenoir.biz/en/agilethoughts/why-developers-dont-tdd-a-radio-drama/. – Lance Kind Aug 15 '19 at 05:34
  • People like @Coder do not understand unit tests/they have never written in their life. – RamPrakash Feb 22 '22 at 15:36
2

As others already noted, it depends largely on the type of software. The 3:1 test/development time ratio you mention may be a bit too much for average projects, but may be perfectly OK for mission critical apps and may be even too little for a life critical system.

99+% unit test coverage is similarly maybe too much to expect in the case of an average app, but too little for a life critical project.

In my experience, considering that a significant part of production code is error handling code, a coverage of 80-90% would be sufficient for most apps, and this could require roughly about the same amount of time spent writing unit tests as production code. (Then again, if one is earnestly working in TDD fashion, the two are completely intertwined to practically become one single task, so one can only guesstimate the actual ratio.)

Péter Török
  • 46,427
  • 16
  • 160
  • 185
  • do you have experience in mobile apps? what would be an acceptable test/development ratio for a simple mobile application? – Maggie Aug 08 '11 at 12:41
  • @Maggie, unfortunately not. (So if I needed to write one, I would probably spend more than my usual time on unit testing :-) – Péter Török Aug 08 '11 at 12:44