15

Our organization is considering integrating unit testing into our software development workflow. I've heard lots of anecdotal stories about how it encourages better, easy to maintain, and well-planned code. As a programmer, I understand the benefits.

However, in order to build consensus around using unit testing in projects that might affect customers, I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.

Does anyone in the community have any historical data or experience they would be willing to share as quantitative evidence in my case for unit testing?

Adam
  • 271
  • 1
  • 4
  • 1
    I'm not 100% sure you will find what you are looking for. The act of performing unit testing doesn't necessarily save any developer time, as it takes time to write the tests and then fix defects found. However, what you will find is data relating to the cost savings of finding and fixing defects early and improved product quality (and customer satisfaction). Unit testing is just one step a series of processes designed to improve product quality. I think you should refocus your efforts - sell product quality, with unit testing as a single tool among many to begin to move toward that goal. – Thomas Owens Sep 07 '11 at 23:40
  • 1
    "I would like to demonstrate its value quantitatively, in terms of saving hours of developer time.". And then, everyone will simply dispute your numbers. This is always doomed from the start. Are you open to suggestions for techniques that work? Or are you stuck on arguing about the numbers instead of doing something? – S.Lott Sep 08 '11 at 01:04
  • 1
    All of these are duplicates: http://programmers.stackexchange.com/search?q=justify+unit+testing. What's wrong with all of the other questions just like yours? – S.Lott Sep 08 '11 at 01:06
  • Historical data? What about 3 days of searching to find a hard to locate bug that was introduced by a simple "no-risk change"? A bug that would have been caught by a very simple unit test? A unit test that would have taken at most 5 minutes to code? – Marjan Venema Sep 08 '11 at 06:51
  • How do you quantify leaving out features because you're afraid to break the application? – JeffO Sep 08 '11 at 12:10

4 Answers4

11

Unit testing is cheap.

Each time you run a unit test, you didn't have to pay a tester to perform that test manually, or perform the test yourself, by hand. A computer can perform the same tests over and over tirelessy, with perfect fidelity (executing the test in exactly the same way every time).

Each time an automated test catches a bug you missed during refactoring, you save money by preventing QA people from having to report it, and untold money and reputation capital by preventing that bug from going out the door to a customer.

Automated testing, especially unit testing, saves money because it catches bugs early. The later a bug is found, the more expensive it is (by orders of magnitude):

enter image description here

Community
  • 1
Robert Harvey
  • 198,589
  • 55
  • 464
  • 673
  • Thanks, Robert. I'm certainly in agreement, but it's going to be hard to sell this argument without some sort of quantitative evidence to back it up. – Adam Sep 07 '11 at 23:15
  • Where's this chart from? – Aaronaught Sep 07 '11 at 23:21
  • 1
    @Aaronaught: http://jonkruger.com/blog/2008/11/20/the-relative-cost-of-fixing-defects/ – Robert Harvey Sep 07 '11 at 23:23
  • 2
    Huh... doesn't seem to be based on any actual data. Not that I disagree with the general idea but it'd be nice to see something a bit more material. – Aaronaught Sep 07 '11 at 23:26
  • @Aaronaught: It is from a Google image search for "Cost of Unit Testing." There are a bazillion such graphs; they all look the same. The original was probably from "The Mythical Man Month," but I don't have a copy in front of me right now to verify that. – Robert Harvey Sep 07 '11 at 23:29
  • In any case, there are plenty of good programming practices that cannot be "proven" in dollars and cents. As an organization, you have to decide whether you want to do it the professional way, or do it the schleppy way and wind up fighting fires for the remainder of the product's life-cycle. I've been in both situations, and I know which one is better. – Robert Harvey Sep 07 '11 at 23:31
  • It'd still be nice to have a source on that data which describes, for example, how measurements were taken and what metrics were used for "cost." Otherwise the chart is just pulled out of thin air and is as good as any made-up statistic. In any case, I think using it as the be-all-end-all is unwise, as the cost of fixing an issue in production varies according to what your production environment actually is, and what your customers are expecting. For the record, I am 100% for unit testing, but I don't think the chart really helps to answer the original question. – Mitch Lindgren Sep 07 '11 at 23:32
  • Barry Boehm is apparently the authority on software defects as a function of the delay in finding them. He is the inventor of the Spiral Model, and has done a number of studies on defect cost as a function of time. See http://www.google.com/url?sa=t&source=web&cd=4&ved=0CDEQFjAD&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.103.4004%26rep%3Drep1%26type%3Dpdf&rct=j&q=Barry%20Boehm%20cost%20of%20software%20defects&ei=tgBoTr8atIqwAoDC0JAO&usg=AFQjCNGszgYGu7QzWZ8fyr3j4sHPYC2SCA&sig2=G1oU9MS3M_1yWQoGhlGP9g – Robert Harvey Sep 07 '11 at 23:40
  • @Robert Harvey You are correct - this data is from Boehm. However, I'm not sure which work of his it's in. It's not Software Engineering Economics - I looked. I loaned McConnell's Rapid Development to a process engineer at work recently, but this study and two similar ones are cited in that book - if someone has a copy handy, they could look it up and provide citations. – Thomas Owens Sep 07 '11 at 23:43
10

Here is a chart from my experiences with and without unit testing. How many of the problems on the right-hand column apply to your project?

                  With Unit Tests               Without Unit Tests
----------------- ----------------------------- -----------------------------
Development       Code/Test/Refactor            Cut and Paste/Hack 
Process           or Test/Code/Refactor         to minimize code change 
                                                and avoid complete retest

Module Coupling   Limited by continuous         Tangled due to need to
                  refactoring                   add features with minimal
                                                code change

Program Design    Evolving through Refactoring  Frozen due to risk of change

Velocity          Increasing as higher-level    Decreasing as code volume and
over Time         functions are factored out    coupling increase 

Bugs detected     Mostly by developer           Mostly by QA or Customer

Cost of bugs      Lower                         Higher

You should be able to point to specific instances where the code volume was increased unnecessarily, or the module coupling was increased to minimize the need for retest. However, if they don't believe they have a problem, and are not interested in improvement, there may be little you can do. Even when continued maintenance becomes uneconomical, they may believe that is simply the normal end of all software projects.

I have been successful in improving testing only when management knew they should be testing, but weren't sure how to do it economically. When an organization is uninterested in process improvement, great patience is needed to effect change. How patient are you?

kevin cline
  • 33,608
  • 3
  • 71
  • 142
5

I once did a statistic over a number of versions and various modules of a large system, measuring basically the number of unit test LOC per functional code LOC, plotted against the number of bugs found during manual testing per functional LOC. The result looked something like this:

More Bugs
^
|
| +
|+    +  +     +
|   +    +          +   +       
|  +        +    +     +    + + 
|+     + +         +      +  +  +   +
|    +        +        +    +    + +
---------------------------------------> More Unit tests

A linear interpolation said that there were nearly 50% less bugs in the code with the most unit tests, compared to that with none. But as you see, there is a large variance, most likely related to complexity of the modules, experience and competence of the developers and maturity of the code.

Michael Borgwardt
  • 51,037
  • 13
  • 124
  • 176
3

Unit testing is invaluable.

I have many experiences with this.

When we first didn't adopt Unit testing, we spent the majority of the time solving bugs and dealing with mad customers because things didn't work as expected. Debugging code when you have no idea of where the bug may be is a hassle, especially because you first have to reproduce it in order to solve it.

When we decided it was time to start working with UT, it all changed. We got no more calls from mad customers at all. It may sound excessive but it went exactly like this.

With UT you find the bugs before your customers or QA. This is the big deal.

Jose Faeti
  • 2,815
  • 2
  • 23
  • 30