Research on software defects

Question

There is a chapter in "Making Software: What really works and why we believe it" By Andy Oram and Greg Wilson about software defects, and what metrics can be used to predict them.

To summarize (from what I can remember), they explained that they used a C codebase that was open source and had a published defect tracking history. They used a variety of well known metrics to see what was best at predicting the presence of defects. The first metric that they started with was lines of code (minus comments), which showed a correlation to defects (.i.e. as LOC increase so do defects). They did the same for a variety of other metrics (don't remember what off the top of my head) and ultimately concluded that more complex metrics were not significantly better at predicting defects than simple LOC count.

It would be easy to infer from this that choosing a less verbose language (a dynamic language?) will result in less lines of code and thus less defects. But the research in "Making Software" did not discuss the effect of language choice on defects, or on the class of defects. For example, perhaps a java program can be rewritten in clojure (or scala, or groovy, or...) resulting in more than 10x LOC savings. And you might infer 10x less defects because of that.

But is it possible that the concise language, while less verbose, is more prone to programmer errors (relative to the more verbose language)? Or, that defects written in the less verbose language are 10x harder to find and fix? The research in "Making Software" was a fine start, but it left me wanting more. Is there anything published on this topic?

Answer 1

The only interesting paper that I found was Comparing Observed Bug and Productivity Rates for Java and C++ (ACM membership required), published in 1999. The author used a modified Personal Software Process to gather productivity and defect information. The author found that a C++ program contained 2-3 times more defects than a Java program, C++ generated 15-30% more defects per line, and Java was 30-200% more productive in lines of code produced over time. Since only a single programmer was used, and this individual was experienced in C++, but only learning Java, it's not definitive and an experienced Java programmer would be more proficient. However, this study was notable because it provides a methodology for measuring productivity of a programming language.

I would like to point out a few possible issues though...

Lines of code is a shaky measurement, at best. It's easy to count using various tools, and these tools often have configurable rules so that you can get the optimal counts for your organization's coding style. However, a line of Python is not the same as a line of C - you can do a lot more in a line of Python than a line of C.

The metric they discuss in Making Software, defects/SLOC, is called defect density. It's a fairly common metric, but because it's based on SLOC, it suffers from the same problems as any SLOC-based metrics. You need to have a standardized method of counting and use this consistently. You also can't easily compare defect density across programming languages with any value.

Also, simply comparing programming languages isn't necessarily a fair comparison. You also need to compare available tools and resources. For example, using static analysis and code reviews are known to reduce defects. Different languages have varying support for static analysis, unit testing, and you might not have access to people proficient enough in the language to conduct a code review.

Ultimately, I think that talking about verbosity versus defects is the wrong thing to do. It's better to look at techniques and processes to reduce defects to an acceptable level for your product. That includes your choice of programming language, with regards to productivity given your team and supporting tools.

Answer 2

There's interesting stuff out there. These are my favorites, as they include empirical data, and they don't always match Accepted Dogma about bugs:

• Andy Ozment's Milk or Wine - It contradicts the Management Dogma that Every Piece of Software has an infinite number of bugs (i.e. fractal bugginess).
• Famed Cambridge security researcher Ross Anderson wrote Open and Closed Systems are Equivalent
• Eric Rescorla, Is Finding Security Holes a Good Idea?
• Geoffrey Phipps, Comparing Observed Bug and Productivity Rates for Java and C++
• David Atkins, Thomas Ball, Todd Graves, Audris Mockus, Using Version Control Data to Evaluate the Impact of Software Tools
• Todd Graves, Alan Karr, J.S. Marron, Harvey Siy, Predicting Fault Incidence Using Software Change History

Answer 3

Some languages have shorter programs because the provide pieces of implementation for the programmer. If the coder doesn't have to implement something, they can't very well introduce defects into it. If I were to implement a hash-map in C, I'd probably make mistakes and defects. In python, I wouldn't because I already have a hashmap.

Other languages are shorter because they allow the programmer to write less text to get the same program. Compare C++:

for(int x = 0;x < 10;x++)
{
    std::cout << x << std::endl;
}

and Python:

for x in range(10):
    print x

You can see even on such a simple example that python takes way less characters then C++. But its not about the amount of characters you have to type, but rather the number of things you could have gotten wrong. The python simply has less possible mistakes to make then the C++ version.

All else being equal, I think more conciseness leads to less bugs because they are simply less things to get wrong. But all else may not be equal. The verboseness could be useful for other reasons and thus pay for itself.

Answer 4

It would be easy to infer from this that choosing a less verbose language (a dynamic language?) will result in less lines of code and thus less defects. But the research in "Making Software" did not discuss the effect of language choice on defects, or on the class of defects. For example, perhaps a java program can be rewritten in clojure (or scala, or groovy, or...) resulting in more than 10x LOC savings. And you might infer 10x less defects because of that.

no no no :-) There is no way to infer from the fact that more lines of C code correlate with more defects that a language that is more expressive will have less defects for the same problem. The first is a no brainer, for a given developer, given langauge, more lines of code will have more bugs.

But is it possible that the concise language, while less verbose, is more prone to programmer errors (relative to the more verbose language)? Or, that defects written in the less verbose language are 10x harder to find and fix? The research in "Making Software" was a fine start, but it left me wanting more. Is there anything published on this topic?

It's definitely possible. Maybe it's less readable? Maybe it has functions names and operators that are hard to remember and easy to confuse? Take C and rename printf() to p(), sin() to s(), cos to c(). Shorter code, right?

Studying this sort of stuff is extremely tricky. Think about all the variables you need to control for. Experience, size of project, application area, lanuage, enviornment, management. Maybe this works for numerical analysis but doesn't work for web development? Maybe you can show it for students but not for senior developers? Perhaps it's true for a developer sitting in an open space but not in an office? Maybe it's true for Americans but not for Japanese? Maybe it's true for open source but not for commercial developers?

There are numerous studies on the productivity of programming languages, e.g.:

Haskell vs. Ada vs. C++ vs. awk vs.... an experiment in software prototyping productivity

Back in the day when people were pushing for Ada there was a few studies showing how better it is in terms of defect rates and productivity. Ada is an example of a language designed towards the goal of eliminating defects.

This question reminds me of this blog post Hume, Causation & Science

My suggestion is to find out what works for you given your strenghts and application area. Don't look for studies to tell you what to do, a study can give you some good ideas but you need to understand the limitations of these studies.

Here's a personal observation: You can train yourself to lower your defect rate. I found that doing TopCoder competitions where you get immediate feedback for defects will train you pretty quickly to eliminate some common defects and think about corner cases (which is where a lot of defects are found). This sort of stuff will make an interesting study...

In terms of predicting defect rate I read a paper a while back on correlation between static analysis results and defects: The Effectiveness of Automated Static Analysis Tools for Fault Detection and Refactoring Prediction (tl;dr not exactly correlated but bad code tends to have more issues though the defects may not be identified).

Answer 5

I haven't read Making Software, so I can't comment on the conclusions they have drawn in their book. Plotting Errors vs Lines Of Code can be misleading at best, because lines of code doesn't necessarily imply the degree of complexity of the logic contained within. Lines of Code is often more useful when determining time vs effort/cost, as a management tool at very roughly gauging a team's capacity for work, and even then it can be flawed.

When using metrics to measure code quality and the likelihood of error, I've found the following to be very useful:

1. Code Coverage
2. Instruction Path Length
3. Cyclomatic Complexity

The use of a good profiler can be of great assistance in providing useful measurements to help not only optimize code, but to get a feel for the likelihood that a significant number of bugs are likely to be inherent in the system. FWIW, I find the Redgate ANT profile to be one of the better .NET profilers.

Now back to your question. Software Metrics have been discussed since at least the early 70's to the best of my knowledge, although If I had to guess I'd say it's likely the topic was of interest even earlier than that.

...is it possible that the concise language, while less verbose, is more prone to programmer errors (relative to the more verbose language)? Or, that defects written in the less verbose language are 10x harder to find and fix?

I would say that not only is it entirely possible, it is also very likely, both for the brevity of the language, and also due to its inherent complexity. For example, Assembler is about as brief as a language can get syntactically, and requires very careful attention to detail in order to minimize bugs, yet in being more verbose you are given many more opportunities to confirm the state by setting breakpoints at virtually every state change. That said, it's usually harder to locate problems by eye alone in assembler as compared to a higher level language with well factored code.

If on the other hand by reduced verbosity you mean a high-level language which handles a number of common operations through a single command, then you need to assume that the command as a shortcut is itself without error, and that the purpose to which you've applied the command is also without error. Making the command syntax less verbose can also have the effect of introducing an element of obfuscation to the code depending on the manner in which the syntax is used.

Is there anything published on this topic?

The Wikipedia article on Software Metrics has some links to some papers, and so too do each of the metrics that I listed earlier. As to how well peer reviewed all of that information is I am not entirely certain, but yes, there appears to be a lot of published information out there on this very subject, which is quite likely the meat and potatoes of many an advanced CompSci thesis. ;)

Answer 6

It would be easy to infer from this that choosing a less verbose language (a dynamic language?) will result in less lines of code and thus less defects.

Easy and as close to correct as you can get with that dataset. Lines of code are where the errors must be. Less code is less places to harbor errors.

But the research in "Making Software" did not discuss the effect of language choice on defects, or on the class of defects.

Actually. It did. It gave you the facts. Fewer lines of code means fewer errors.

For example, perhaps a java program can be rewritten in clojure (or scala, or groovy, or...) resulting in more than 10x LOC savings. And you might infer 10x less defects because of that.

That would not be correct. They said it correlated. They did not say that the relationship is linear.

There are errors outside the code. Specification errors. Testing errors. There are non-code defects as well as code defects.

But is it possible that the concise language, while less verbose, is more prone to programmer errors (relative to the more verbose language)?

No. Concise languages have fewer errors. It really is that simple.

The problem is that super-concise languages can be difficult to write. So the cost goes up even though the number of errors goes down. Maintenance can be very difficult. Languages like APL or I are often write-once languages. The code is so dense that it's hard to make a change; it's often easier to begin again from the beginning and write all new programs.

The other problem is that non-code errors start to dominate after a while. A good, concise language -- particularly a DSL that's really a good fit to the problem domain -- reveals that there are numerous other sources of errors.

---

To avoid a lot of nuanced (and pointless) discussion.

• The study quoted is obviously incomplete.

• If you have a more nuanced explanation, please conduct your own study and publish your own results so that we can discuss them.

• Please focus on question, if possible. If you want to introduce new evidence, either open a new question or comment on the question itself to introduce new evidence.

Answer 7

Edit: I've rearranged my answer because I want to emphasize a different point.

Verbosity can sometimes help to make defects easier to detect and fix. Consider the quadratic formula (positive root only for simplicity's sake):

float quad(float a, float b, float c)
{
   return (-b + sqrt(b*b - 4*a*c)) / (2*a);
}

And this equivalent implementation:

float quad(float a, float b, float c)
{
    float discriminant = b*b - 4*a*c;
    float numerator = -b + sqrt(discriminant);
    float denominator = 2*a;
    float root = numerator/denominator;
    return root;
}

The latter, while somewhat extreme, is a common style that makes stepping in the debugger easier, and also reduces the risk of mis-bracketing -- a common mistake. Compound expressions, while obviously a common feature in high-level languages, aren't ubiquitous, and they do often make bugs harder to find.

It's perfectly conceivable that a math library written by someone who prefers the former style might be 3KLOC, and a math library written by someone who prefers the latter might be 10KLOC -- and that the latter has less defects.

The research in "Making Software" was a fine start, but it left me wanting more. Is there anything published on this topic?

I don't think sheer LOC is a good estimate of the number of bugs in many (most) languages, even if it happens to be an effective measure in C. Constraints will often increase LOC while reducing defects. There are many kinds of constraints, including many which are language features not available in C:

• Proper-English naming conventions
• Unit tests/TDD
• Runtime assertions
• Type-safety
• Pre/post-conditions
• Invariants
• Units of measure (arguably a type of type-safety)

But according to this study, TDD can nearly double LOC but reduce bugs. (I don't believe that this is a result of TDD so much as it is a result of unit testing, but that's another story.)

No, TDD is not a language feature, and no, I don't think unit test code should count towards LOC directly, but it is very much another form of constraint, and it shows that constraints, even if they increase code, work to reduce bugs.

Also, if for instance all languages supported units of measure except one language, all else being equal, that language would be more concise, and programs in that language would have more bugs, including ones that can crash spaceships.