4

Most Defect density metrics are like this:

Number of bugs / Size (KLOC, Story Points, Function points..)

I have read that it measures the effectiveness of QA but I do not get it:

  • I can have a super senior developer and a sloppy QA. Therefore, the sloppy QA guy finds nothing because there is nothing.
  • I can have a bad developer and a good QA guy, who finds most of the issues.

Therefore, how this could be measuring the QA? In my view, it reflects both quality of dev and effectiveness of QA.

John V
  • 4,898
  • 10
  • 47
  • 73

3 Answers3

4

I have read that it measures the effectiveness of QA but I do not get it

It is surely a good idea to take any such statement about metrics with a grain of salt.

The only way I can think of to use the given metrics to measure the effectiveness of QA, is to take the same piece of code, give it to different QA people and let them independently test it. In theory, the more bugs they find, the better the QA (however, in reality the severity of the bugs found should also be incorporated).

Of course, by doing so the size of the code is fixed, so using the density as a number is quite pointless, you could actually just use "Number of bugs found by QA" as a metrics.

As you correctly have noted by yourself, as soon as you vary the QA persons and the subject under test, the metric does not tell you anything reliable about the effectiveness of the QA any more.

Since you did not give any reference where you found this questionable statement, I did a quick web search to see what others wrote about this metrics. It brought me to this page, mentioning two uses for the defect density metrics:

  • For comparing the relative number of defects in various software components so that high-risk components can be identified and resources focused towards them.

  • For comparing software/products so that quality of each software/product can be quantified and resources focused towards those with low quality.

"Effectiveness of QA" is nowhere mentioned. So according to this source, defect density is a metrics for quantifying quality aspects of the software, not of the development or QA process.

(And if you read something different in a text book, you better ask the authors of the book what they had in mind with their statements.)

Community
  • 1
Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • Well but in reality, you have no time to for testing a piece of code by separate QAs. What I mean is, if the Defect density is low - I have either good developers or good QA. (considering density only of those found after release). So how is this QA effectiveness? – John V Oct 08 '18 at 14:52
  • @JohnV: my answer is not about the time you have in reality, it is about the stated metrics and the correct interpretation of what you read about it - from a theorectical point of view. If *"in reality, you have no time to for testing a piece of code by separate QAs"*, then you probably should not try to use this metrics for measuring effectiveness of QA, it won't give you useful results. – Doc Brown Oct 08 '18 at 14:59
  • Yes and that is my point - nobody has time for this in practice. And yet this metric is in various books mentioned as "measuring effectiveness of QA". If the Defect density is 0 (meaning no bugs reported months after release), either my QA is amazing or my developers are godlike :) – John V Oct 08 '18 at 15:06
  • @JohnV: well, don't believe anything just because it is written in some textbook ;-) – Doc Brown Oct 08 '18 at 15:12
  • So I assume I will carry on with using this metric to indicate our general trend with regards to defects in our system, instead of using it to assess my QA. – John V Oct 08 '18 at 15:14
  • @JohnV: see my last edit. – Doc Brown Oct 08 '18 at 15:21
  • "Number of bugs found by QA" maybe a better metric is number of bugs found by customers. After all, the real goal is to ship quality. – Wes Toleman Oct 12 '18 at 15:00
2

Defect density measures the number of defects found per unit of code measurement (never ever use lines of code for this), for a given period of time. The higher the number, the greater the density of bugs.

The density of bugs can be taken to be an inverse measure of the quality of the app. The lower the density (ie, the less defects reported), the higher the quality of the code.

But being a simple metric it has flaws:

  • If I simply do not test the code, my defect density is zero: perfect quality! Of course in reality, most untested code is far from good quality.
  • Conversely, for the same piece of code, I would get way more defects reported by super-zealous testers than I would for a lacklustre dev asked to do some testing. So I now have two densities (and thus two quality measures) for the one piece of code. And since I'm judging those testers on the number of defects raised, I'm incentivising them to score the quality as low as possible.
  • And of course, there's the issue of early testing. Pair a developer and tester during a sprint, and the output will likely be low in bugs. But it's extremely unlikely that the defects they found during the sprint will be recorded, so I've no measure of quality unless I retest just to gain my metric.

Defect density is just a metric. To be able to read more into it (quality of code, effectiveness of testing, likelihood of the app containing significant bugs etc) requires a heavy dose of subjectivity. Unless you know how effective your testing is, defect density won't be a reliable quality measure for example. But if you can't use metrics to measure effectiveness of testing, how do you measure it? So a subjective measure is needed.

David Arno
  • 38,972
  • 9
  • 88
  • 121
  • Yes but I still do not think I measure effectiveness of testing (as I cannot know whether there were 0 issues and hence 0 found, or 100 issues and all discovered and fixed). I think it reflects the maturity of dev-QA. – John V Oct 08 '18 at 13:37
  • `If I simply do not test the code, my defect density is zero: perfect quality! Of course in reality, most untested code is far from good quality.` - That is not correct. If you have field reported issues, then you have a defect density of more than 0. Also, not tracking the issues doesn't mean they don't exist. – Thomas Owens Oct 08 '18 at 14:39
  • Yes, lines is a very imperfect measure: While drawn-out code likely has fewer bugs per line (it does little per line, so has little per line to do wrong), it probably has more bugs per unit due to monotony. – Deduplicator Oct 08 '18 at 15:43
  • David, if the teams use the same languages, isnt KLOC still more accurate than story points? If there are coding guidelines and conventions, the differente should not be that bad, in my opinion. – John V Oct 09 '18 at 06:14
  • @JohnV. There are two problems with using KLOC. Firstly, the more lines I write, the lower the density for a given number of defects. So if you are using defect density to measure my app's quality, it makes sense for me to make my code as verbose as my coding standards allow to make my app appear to be of higher quality. Secondly, unless you have offensively restrictive coding standards that remove all self expression in code, different devs will use different solutions to the same problem. 2-3 times difference in the line count is not unlikely there. So you aren't comparing fairly with KLOC. – David Arno Oct 09 '18 at 09:39
  • @DavidArno I agree with that. But on the other hand, it is likely that all projects will do that and therefore the distortion will be everywhere. I think that is why the metric was quite heavily used by IBM an others. – John V Oct 09 '18 at 09:57
1

It was probably refering to defects found after release. ie by users not QA.

In this case, it is a measure of QA effectiveness as poorly coded work would be easier to find bugs in and hence not release. Very well, but not perfect code, might have a couple of minor, hard to find bugs slip through, but not many.

Of course it could also measure project manager-careing-more-about-deadlines-than-bugs-ness

Ewan
  • 70,664
  • 5
  • 76
  • 161