The cost of longer delay between development and QA

Question

At my current position, QA has become a bottleneck. We have had the unfortunate occurence of features being held out of the current build so that QA could finish testing. This means features that are done being developed may not get tested for 2-3 weeks after the developer has already moved on. With dev moving faster thean QA, this time gap is only going to get bigger.

I keep flipping through my copy of Code Complete, looking for a "Hard Data" snippet that shows the cost of fixing defects grows exponentially the longer it exists. Can someone point me to some studies that back up this concept? I am trying to convince the powers that be that the QA bottleneck is a lot more costly than they think.

Answer 1

You don't need any references, IMHO. Here's what you could (rather should) do:

Quantify the Cost of Delay! Let's assume that it takes 1 week to test the feature(s). A 2-3 week delay implies that the feature won't be available till at least the 4th week. And that too assuming 100% success. Add a fixing time of another week so that's about 5 weeks of delay.

Now, if possible get access to the expected deadline of the project/feature. By when does the client expect it? Will it slip? If not, will others slip as a consequence? So by how much will the 'release' be delayed as a result?

What is the 'cost to company' for that release i.e. how much does the client expect to profit from that release? If they expect $5200/yr of profit from that release then every week slipped costs them $100 in lost revenue. That's the client view. You may or may not have access to this data but it's worth taking into consideration and stating how the delay can impact the relationship.

Now, what's the loss to the developers? Once the developer moves on to other features, you ask him/her to break their cycle and 'fix' the previous feature. What's the time/effort loss? Convert it to cost to company by using the salary as a multiple for every hour wasted as a result. You can use that to say the amount of "profit/revenue" which the waste is "eating into".

What you've stumbled on can be conveniently quantified using "Cost of Delay" - advocated by Don Reinerstein in Principles of Product Development flow and also by Dean Leffingwell in Agile Software Requirements. You should be able to back every such claim by economic factors to convince the 'higher powers' whose primary language is $$ - you must speak their language to convince them :)

Beast of luck! (pun intended :)

Answer 2

I don't really think that Code Complete is the right resource for you here. This isn't a code problem, it's a process problem, and maybe a management problem.

If part of your process is particularly weak, then it's time to bust out the Theory of Constraints:

1. Identify the constraint.

   This means finding the slowest or most inefficient part of the overall process. In your case, it's testing. But what part of the testing? Is it:

   • Preparing the test environment?
   • Determining what to test?
   • Functional (acceptance) tests?
   • Regression tests?
   • Exploratory testing?
   • Reporting bugs/defects from the tests?
   • Determining the steps to reproduce a bug?
   • Getting clarifications from developers or project managers?
   • Fixing the issues found in the QA stage?

   These are all very different problems and call for different solutions. You need to decide which is the most costly/important. Justifying it to management should not be hard, as all of the above activities cost time (AKA money) and only a couple of them are value-added time.

2. Exploit the constraint.

   In other words, optimize around the constraining process. Don't ever let the testers be idle. This essentially amounts to:

   • Putting testers inside development teams, if they aren't already, so there is a continuous feedback loop with developers.
   • Having frequent test deployments, so there is always something new/fixed to test.
   • Making communication faster and more frequent. For example, favour instant messaging over email threads.
   • Ensuring that testers have the best tools available (fast machines, multiple monitors, streamlined bug tracking, etc.)

   This stage isn't about optimizing the test process itself (yet), it's more about reducing overhead. Don't waste testers' time. Eliminating time that is truly wasted should also be an easy sell to management.

3. Subordinate other activities to the constraint.

   At this point, the testers are as productive as they can possibly be by themselves, so we need to start borrowing productivity from other areas:

   • Instruct developers and operations staff to give the testers first priority, no matter what else they're working on.
   • If you don't have cross-functional teams, reserve a meeting room every day at a preset time so that the testers never have to waste time trying to book one.
   • Divert some larger percentage of developer (and possibly operations) time away from feature work; for example, focus on bug fixes, tech debt/refactoring, code review, and unit testing.
   • Test continuously and incrementally - don't develop for 3 weeks and then kick it over to the testers. Have developers work on making their code immediately testable, e.g. with scaffolding or prototype UIs.

4. Elevate the constraint.

   If the testers are working at full capacity - both in terms of productivity and minimal overhead - and it's still not fast enough, then you need to start investing more in testing.

   • If you rely on manual test deployments, automate it through the use of continuous integration and configuration management scripts.
   • If test plans take a long time to create, work on getting better acceptance criteria (i.e. INVEST). Most organizations are initially very bad at this.
   • If acceptance tests are taking too long, start automating them. Use a tool like Cucumber or FitNesse, or write xUnit-type tests if you must. There's also Selenium, Watij, and other browser automation tools if the UI testing takes a long time.
   • If regression tests are taking too long, automate that too. If it can't be automated, focus on improving quality out of the gate, i.e. with even greater emphasis on code review, unit testing, static analysis tools, etc. Developers should be fairly confident that there are very few actual bugs before passing it to QA (product defects are a different story).
   • If exploratory testing is the bottleneck, you could potentially defer some of that until after the release, or contract out to a testing firm to do highly parallelizable things like testing the same workflow in multiple browsers.
   • If testers are unable to repro a lot of bugs, consider building out a capacity-testing environment to be able to start reproducing intermittent errors. Or, try just running your automated acceptance tests in parallel and continuously, all day long, keeping detailed logs.
   • If it takes too long to fix the bugs, then start partitioning your projects and/or seriously consider rearchitecting your solutions. Or, alternative, don't fix some of them; not every bug is critical, you should be able to prioritize them alongside feature work.
   • If all else fails, hire more testers.

5. Go back to Step 1.

I would like to say that this is all common sense, but unfortunately it's not, at least not in most organizations. If you are getting a lot of resistance from management, an invaluable technique is Value Stream Mapping (a technique from lean manufacturing), which you can use to show how much time and therefore money is actually being wasted every day by a release candidate being unable to move to the next stage. Opportunity cost is hard to understand but this is one of the best ways I've found to visualize and demonstrate it.

And if none of that works... then maybe you're in a dysfunctional company, get out before it's too late!

But you won't solve this problem simply by dropping a few papers on the manager's desk and asking them to throw money at the problem, because they will assume (correctly) that throwing money at it might not actually solve it, and might even make it worse. You need to provide solutions, and that's what this answer is about. If you introduce the issue as "here's a list of ways we can start solving this problem, in descending order of cost/benefit" rather than "we need more testers", then you will have a lot more success.

Answer 3

Pages 29 and 30 may have the data you are looking for, although a follow up might be required.

I would look into the research mentioned in this sentence on page 30:

Dozens of companies have found that simply focusing on correcting defects earlier rather than later in a project can cut development costs and schedules by factors of two or more (McConnell 2004).

BTW, it was your question that made me pick the book up again to refresh it :-)

Answer 4

What you are describing is a bottleneck in a process. Lean theory states that there will always be a bottleneck in a process, but its severity can vary widely. If QA hired one extra, then development might become the bottleneck.

To understand the cost, imagine the following situation. You picked one of the developers. His work would never be Quality Assured, but always get queued indefinitely. Imagine that, this would mean that the QA could Quality Assure the work of the rest of the developers in a timely fashion and there would be no cost of delay.

In that scenario, the cost of the delay is the cost of the salary of the developer, because his work would be wasted.

The reason why I argue in terms of cost and not the value created by the process, is simply because it is harder to document the value of a process, even though it is far better.

Answer 5

I keep flipping through my copy of Code Complete, looking for a "Hard Data" snippet that shows the cost of fixing defects grows exponentially the longer it exists. Can someone point me to some studies that back up this concept?

Exponential cost of finding a bug seems to be based upon a NIST study. The study was a survey assumes distinct waterfall stages:

Software Development Stage         | Cost
-----------------------------------+------
Requirements and Design            | 1X
Code and Unit Testing              | 5X
Integration and System Testing     | 10X
Beta Testing                       | 15X
Post Release                       | 30X

(table here from here)

One of the aims of Agile software development methodologies is to remove these distinct stages and reduce these costs. The figures do not apply when using other methodologies to waterfall.

Answer 6

Your probelm is not with QA, in fact, if your QA is doing Testing, the delays are pretty much the least of your worries. Please let me expalin (again, as it's common misconception in the Programming industry)... QA Assures the quality of the product by overseeing the entire SDLC, from Requirements (maybe earlier), through development, verification, release and support. Testing ensures that no obvious defects exist in the code. There is a very large and important difference. If you had true QA, they would be all over the Test/ V&V department asking why they were costing the business time (and therefore money) delaying releases, or all over project managment ensusring they were managing project scheduling properly, or all over managment making sure there were enough testers for the code being produced etc...

So assumeing by QA you really mean Test, back to the orginal question. Code Complete got it right - the cost of a defect is the time taken from insertion to correction. Early detection is only useful if you also correct it early, but most peoples interpretation is wrong.

(Note : I am playing Devils advocate here, don't take any of this to literally as I know nothing of you situation) The delay cause by your Test department is a cost, absolutely, however, I have to ask, if you are waiting for them to find your defects, what are you doing - should you not be finding you own defects? Perhaps if they had less work (through higher quality input with less defects from you), the delay would not be as significant and the cost less - as a manager I would ask you how you plan to reduce the defects in the code you deliver to test, as (based on your argument) those defects cost more if found by test then yourself.

Also as you manager, I might state that is not Tests job find your defects, Their job is to find there are no defects - if you are expecting them to find defects, maybe you have not done your job well enough.

Be careful how you approach this. If you dont have a solution to the problem, you are likely to come off second best.