3

I work on a large software programme - 100 developers in financial services.

The common wisdom of Continuous Integration is to get feedback early from your changes.

The common wisdom from Continuous Delivery is that by getting good at releasing small chunks, you reduce the risk of failure, because you can roll back easily - and so releasing small chunks helps you deliver to production rapidly and often.

A business value diagram in Lean allows us to see the flow of business value from left to right (similar to a production line) and from this you can identify where your change items are getting stuck, and where the bottlenecks in the process are.

The challenge in software development is identifying precisely what the widgets on the production line are.

If you read The Phoenix Project, then the changes are the change records flowing through the system(although this is heavily IT infrastructure focused). If you talk to a Scrum master - then the changes flow through the system are stories. If you talk to a developer, then the changes that flow through the system are GIT commits. (Which can and should align to stories).

The simple reality is that we do small releases once a month, and large releases once every three months, due to the transaction cost of the regression test. (Don Reinersen's book The Principles of Product Development Flow is amazing on the tradeoffs of cycle time and transaction cost.

So in trying to identify the constraints on the system - instead of finding a work area where the items are piling up - to me it seems that the batch size itself is a constraint. By batch size, I mean the number of deliverables in a release. A release every month with a large number of developers would have a large number of stories/commits. I'm trying to quantify this.

We know that the economics of batch size is a u-curve optimisation problem, and that the transaction cost of a regression test and release is substantial.

My question is: How can I determine the optimal release frequency for maximum throughput?

Thomas Owens
  • 79,623
  • 18
  • 192
  • 283
hawkeye
  • 4,819
  • 3
  • 24
  • 35
  • 1
    I'm curious as to why the cost of regression testing is related to number of work items completed. That doesn't make much sense to me. To me, you are either regression testing the entire system and the time is related to functionality (more functionality = more regression testing) or you are regression testing changed components and maybe critical functionality (more changed components = more regression testing). It looks like you suspect that your final regression testing is the bottleneck. – Thomas Owens Nov 28 '16 at 19:06
  • @ThomasOwens It's a good question. I think there are two main factors at play. The total features tested should grow but the new/old marginal growth rate should diminish over time so I think we can ignore that. The other aspect that I've seen is that the number of defects (and time to fix) is roughly proportional to the number of changes. This is probably the bigger factor. – JimmyJames Nov 28 '16 at 20:25
  • @JimmyJames The assumption being made seems to be that the bottleneck is regression testing and the desired solution is to determine, in advance, how much work can be done to avoid a bottleneck at that phase. I think that the question being asked is wrong. I don't understand how regression testing can be a bottleneck, since it shouldn't be dependent on number of work items being done in a release. Time to regression test should either be a fixed time or dependent on modules being modified. Creating a VSM and applying standard lean tools should be applied to reduce the time to regression test. – Thomas Owens Nov 28 '16 at 20:31
  • @ThomasOwens: The size of the batch (or, rather, the length of time for each batch) affects the number of regression tests performed, in inverse proportion. It seems like a simple optimization problem to me, readily solvable with perhaps a bit of algebra. It's not a bell curve; the longer the batch is, the fewer regression tests need to be performed. On the other hand, the regression tests might take longer for longer batches. I still think it is a rather mundane math problem though (albeit with a handful of arbitrary guesses). – Robert Harvey Nov 28 '16 at 20:53
  • 1
    Can you elaborate on the transaction costs of regression testing. I just got through your first book ;) – JeffO Nov 28 '16 at 20:55
  • @RobertHarvey I don't understand that assumption. Let's say that I have a system with 5 modules (A-E). If I make 5 changes to module A, 3 to module B, and 0 to the other modules, there are three reasonable possibilities, from my perspective. 1: My regression test suite is executed in full on the entire system. 2: The regression test suite is executed in full on modules A and B. 3: The regression test suite is executed in full on modules A and B and "critical functionality" from the other modules. The time to execute doesn't change based on the number of changes to modules A and B. – Thomas Owens Nov 28 '16 at 20:57
  • @ThomasOwens My experience is that regression testing is not independent of the number of features. See my answer for details. More features -> more defects -> more test/fix cycles – JimmyJames Nov 28 '16 at 21:28
  • You are assuming the types of features or defects have no relation to the release schedule. Some features may necessitate a faster release due to demand; some defects may take longer to correct due to difficulty. Are abstract feature and defect counts relevant? – Frank Hileman Nov 28 '16 at 23:53

1 Answers1

0

I've been thinking about this also and to do this, I think you first need a hypothesis. Here's where I would start:

Regression Overhead

You are bound by the minimum time to regression test. I'm going to assert that you can't release faster than that. I think a basic formula for this would be based on the following components:

  • Time required to test all unchanged features
  • Time required to update/add features
  • Time required to test new/changed features

Time to regression grow over time as you add features but the marginal rate of increase will diminish over time. Your situation may vary but I think we can assume this is fixed. If it's growing significantly, having releases on any fixed frequency will not be sustainable.

Defects and Time to Fix

Hypothesis 1

Let's assume that the number of defects found in a release is proportional to the number of features in the release. We'll also assume that the time to fix and retest those defects is proportional to the number of defects.

Hypothesis 2

Again we assume that the number of defects found in a release is proportional to the number of features in the release. But now we assume that the time to fix and retest these is some function of the number of defects that is not a straight multiple. the reasoning is that if you find some defect and fix it, then regression testing must be run again. These defects prevent some other defects from being discovered. This creates a lead time on fixing defects.

Testing the Hypotheses

In order to determine whether either of these match reality, I think you need to determine/guess the bare minimum time to regression test everything. You can't go faster than that. Then you need to try different release schedules. This might require some spikes. Then you start trying to decompose regression time for the number of features. You might see something like this:

enter image description here

Or maybe it looks like this:

enter image description here

Where the green is your regression baseline time. Or maybe it looks like something completely different. Based on my experience it's more like the latter.

If it's the former, it would suggest that you should go with really long release cycles because that minimizes overhead of the baseline regression. However there's another factor that you haven't mentioned: business value. If a feature produces value for the company (and why are you doing it if it doesn't) the longer it takes to put it into production the less value is adds. So even if it is a straight-line cost, you need to consider that.

Once you understand this relationship, then I think you will most likely want to make the dev time for a set of features match the QA time for those feautres (considering other overhead activities like fixing defects, updating tests, sharpening the saw, etc.)

Community
  • 1
JimmyJames
  • 24,682
  • 2
  • 50
  • 92
  • Thanks Jimmy - would you consider a u-curve optimisation between decreased transaction cost per story (release and standard regression) and unit cost (story build and test increasing linearly with time) – hawkeye Nov 29 '16 at 07:08
  • @hawkeye Sorry, I don't understand the question "would you consider a u-curve optimisation between decreased transaction cost per story and unit cost". What about these are we considering? – JimmyJames Nov 29 '16 at 14:54
  • I'm talking about this: https://continuousdelivery.com/wp-content/uploads/2011/12/Fig2-2rev1-Humble.jpg and this: http://www.innolution.com/uploads/misc/u-curve-optimization.jpg – hawkeye Dec 01 '16 at 09:49
  • @hawkeye I understand the U-curve. I don't understand what I am 'considering' about it. – JimmyJames Dec 01 '16 at 15:21