Best practices for releasing patches that affect multiple development stories

Question

My team uses a Kanban style approach to development where feature and bug fix stories filter through to Production as and when they are ready. We currently are using SVN as our VCS and take a feature branching approach to developing each story. A very simplified version of our workflow would be this:

1. Branch, develop story
2. User acceptance demo of story from branch
3. Reintegration to trunk, TeamCity build of package which is released to Test
4. Into Production if all is well

However one issue that has come up recently is if we have a number of feature stories for the same application going through at the same time this can cause problems if one of those stories has a bug. So say story 1 was reintegrated and had a bug, but stories 2 and 3 were reintegrated before that bug was discovered. We now have 3 releases containing that bug which means we end up doing one of two things, neither of which are great:

1. Create a new patch release to follow story 3, which means a large Production deployment as stories 1, 2 & 3 + patch have to be deployed as one.
2. Rollback the trunk to the point of the bug being introduced in story 1, fix the bug, then reintegrate stories 2 and 3 and rerelease, which is a lot of work and open to manual error.

So my main questions would be:

• is it more optimal to conduct System testing on branches prior to their reintegration to head off this scenario? How about the risk that testing is not taking place on the true trunk code?
• or are there other possibilities given by the use of a more modern DVCS that we are restricted from trying by our use of SVN?

Answer 1

You can release to a QA 'trunk' before going to the release trunk. This lets you get more testing in, and lets you promote good integrated builds to a release. The difficulty comes if you find a bug, you have to decide either to fix it and get it integrated, or to cherry-pick those branches that passed onto the release trunk, skipping the feature with the bug.

This means your QA trunk can slowly get out of sync with the release if you have lots of buggy branches being integrated (ie as some features will be merged to QA that do not make their way to Release). You can solve this by either always fixing the bug and then merging the update to QA which will get promoted to Release on the next cycle, or reverting all changes from the feature branch from QA if its failed and is not going to be part of the solution any time soon.

You should be testing on branches too, but these will not necessarily show up bugs caused by poor integration with other features. It should show that they are good enough to be promoted to the next stage - ie QA trunk.

Of course, you could just release the bug with a note explaining its effects as a known problem. No doubt you already have bugs in the deployed product that are found by the end user. You can deal with the new bugs in the same way you deal with ones found in live environment.