How can I advocate a semi-strict release schedule in a risk-averse environment?

Question

Recently I've been increasingly plagued by what I would have to describe as one of my most frustrating and morale-killing experiences in this profession: Having to sit on a release that has been tested, re-tested, staged, and for all intents and purposes is ready to ship/deploy.

As an all-around solutions guy and not just a hardcore coder, I do understand and have even advocated the need for proper change control. But lately, the tenuous balance between covering our bases and shipping on time has gone all lopsided, and I've had little to no success in restoring it to something sane.

I'm looking for compelling arguments to help convince risk-averse management that:

1. The dev team should (or must) be able to set its own release schedule - within reason of course (1-3 months should be conservative enough for all but the biggest Fortune 500 companies);

2. Software releases are important milestones and should not be treated cavalierly; in other words, unnecessary delays/stoppages are highly disruptive and should be considered only as a last resort to some critical business issue; and

3. External (non-dev/non-IT) entities who want (or demand) to be involved as stakeholders have a responsibility to cooperate with the dev team in order to meet the release schedule, especially in the last week or so before the planned ship date (i.e. user testing/staging).

The above are assertions that ring true for me based on experience, but it looks like I'm now in the position of having to prove it - so I'm asking for something a little meatier here, if such a thing exists.

Can anyone who has had to "sell" the idea of a fixed (or maybe semi-flexible) release cycle to management give some pointers on what arguments/strategies are effective or persuasive and what is not? Aside from the obvious schedule contention and sunk costs, is there any hard data/evidence that would be useful in making the case that shipping is actually important, even in a "corporate" setting?

Alternatively, I'm open to hearing constructive arguments about why schedule flexibility (even over a period of weeks/months) is more important than shipping on schedule; it's hard for me to believe right now but maybe they know something I don't.

_{Note we have staged releases, and this went through every stage except production. Issues are tracked using a commercial bug tracker and every issue - 100% of them - that was assigned to this release was closed out. I realize it's difficult to believe and that's really precisely the point - it makes no sense that a 100%, feature-complete, fully-tested, approved-by-stakeholders release would be delayed by management for unexplained reasons, but that's what happened, that's what's been happening, that's the problem to be solved.}

Answer 1

Joel Spolsky says that a software development team is a scheme for converting capital (money) into working software. Honestly, from your question, it sounds like your team is good at that: you're getting decent software for reasonable amounts of money invested.

Now, of course the next step is to put the software into service. Until your company chooses to do that, there's no way for them to harvest a return on their capital investment. Software sitting on the shelf ready to deploy is kind of like inventory in the stockroom: the costs are sunk, the company's capital money is already spent. There's also an opportunity cost: your group chose to do this project rather than something else.

What's more, the value of newly developed software sitting on the shelf is kind of like the value of a crate of cheese sitting in a restaurant's fridge: It slowly rots. Its value declines because your competitors catch up. It also declines because it gets harder and riskier to put new software into service as its developers move on to other things. After a couple of years on the shelf, it may actually be worthless. In that case, your company has chosen to discard the capital they invested in developing it. It's possible that the owners of the company -- the shareholders -- will be displeased about this.

There's one thing you, as a developer, must figure out. Is your software really, truly, ready to deploy at the end of the release cycle you're proposing? Is it ready to go, or is there a whole bunch more work to do and money to spend as your company puts it into production? Does your company own the servers and software licenses you need to roll out your software? Does your work product include a deployment plan? Have the end-users been trained? Has the production data been converted? If your new software involves a change to the way your company does business, is the company ready to make that change? Have you worked out a scheme for running things in parallel, to make sure the new stuff works as well as the old stuff?

All those rollout costs can easily dwarf the cost of just developing the software. A wise project management team does its best to plan, budget, and schedule all that stuff. Have you done that work? Have you made it clear to management? If not, it's possible they're wise to go slow on your rollouts.

It's possible that a modern agile software rollout schedule is out of sync with the pace of change the rest of your company expects. Your end-users -- your stakeholders -- simply may not be able to absorb change at the rate your team can produce it.

It's also possible your management team is dysfunctional. Has your team developed something the rest of the company doesn't want? Does your new stuff threaten the heck out of your sales or production team? If so, some executive may be torpedoing it by saying it's too risky to roll out. There's nothing you can do about that unless you're the CEO.

You asked for compelling arguments for timely software rollouts. There's one really compelling argument: return on investment. The company chose to invest in this software, and now they're choosing to waste the investment as your software slowly rots on the shelf.

A secondary argument for timely rollouts is the morale of your team. Hurry-up-and-wait is not a good way to motivate creative developers to do good work. You may lose your team if they don't get the satisfaction of seeing their work go into service.

Answer 2

First, watch this video:

http://the99percent.com/videos/5822/Seth-Godin-Quieting-the-Lizard-Brain

Executive Summary:

The way you ship on time and on budget is this: when you run out of time or run out of money, you ship. That's it.

The reason most projects don't ship on time is because someone comes along with "just one more feature or fix" to insert into the release, right before you're ready to ship, at a time in the development cycle when your resources are most scarce, and now you have to scramble. This is called thrashing, and it happens because as long as you don't ship, you don't have to worry about people telling you your product sucks.

The way you cure thrashing is by forcing people to do it at the beginning of the product development cycle, not at the end.

Making product changes in the weeks just prior to the release date is very disruptive, for obvious reasons. This is true whether the change is a new feature, a change to the software development process, or an attempt to fix a long-standing bug that's not in the development schedule.

When someone comes to me for a fix or change late in the development cycle, I always ask them this: "What do you want me to not get done so that I can get your thing done?"

If you need a money motivator, it is this: studies prove that the later you wait in the software life cycle to implement a feature or fix, the more expensive it is. Exponentially. It's not hard to prove this.

Answer 3

You have clearly described the problems you are facing, however what I am unclear on why the managers are behaving this way. Can you clarify? For instance, you believe it's ready to deploy, does someone disagree, if so who and why? Are they ex governement beaurocrats

This sounds a lot like a capability vs desire alignment problem, you have the desire to release, but not the capablity (authority), those with the authority do not have the desire.

You need to find out why they lack the desire - it is marketing reason (keep it for another year while we milk the old version for a bit more), is it fear/confidence (If it has bugs, it will make us look bad, cost us money ...., lack of resource (Can't afford the cost of shipping/packaging/PR stuff), is it commercial where your some obscure sink to poor excess profits and they don't want you making money (Not as silly as it sounds).

I also suspect theres litle respect between parties involved, therefore reasonable adult discussions are not occuring.

Sorry - not the specific answer you have asked for, hopefully a couple of things to think about though.

Answer 4

First thing to consider is to make sure your releases aren't high risk.

Ideally, your change requests should have section called like Risk Mitigation, containing instructions on how to rollback to prior version if things go wrong. Risk-wise, no amount of prior testing can substitute a simple option to rollback the change.

• ^{In risk-averse environment, I can not imagine a way to make irreversible changes painless.
  If many/most of your releases fall into this category, you may want to reconsider your architecture.}

The risk of NOT releasing is to be exposed, if you want dev team schedule to be treated seriously.

If your releases deliver fixes for production / support issues and outages, this is fairly easy to do by just listing these and pointing out that production is at risk of repeating these unless it's upgraded.

A really effective measure available to dev team to fight against release slippages it to make sure that risk of not releasing increases over time. This can be achieved with internal releases running at fixed schedule, providing "cumulative" list of solutions to production issues.

• Simply speaking, guys blocking your release XX need to be aware not only that they incur the risk of having N kinds of production issues to remain unresolved, but also that week (or month) later, they will have release XX+1 in their approval queue, blocking which will incur the risk of N+M kinds of production issues remain unresolved - and that this will also be the case with XX+2 and further "internal" releases supplied by dev team.

---

The way described above essentially makes tighter and more explicit connection between your application updates and production issues.

Because of that, you may expect more involvement into production issues escalations. You may use these as opportunity to promote your vision of stricter scheduled updates. You may even trigger some escalations yourself to speed up the process of adopting your desired approach.

• ^{"...recommend to consider upgrading application to newer version (XX or later). The one that is currently deployed to your environment (XX-2) is severely outdated - two major releases behind the current codebase. As a result, details for such simple failures are deeply hidden in logs and undecipherable garbage is reported by application to clients instead of clear failure descriptions - causing users and support to waste time investigating issues that are really quite simple"
   
  Above is a comment I added some time ago in support issue tracker for the ticket related to particular production incident. Soon after that, "stakeholders" contacted dev team asking how to keep track about our releases and how they can help in deploying to their environment. Oh and they also quickly upgraded from XX-2 version, too. :)}

When production escalation happens, you better be prepared to present your vision quickly and clearly.

One thing you'll find useful is to pick and track who's responsible for particular release slippage. Basically, you need to be able to quickly and clearly answer to question like "who holds responsibility for the fact that planned release did not happen?" If you're not ready, they may choose the path of least resistance and put the blame on you. As pointed out in comments to another answer, "You could get yourself in hot water that your management is trying to protect you from."

The last but not the least,

it'll take time

(you probably already know that but it looks worth stating explicitly)

What you look for can be described as attitude change, from "it is risky to release" to "it is risky NOT to release". Attitude changes rarely happen overnight, patience might be needed to get the shift completed.

Answer 5

There is a big question mark hanging over your question, and that is why your company doesn't wish to release the software. It's not always just a simple case of risk aversion. Often there are other causes underlying which don't often get passed down from the lofty managerial heights. So my question to you is, "Have you sat down with your boss and asked him why the product release is being held up?".

There is a big difference between managing risk, and avoiding it. There may be legal or marketing reasons for delaying a product release. It could be trust issues between management and the development team. The product may not suit the needs of the customer, even if it is exactly what the customer asked for months ago. It could even be a case of someone in management seriously covering-ass while trying to shift the blame for delay elsewhere, or even a delay by a third party that is required to complete something before your product ships. I could come up with dozens of scenarios. Quite often the developer assumes risk aversion without understanding the other side of the equation that hasn't been made apparent. On the flip side, it may simply be complacency, conflict between individuals within a company, or even a fear that the product will fail in the market for some reason which results in seemingly ridiculous delays.

In all cases, it's important to have more information about the reasons for delays in product release, and to use that information to build a case for releasing your product as soon as you can. Yes, it can be very frustrating, but there are other ways to deal with these situations without risking either confrontation with your bosses, or a burnout. In similar situations, I myself have gathered information, documented whatever progress I've made, and then if worse came to worse, simply shelved the project and moved onto something else. Where I have been able to get a project back on track for release has usually been the result of making a sound business case based on the information I have received as feedback to the questions that I have raised. Where I have been unable to convince the management that the product should be shipped, I have documented the reluctance to ship as simply being a case of project finished, and awaiting release approval by management. I then make sure that my manager signs my document as accepted and understood by management so that my own butt is covered if they should attempt to blame me later for a non-release.

You always need to dot your I's and cross T's to avoid shipping delays being later blamed on you (no matter how unfairly), and also it's important to avoid becoming too emotionally bound to such issues unless you like the idea of a well earned ulcer. By looking at your own situation with a certain professional detachment, you may find a solution presents itself because you have effectively placed yourself at a greater "distance" as it were from the problem. If you can't make your case, you may need to let it slide for a while, but in order to make your case in the first place, you need to seem professionally detached, and have arguments based on information that is intimately connected to your company and its inner workings.

Something else that I've just thought about which might be of some help to you, is to perhaps read Lean Software Development, and see if there is anything valuable in there that might relate to your company's situation, particularly in the first few chapters. I think it was chapter 4 that discusses the issue of delivering as quickly as you can and has something relating to delay costs.

Answer 6

I'd say the easiest / most effective way to sell a release is to down tools for a while when its ready to go. Do something else, start a new project, work on bugs for the existing project. Do not do any more to this one until its sent out the door - after all, why bother doing it at all if it doesn't go when its ready.

but in the real world, the actual release is someone else's problem, you should be shipping it to them and then treating that as a release. Once its out of your door, its shipped. If they just sit on it, it's not your problem (in fact, its great, no bugs get reported on it! w00t! Bonuses for a bug-free release all round ;) )

So you need a change control system in place anyway, and a release manager. Then it's all easy (except you do need to manage the change control, so source control plus deployment builds are very necessary. It requires organisation, but if you're organised, its easy).

Answer 7

I can't imagine that putting the Dev team in charge of the business the way you describe is a "good thing" It might mask the real problem, but unless the Dev team is in the right position to weigh the inputs from sales, marketing, etc. etc. you risk releasing for the wrong (and potentially bad) reasons.

If I understand your problem- you have completed the project and have a deliverable you can't show to anyone outside of your company. (I hope this summarizes it, I read the whole thread and I'm having trouble putting my finger on it)

Have you tried:

1. Ask management for a customer or set of customers to act as stakeholders during development? Usually you can find one customer to take intermediate releases and give feedback. They can be great advocates when it is time to release. Even a "limited release" to one customer might be enough to help sway management
2. Building a release plan including point releases. That is, you can release 3.4 today because the 3.4.1 release is planned for today + 1 month. This along with the good idea you already mentioned that a release cadence helps this message.
3. Get support, sales, or marketing on your side. Build in fixes that would solve the top 3 support requests and you can get an ally from another department. If you can't get a customer stakeholder as in #1, try it with a few sales people. Offer to be available for sales meetings and bring along the product. When you are on the road, show the sales guy you are with the product (in whatever state it is in) get them pulling for you.

To answer your other question about "why would management sit on a release?" Companies do this all the time in non SW fields and I don't think it is unprecedented. For example, you have a new release that is ready to go, all tested and done. But the old version hasn't been displaced by the competition yet. Once the competition releases a competing product you the old version, management releases the version waiting on the shelf and disrupts the market, sets back the competition and maintains sales and the reputation as a market leader. Releasing too soon tips their hand to competitors who might have a better idea of your roadmap and try to beat you to the endgame.

With all that said... Hanlon's Razor is probably the right answer :-)

Answer 8

Go away from that company, they don't understand software. But seriously, software is what makes the system work, imagine if you were making cars, are car factories slow? do they wait on car releases? Are they incremental and bureaucratic? No, they try to do things the fastest and cleanest way possible. You have a management problem and you should work on it as a management conflict, try to speak them into their terms. Ask what risk means to them, then try to minimize it. The feelings of your devs are also important, since they must cope with the decisiones that will be made by your intervention. Will they be happy doing all nighters to ship on time? What would the prizes/penalizations be? Etc. Now i will give you a practical approach to this problem, a bit extreme, but ask for an audit. That will give you tools to talk to management about the problems you see, but from an external standpoint, and will also improve your image with them.