Estimated work remaining doesn't get lower

Question

I'm working as a developer / architect on a software project. The project manager decided not to follow the Agile principles, but to rather have an Excel sheet with all the feature requests and their estimated (remaining) effort. I'm required to update the estimated remaining time of each feature that I'm implementing weekly.

The problem is: with bugs and requirement changes constantly incoming, the remaining effort doesn't change as fast as the manager expects. Say, it was "12 days remaining" about 2 months ago, and today it's still "7 days remaining". The manager is rightfully getting nervous about this, because he can't give a release date for the potential customers and the marketing department this way. What is the best way to solve this problem? I think the main goal would be to be able to give a release date that makes sense. How could one achieve this?

Now, my suggestion at the beginning was to:

1. Set priorities for all the features. This didn't work out well, since about 70% of the features got priority 1, all the others got priority 2. And the verdict was: all features must be included in the first release.

2. Reduce the number of features, or reduce the complexity of the features which are the most difficult to implement or unnecessarily complex (meaning: no user will ever understand how it works). This also didn't bring much, all features must be implemented as they are, and even more features were added.

3. Add bug fixing & feature change estimations as they come. This is what we normally do in another project at this company. The result is, that sometimes the work remaining goes up between weeks instead of going down. This also wouldn't solve the main problem (release date?).

I'm running out of ideas... The manager has told me that he thought my estimations mean the time required to write bug-free code... No comments on that remark. He also said he expected me to include the estimation of bug fixing in the initial estimation. Yes, to include the effort to solve a bug which we don't even know if it exists...

His other suggestion was that I write my tests so that they cover every possible case. It would make sense, but having such complex features and a very large and complex background service we are calling from this application, it would take forever to cover every scenario with integration tests. I don't think we can ever achieve 100% coverage.

Edit: The questions How to answer "When will it be done?" and What can I do to get better at estimating how long projects are going to take? might appear similar, but are not relevant to my situation. I have already given an effort estimation. My question is related to the progress of the development, and changing estimations / deadline.

Answer 1

It's a software project, therefore it's overrunning.

That's a slightly flippant answer, but this is a very common problem. You say there are "requirement changes constantly incoming": that's why you're overrunning. You absolutely have to have a requirements freeze or you will never ship. Only with the requirements frozen can you issue a release date.

Your manager also needs to be aware of the cost/quality/features tradeoff triangle. You can only have a perfect product with lots of features if you're prepared to spend decades building it. Startups and small companies tend to aim at a "minimum viable product", minimising all three quantities.

Put the spreadsheet in the revision control system. It doesn't matter if it's binary and undiffable, the important thing is to track changes. Then you can dig out the spreadsheet from three weeks ago and point out that you've done three weeks work and reduced its expected time by three weeks - but today's spreadsheet has another four weeks of features added!

Bugfixing itself usually shows up deficiencies in the specification. You have to triage bugs and focus on minimising risk to the user. It's useful if your software can fail gracefully and retry, but this option is not available for security bugs.

Answer 2

What you experience is called

feature creep

The Dilbert site has a nice collection about this topic.

Now, my suggestion at the beginning was to: 1: Set priorities for all the features.

I would have proposed the same.

This didn't work out well, since about 70% of the features got priority 1, all the others got priority 2. And the verdict was: all features must be included in the first release.

Your Product owner or the equivalent of it is either a weak person or has a conflict of interest with the customer. Unless the resources available on your side is adequately sized, this is not going to happen from the start. Especially knowing, that in typical projects there will (and should) be feedback, and as a result there will be changes That's a given.

What turns it into a problem is the fact, that nobody in your team seems to have the balls (or maybe the authority) to tell them, that this ain't gonna work out.

1. Present data. Collect data about features and estimations at various stages of the project. Draw a burndown chart of it. Try to make some forecasts. You may also try to use Evidence Based Scheduling.

2. Smaller releases Propose to release the product in multiple smaller milestones instead of one big release. Try to get consensus about what should be part of each release. To achieve this, people will have to prioritize their wishes. Explain them, that this is a good thing for both sides.

3. Get backing. If the PO or the equivalent of it is the problem, try to find someone else in your company, with the necessary authority, to give you some backing. Explain the problem as above and sketch out your ideas. Make it clear, that you want the project to become a success, but are worried about the way it goes. Then ask for advice.

The key is to get them to realize, that the day has only 24 hours and resources are limited, while features wishes aren't. If they want more features, fine, but they will have to pay for it. Mainly money, but to some extent also time.

At the end, it is a team effort, and the goal is the project success. The goal is not to sink thousands of dollars only to finally fix the blame. That does not help anyone.

Answer 3

I'm a bit worried that it's the program manager, not marketing, who is continually adding new features. But in a sense, that's not your problem.

Whenever the manager comes up with a new feature request, try to estimate not just how long it will take to code up, but add in a generous margin for bug fixing and reversion testing caused by the change. Anecdote: a few years ago, I was regularly asked by my then program manager to estimate the time it would take to add features. Some time later, he told me that he always doubled every estimate I gave him. I was only looking at how long it would take to design and code up, not how long it would take to integrate, test and debug.

If you're not already doing so, create a database of all known bugs. For every bug, keep track of how long it took to fix. Monitor how the number of bugs decreases over time. From these two bits of information, you should be able to work out how long it will be before the number of outstanding bugs is trivially low.

If the number of bugs isn't noticably decreasing over time, then it's so bug-ridden that it's nowhere near ready to ship.

Answer 4

I think most of the people in software development will sooner or later go through a project like the one you describe. The good thing I can tell you: You'll learn a lot. Once you went down the hard road, you'll see things differently and your next project will already be better.

Here are some thoughts what you can do now:

• It seems like your project manager has a business background and no experience with software projects. (These people use to think that programming is the same kind of work like e.g. loading parcels to a truck.) Sit down with him and explain that there's no such thing as "bug-free code" or "100% test coverage". Also explain that estimation in software development is not easy, because unlike the truck worker, you often have to estimate something which you have never done before.

• You have to stop feature creep or you'll never be ready. Explain that each new feature will also slow down development of the other features, since you have complex dependencies (I guess that's the case for your system)

• Your project manager has to clarify what's more important: the release date or a complete feature set? I experienced both situations, depending on the industry. But you have to make clear that this decision has to be made.

• If people are not willing to give priorities, do them on your own. Of course it would be much better if priorities would come from the "product owners", but in my experience it is better do prioritize on your own than having no priorities at all (70% priority 1 is pretty much the same as "no priorities at all")

• You have to include a buffer in your estimations for bug fixing etc. Sure you don't know which bugs will happen, but you do know that some bugs will happen for sure. For your estimation Excel sheet, make sure you include everything which consumes time, lets say: feature development (this is what you already have), meetings, bug fixing, testing, release / deployment work, system administration (if you have to), holidays, sickness leave, phone conferences, plus: a buffer for estimation errors

• Grab a book about software project management (e.g., something about Agile, or "Ship It" from the Pragmatic Programmers is good for the beginning). This is more for the long run

Answer 5

In some ways this is your failing for not correctly "managing the client". You say they have expectations that you find unreasonable, and that's the core of the problem.

Most companies and people have no problem spending money if they think they are getting value from it. In most cases this is limited by budget, but a company will spend $2,000 if they think there is value and bawk over $2 when they think there isn't. This is fair, and it's how it should work.

Now to correct your problem, your need to do a few things.

First blow your own horn a bit. List out all the feature creep you have done and and show it off. Let the customer know that the time and money they spent has value. If they can see the value they will be happier with the time frames.

Second, finish your tasks on time. Your failing to do this. With good cause mind you, but still it's a failure. If your were supposed to make a widget by Wednesday then finish the widget by Wednesday. If your done with the widget on Tuesday then mark is done. When the client comes back and says "you forgot to implement this feature we didn't tell you about" then track that as a new thing and stick a new date on it. The important part here is to show that your getting done what you say your going to get done when you say your going to do it. It's all perception, but by not putting a close (or whatever) in that column, your looking bad.

Third, try to change the words being used. I try to use "issue" and "error". Errors are things that produce error codes. And if you have a lot of these then your writing bad code. Issues are everything else. This allows "you forgot to implement a feature we didn't tell you about", to be a issue and not a bug. It still gives it importance, but does not make it sound like bad code.

Mostly what you have here is a perception failure. Your failed to set expectations and now you need to go back and do so. This can be quite difficult, but rewarding. Be careful to set expectations that you can meet. And keep in mind that the client is not going to want to move some of his expectations after this long. Also consider hiring someone external to the project to do a code review or something of that nature. Having a third party review the situation usually helps. It has it's own downfalls, but if you written good code, having a third party agree and state so, will help the client feel better about the situation.

Answer 6

I had one manager who I liked that described a project as "pushing a wall forward". That is, for most of the project there's an almost overwhelming span of tasks that you have to push forward at more or less the same rate -- like pushing a wall.

But at some point, he would say, you hit a critical threshold, and then the wall just sort of disappears, and you're done.

I've also heard it in the sense of "converging" and "diverging". Diverging means the scope of the project is expanding, and converging means the scope is winding down and narrowing. That's a natural process for projects -- you might be able to find some literature describing it online.

One thing to watch out for is that converging and diverging are also personal conversation style -- some people in conversation like to think more and more broadly, loop in more and more related ideas, and hate settling on answers. Convergers are the opposite -- they like to tie together ideas, summarize, and come up with clear answers. Convergers and divergers, as you might expect, can drive each other crazy.

So you might watch out that you have a diverger somewhere in the project definition, who is dissatisfied with having an answer. Apart from that, it may be that you're still in the natural process of discovering necessary work -- before you hit the crossover point where it all starts to wrap up.

Answer 7

You need to freeze on a set of requirements that you are going to work on and give the ETA for it and count the time for bugs and some additional buffer time while giving the ETA. Any changes that come happen after the release of the set of requirements that you froze upon or they should go to the manager who would then request you to prioritize and give a new ETA.