18

I am the Director of a small startup organisation. We currently have two programmers (one experienced, one less experienced) who are building a web application platform.

One of the greatest challenges so far is the planning process. The programmers usually are in charge of planning their own work, but we keep exceeding their self-imposed deadlines. For instance, a task that they estimate will take 2 days, ends up taking 8 days.

For me it is hard to support them in planning, as I lack the technical know-how to precisely estimate how long a certain task will last.

Do you have any idea:

  1. What is the reason for this, is this common for programmers?
  2. What I can do to support them in planning? Are there any methods or tools that are useful to programmers in small teams?
gnat
  • 21,442
  • 29
  • 112
  • 288
John B
  • 193
  • 6
  • 2
    Do you have any advisors, if not find some. You will need to know what the status of the work is fast. If you cannot check for yourself you need help as a director to oversee it. Planning is always hard for development (look for topics here, you will find plenty). Do you have a good idea of the quality of the product being developed? The main issue is you cannot know if you are not a developer or at least experienced in it. – Luc Franken Oct 15 '13 at 10:48
  • @LucFranken We do not have any advisors yet. Fortunately the quality of the product seems good, it is just that deadlines are often not made. Thanks for the tip, I will look into getting an external advisor. – John B Oct 15 '13 at 10:52
  • 3
    @John B, You could take a look at similar questions here (http://programmers.stackexchange.com/questions/tagged/estimation?sort=votes&pageSize=15), but the fact that you're non-technical will eliminate most of them as helpful ones. But these ones might be helpful: http://programmers.stackexchange.com/questions/16326/how-to-learn-to-make-better-estimates, http://programmers.stackexchange.com/questions/39468/how-possible-is-it-to-estimate-time-for-programming-projects, http://programmers.stackexchange.com/questions/208700/is-it-okay-to-add-time-for-unexpected-delays-to-a-time-estimate – superM Oct 15 '13 at 11:04
  • With quality I also mean the internals, not only the visisble output. Sometimes delays come from internal issues in the project. – Luc Franken Oct 15 '13 at 11:04
  • 1
    @superM Thanks so much, this is very helpful. Several threads are very useful to me as a Director, and others I will share with my programmers to see if they find them useful as well. Thanks for your help. – John B Oct 15 '13 at 11:14
  • @LucFranken I will ask an external professional to scrutinize the internals as well - I will take your advice on board.. – John B Oct 15 '13 at 11:14
  • 2
    There is a very good book on this topic: Mike Cohn´s Agile Estimating and Planning. http://www.mountaingoatsoftware.com/books/agile-estimating-and-planning – Hbas Oct 15 '13 at 15:31
  • 3
    I'm surprised no one has linked to this yet: http://www.joelonsoftware.com/items/2007/10/26.html – paul Oct 15 '13 at 20:28
  • 1
    Its the 90-90 rule http://en.wikipedia.org/wiki/Ninety-ninety_rule – exussum Oct 15 '13 at 21:07
  • Also related -- [Hofstadter's Law](http://en.wikipedia.org/wiki/Hofstadter's_law): "It always takes longer than you expect, even when you take into account Hofstadter's Law." – Ben Lee Oct 21 '13 at 21:33

6 Answers6

20

What is the reason for [their estimate of 2 days taking 8 days], is this common for programmers?

It is, if:

  • It isn't actually clear what they're supposed to do, so they take more time getting it right (and they should then say so, not guess about how long it will take)
  • They're unfamiliar with the task at hand (then they should mention that and include research in the estimate)
  • The integration of the finished task with the larger product takes longer than anticipated (which might mean the product's architecture is inferior)
  • The developer likes to reinvent the wheel, and doing so stumbles over problems that have been solved by others, freely available in a library
  • Changes are introduced by the product owner while the task is being implemented, requiring a different approach and the redo of work already done
  • The developers don't work in a productive environment (so when at home they guess it would take two days, but at work they'll require eight to compensate for all the distractions)

To name a few things.

Perhaps it's best to ask your developers why they think their estimates are way off. :-)

CodeCaster
  • 2,394
  • 1
  • 15
  • 20
  • 1
    Thanks for posting this answer. We will keep this list at hand as a check list to ensure that there is a minimum of 'unknowns' in our planning process. I think it will also help the programmers to read this list. P.s. I would upvote it if I could, but as I am new I do not have enough reputation points yet :) – John B Oct 15 '13 at 11:17
  • 1
    You're partially right, though I don't think a competent programmer would be beyond improperly estimating the time required to do a project. As such, I think you should always plan for [Hofstadter's law](http://en.wikipedia.org/wiki/Hofstadter's_law), even when all aspects of this list are observed. – Neil Oct 15 '13 at 16:23
  • 1
    Insufficient knowledge of the code base may also be a contributor to false estimates. – TheMorph Oct 16 '13 at 02:50
16

The general techniques are somewhat common-sense, the important thing to know is that they don't require much technical expertise.

The starting point with planning is to identify the exact problem that needs to be solved and have a clear and unambiguous requirement. If you don't have that, your estimates will be incorrect. Having this documented in some kind of feature specification before anyone starts writing code will mean that any questions that need to be asked will have been asked before coding begins. This is a surprisingly effective timesaver. Going back and clarifying requirements breaks up one's flow as a programmer and waiting for responses can block progress.

Once you have identified the requirement you need to identify the work tasks involved in resolving it. This is a classic divide and conquer exercise - any task that can be broken down further needs to be broken down further.

In a larger team you can use estimation poker to get an estimate based on the experience of everyone involved. That doesn't work as well in a smaller team, but it is still useful to get an independent estimate from both your developers and maybe to include one from yourself too- your lack of specific expertise can be helpful here because in explaining to you what the task involves from their perspective, the development team will probably grasp the problem better.

With a smaller team it can help to get a best/expected/worst case estimate for each task, that gives you a range of values, but if you are getting a lot of overrun estimates, you can incline towards the worst case until your devs learn to estimate more accurately.

In a small shop, developers often end up doubling as sysadmins, support team and even testers ( although of all the things they could be doing, testing is the one you should try to avoid at all costs ) so you need to account for that. Figure out how much of their time your developers actually spend working on new features and factor that into your estimates. If a task is estimated at 2 days but your devs are only able to work on new development 60% of the time, then you are going to need 4 days for it to be finished. You might be able to help with this as well by controlling the pipeline of other tasks they need to handle so that non-urgent admin or support tasks can be batched together somewhat rather than being handled on an ad-hoc basis. A lot of programmers ( certainly including myself on this one ) are not great time managers, so anything you can do to lend a hand in that respect will help. Single-tasking is always easier for programmers than multi-tasking. Blocking out time during the day can also help with this.

Keep a record - each time you have a planning session, record the estimates and actuals. You can then use this a) as a guide for how much to inflate their estimates by during planning and b) to help them refine their estimation skills. At the end of each iteration ( or whatever equivalent you have ) the whole team should review the work that was done and figure out why it took longer than expected so that this can be incorporated in future estimates. This needs to be a blameless task - you seem to have the right attitude here but this answer may be around a while so I'll make the observation. If someone says "I made a mistake here" you can turn that into "what could you have done better", but telling people they were too slow or got things wrong will only make matters worse.

I am aware of no silver bullet for this type of problem but the biggest factor is communication- which is actually easier with a smaller team - and using feedback to refine your collective skills.

glenatron
  • 8,729
  • 3
  • 29
  • 43
  • Thanks, this is very useful as well. Keeping better track of the estimated and actual delivery time is something that we have done in the past, but possibly insufficiently due to work pressure. We will start doing that in a more structured manner. Also, we will try to streamline the communication between the developers and the managers even more to make the process easier and save time. We found that often there are differences in the way managers and programmers communicate, which can lead to misunderstandings and therefore sloppy planning. – John B Oct 15 '13 at 11:22
  • 1
    @JohnB this is precisely right. If there was a way to have _completely clear and unambiguous_ communication between devs and managers software projects would always run smoothly. Unfortunately, that isn't how humans work... – glenatron Oct 15 '13 at 12:41
  • If you want even more information towards this, you can read a good text about Scrum, as for example the estimation (/planning) poker and the review part glenatron mentioned are part of it. – TheMorph Oct 16 '13 at 14:17
6

You wouldn't be the first by a long shot to try to figure out the best way to plan development time. This is in part due to the fact that it is difficult to quantify something you cannot actually see being built, and partly due to the mythical man-month, which is a direct contrast to the intuitive idea that if you have 2 programmers, you should be able to develop twice as fast than if you had 1 programmer.

As you've probably realized already, it is a lot more complicated than that. One approach to estimating development time as rounding a group of highly qualified individuals for what concerns software development and ask them to estimate the amount of time it would take to finish a project (explaining as detailed as possible). You take the highest of all the estimates and you double it. Yes, you read correctly. You double the highest estimate and you shall have a reasonably accurate estimate. I know, because with time, this is how I've been able to accurately tell my bosses how long it takes me to do something. I gather the opinions of my fellow programmers and my own and double the highest estimate. If this seems like too high of a value, consider that testing new functionality is crucial and consider potential bug fixes afterwards, and it will seem like a more reasonable figure.

In my own personal experience as a programmer, I can tell you that it helps to break down projects into milestones. How long does it take to reach milestone 1, then from milestone 1 to milestone 2, then milestone 2 to milestone 3, etc.? When broken down like this, the answer is usually far more accurate than trying to estimate the entire project in its entirety. Strangely enough, if you sum up the estimates of all these milestones, it will usually be larger than the original estimate on the entire project (if the programmer is being honest with himself anyway), which only leads me to think that detail is the key here.

Perhaps you lack the technical know-how, but you should still try to follow on a more general level. Programmers have no problems with repetition. It's the twists and turns that occupy all the time in developing a program. So most likely, the more functionality that you wish to include, the more complicated the program is going to become, and assuming this new functionality has no influence on previously implemented sections of the code, development will be linear according to the amount of work to be done. Most likely, new functionality heavily influences previously implemented sections and thus development involves not just implementing the new functionality but fixing the old code, making development time exponential.

My advice to you would be to, without telling the programmers how to do their job, try to get a handle of how the program works at a general level and you should soon be able to see how new functionality would modify that behavior and thus provide a reasonable estimate on how long it would take to do it. Combine this with their estimates (double the highest), and you start getting a better idea of how to estimate development time.

I hope that helps!

Neil
  • 22,670
  • 45
  • 76
  • Addendum: Programmers have few problems with _estimating_ repetition. I, at least, have lots of problems with boredom from repetition (which sometimes leads to a rabbit hole of automation, a short-term time sink that might pay off long-term) ;) – Izkata Oct 15 '13 at 15:13
  • 3
    @Izkata, if programming were about copying and pasting, I wouldn't be in this business. It's the *lack* of repetition that I enjoy about my job. :) – Neil Oct 15 '13 at 15:19
6

One of the reasons estimates are often way off is because estimation is actually quite hard and requires experience and knowledge about the system to be changed. Most often it is helpful to break big steps up into smaller ones.

Now you have many possibilities of those I will mention two:

Planning Poker

This works quite well in small agile teams.

Excerpt from wikipedia:

  • A Moderator, who will not play, chairs the meeting.
  • The Product Manager provides a short overview. The team is given an opportunity to ask questions and discuss to clarify assumptions and risks. A summary of the discussion is recorded by the Project Manager.
  • Each individual lays a card face down representing their estimate. Units used vary - they can be days duration, ideal days or story points. During discussion, numbers must not be mentioned at all in relation to feature size to avoid anchoring.
  • Everyone calls their cards simultaneously by turning them over.
  • People with high estimates and low estimates are given a soap box to offer their justification for their estimate and then discussion continues.
  • Repeat the estimation process until a consensus is reached. The developer who was likely to own the deliverable has a large portion of the "consensus vote", although the Moderator can negotiate the consensus.
  • An egg timer is used to ensure that discussion is structured; the Moderator or the Project Manager may at any point turn over the egg timer and when it runs out all discussion must cease and another round of poker is played. The structure in the conversation is re-introduced by the soap boxes.

The important bits here are clarification, discussion, simultaneously calling the estimate, so as no bias is introduced, and consensus.

PERT

It is often hard to give one exact estimate. What is easier is to give a likelihood. PERT uses 3 values for the estimate:

  • most optimistic time to finish (if less problems arise than expected)
  • most pessimistic time to finish (if everything goes wrong - excluding major catastrophes)
  • most likely time to finish (if everything goes as expected) <- this is what your developers most likely estimate right now

By weighting those three estimates you get a more reliable estimate.

t_expected = (t_opt + 4 * t_likely + t_pes) / 6

And / or you are able to use these three values to construct a probability distribution that may reflect the uncertainty of the real world even more. Beta distribution and Triangular distribution are prominent choices. Using this you can now apply statistics like "how likely is it to finish at date x given current estimates" or "if i want 95% certainty, at what point in time will it be finished".

Actually PERT consists of more than these aspects mentioned here, that I omitted for the sake of brevity.

TheMorph
  • 414
  • 2
  • 8
2

It is a fact that if you aren't keeping historical metrics then you can't even come close to giving reasonable estimates with a reasonable degree of accuracy. And asking some other company/person how long it wil take them doesn't help either. The problem is that each company and developer has their own way of doing things. Thus, every company will have different lengths of time for doing the exact same task. Each developer will have different lengths of time for doing the exact same task.

Your best course of action is to start keeping track of time and somehow figure out how to categorize the degree of difficulty for the task. Some companies use lines of code, some use features, some just go by gut feel. Additionally, you also have to take into account if this is similar to something the developers have already built or something new, like new technology, new feature never built before by the team etc... Also, if it is an embedded or real-time system then complexity generally goes up quite a bit.

Unless you collect real data, no matter how many times your developers give you estimates they will really just be pulling numbers from their behind each time you ask them. Yes, collecting real data is a pain and at first it doesn't provide a lot of useful info, but over time it really does begin to provide reasonably accurate estimates.

I would also like to point out that estimates are generally only good over the big picture and not for short term measurements. For example, the developer estimates 2 days but it takes 8. Well the developer didn't account for having to setup a test environment and developing a simulator or that there was an entirely new technology they had to learn or they got stuck on a bug they couldn't figure out or the functionality required a refactoring of the existing system. You can't always predict those kinds of things for small tasks. However, over the course of an entire project those extra 6 days may get washed out by other tasks taking 6 days less duration.

Dunk
  • 5,059
  • 1
  • 20
  • 25
1

I have been a sole developer on a few small projects off my own and have some industrial experience working with a large team. I have noticed that the techniques that a big company uses does not necessarily work for a small team. At one point I was doing more planning and documentation rather than writing code. I suggest that you try to find a good way of working first by trying different techniques (the other answers provide some great insight) and tools, this will cost you some time and effort but you will benefit from it later on. Some tools/ techniques I found to be useful were:

-Pivotal Tracker - Great program for keeping track of stories and encourages breaking down -tasks, it is lightening quick at entering stories and automatically deduces velocity. https://www.pivotaltracker.com/.

-Gdocs for documentation as it is easy to have multiple users editing and discussing at the same time.

-At a company I used to work for we used to have a meeting for each and every story that we initiated, this meeting had to include a senior programmer since he would be better at judging how long a task would take. He would also be better at judging what the difficult part might be in a task.

In summary I believe that the key to working in small teams is to have a solid planning regime that is fast and fluid. Also any difficulties with the story can be identified early so that planning a task will keep that in mind (this could lead to building something differently).

Hope this helps

Edmond Chhung
  • 11
  • 3