52

I am a developer working on a new mobile app for Android and iOS with a big backend component. We have been in three sprints of this project, and we use Scrum with all of its ceremonies (refinement, planning, dailies, retrospectives, etc.).

In two of the sprints the team had to work (unpaid) overtime and weekends, because management were very alarmed we wouldn't complete the sprint commitment on time. Everyone worked hard, but some external dependencies and optimistic estimations made us struggle to accomplish all the sprint stories.

In my experience having a small percentage of stories not completed during some sprints is somewhat normal, and they can be tackled in the next one. But our project manager says it is our fault as we made the estimation ourselves, so we should complete all the items in the sprint.

  1. Is this an acceptable/common variation of Scrum I am not aware of?

  2. How do you suggest that I should act on this?

Peter Mortensen
  • 1,050
  • 2
  • 12
  • 14
MrAn3
  • 645
  • 1
  • 5
  • 5
  • 2
    Does your company have any contractual obligations against 3rd parties, where not fulfilling the sprint goals would end up for them in loosing money immediately? – Doc Brown Sep 20 '19 at 06:41
  • 66
    .. moreover, when your working contract allows unpaid overtime working and weekend working, and your superiors can order you to do so at their own will, then I fear the best course of action is either to add a safety margin of at least a factor of 2 to each sprint eastimation, or to find a different employer. – Doc Brown Sep 20 '19 at 06:44
  • 59
    No this is not an acceptable practice since the abolition of the roman galley. This is a typical panic attack of a project manager whose competence could still developed further. Kicking people in the ass assuming that they do not the best without challenging estimates and situation will not help out of this mess. But this issue is far too broad to be discussed here... – Christophe Sep 20 '19 at 07:07
  • If this is the management's strategy, you counter-strategy should be to only ever deliver "at least two more sprints" as an estimate. – Jacob Raihle Sep 20 '19 at 08:28
  • I don't quite understand why this is considered too broad. – George Stocker Sep 20 '19 at 17:01
  • 27
    Ask yourself what the management view would be if you completed your sprint "commitment" ahead of time. Would the team get the rest of the sprint off (including being paid)? – Qwerky Sep 20 '19 at 17:36
  • @Qwerky OP's PM probably will just force the next sprint to start ASAP (moving the start date eraly)... – Ivan García Topete Sep 20 '19 at 18:42
  • 14
    A Project Manager is not normal in Scrum. The Scrum Guide is quite explicit on the roles: Scrum Master, Product Owner, Scrum Team. No PM mentioned. In fact, many people (including most of the signers of the Agile Manifesto) have repeatedly claimed that projects are detrimental to agility. Also, you are the only one who decides that's acceptable. If it's not acceptable to you, polish your CV and look for a better company. – Blueriver Sep 20 '19 at 18:42
  • 5
    Two things: Commitments are made by the team, not the PM, so commit to less as the immediate fix, however the bigger problem is what happens if you don't deliver? What is the consequence? Typically PMs, TPMs, Scrum masters, product owners, etc... are encouraged to work with the team because... they have no real authority over the team in the matrix structure Agile/SCRUM lends itself to. So this may be nothing more than an @ trying to advance their career at the expense of others. – RandomUs1r Sep 20 '19 at 19:02
  • 2
    Is it normal? It's somewhat normal for managers, not normal for scrum. – Bryan Oakley Sep 20 '19 at 21:58
  • 1
    You don't "commit" anymore, you forecast: https://www.scrum.org/resources/commitment-vs-forecast – Andy Sep 20 '19 at 23:58
  • 4
    Accurate estimation is hard. Punishing incorrect estimations is toxic and encourages people to give estimations most likely to avoid punishment, not ones they think are accurate, or they may fear giving any estimation whatsoever. – Bernhard Barker Sep 21 '19 at 03:01
  • 3
    Your manger pressing you for unpaid overtime is not solved by discusssing Scrum practice and defining terms. – eckes Sep 21 '19 at 14:50
  • This doesn't look like Scrum to me. Anyway You have learned, for this manager make pessimistic estimations. – Robert Andrzejuk Sep 22 '19 at 09:49
  • I'd think long and hard about working for any company that predicates their business model on stealing time from their employees. – spender Sep 22 '19 at 14:20
  • Deadlines are always artificial, whether you wrap them in "scrum" fluff or not. And estimates have a margin of error of a factor of 2 both sides at the outset of a project... (so worst case is roughly 4x as long as best case) see "cone of uncertainty" – Bradley Thomas Sep 23 '19 at 01:41
  • 2
    Quite common to here companies talk about being "agile" and using scrum, and have no idea what it means... Sounds like you can't fix that kind of broken. If a "heroic effort" is a regular requirement of your job, you're working in the wrong company. – angryITguy Sep 23 '19 at 06:30
  • 1
    An estimate is an estimate. Predictions are easy. Except for those about the future. – Michael Durrant Sep 23 '19 at 11:00
  • I would also argue, if they want you to work extra if you don't finish in time, will they also let you go home if you finish before time? Probably not. That's why the agile is assuming normal working hours. – Viktor Mellgren Sep 23 '19 at 12:51
  • Thank you for all your comments and answers. I thought that maybe I was being overreacting for being frustrated about this situation. – MrAn3 Sep 23 '19 at 21:12

10 Answers10

77

A few things stand out to me.

The idea that management has that the team commits to a set of work is inconsistent with the latest versions of the Scrum Guide. The word "commit" or "commitment" is only used twice in the most recent (November 2017) version of the Scrum Guide - once when listing the Scrum Values and once to indicate that "people personally commit to achieving the goals of the Scrum Team".

The idea of goals is important to Scrum. Not only do organizations and teams have broad goals, but in Scrum, each Sprint has a Sprint Goal that is defined at Sprint Planning as a collaboration between the Product Owner and the Development Team. The Sprint Goal is met by implementing items from the Product Backlog, but it is not simply "finish this body of work" and it often doesn't represent the complete Sprint Backlog. That is, you should be able to achieve the Sprint Goal without completing every single Product Backlog Item selected for the Sprint or every single item in the Sprint Backlog. My current thinking is that the work needed to accomplish the Sprint Goal should be somewhere around 60-70% of your team's capacity, however you account for capacity. Different organizations will be different, though, but that's likely to be a good starting point.

Working overtime and weekends is also an anti-Agile Software Development practice. One of the underlying principles is that all stakeholders of an effort are able to work a sustainable pace. Long days and weekends, even if they were paid, is not sustainable for a team.

At this point, there are a few next steps. The team's Scrum Master should be educating management on the values and principles of Scrum and Agile Software Development (such as "commitment" and "sustainable pace"). The team should work on its ability to forecast work and negotiate scope with the Product Owner. The team should also identify and work toward resolving or preventing the impediments that led to this situation (eliminating or reducing the impact of external dependencies).

Thomas Owens
  • 79,623
  • 18
  • 192
  • 283
  • 2
    Great answer - you could even improve it by highlighting or TL;DR ing the most important points: commitment can only come freely from the developer himself and work needs to be **sustainable** – Falco Sep 20 '19 at 15:07
  • Also, if you have delays due to dependency from other teams, the amount of time you are blocked should be visible by your team. – luizfzs Sep 20 '19 at 16:03
  • 2
    I believe they changed the wording to 'forecast'; much like a weather forecast, it can be wrong, it has levels of certainty, and the stories completed in a sprint help the team get better at estimating in the future. – George Stocker Sep 20 '19 at 17:02
  • 1
    @GeorgeStocker Yes - the word "forecast" is used instead of "commit" with respect to the Sprint Backlog and specific work items. However, "commit" and "commitment" is used with respect to the goals of the team. – Thomas Owens Sep 20 '19 at 17:04
  • Education about 'sustainable pace' would be good. However it is not specific to Scrum, Agile or software development. It may easily backfire too. A sustainable pace has _always_ been a concern, whether it is bolting together a car or making software. This is a fact of capitalism. Management is (or should be!) well aware of this fact. However pushing workers to produce more for less cost is basic in most companies although in software this model is increasingly being seen as outdated and inefficient for an industry that relies on creative work (programming). – Michael Durrant Sep 21 '19 at 12:10
  • So basically the argument for sustainable pace may be best made from how it affects the software quality itself (affecting customers and revenue) and less about how it affects the standard of life for the workers. Unfortunately. Longer term and more visionary leaders (less driven by quarterly results often) can change this. – Michael Durrant Sep 21 '19 at 12:15
  • @MichaelDurrant You'd think that's the case, but every time I've seen a fixed schedule project start to slip late, the first two responses have been "work more hours" or "add people", neither of which work (for different reasons). Negotiating schedule or scope usually comes after those two are (hopefully) rejected. I do agree that the best case is around product quality, but I wouldn't ignore quality of life (and implications for turnover) of employees as a secondary concern. – Thomas Owens Sep 21 '19 at 12:59
  • @ThomasOwens I agree, I have seen the same. I think quality of life and turnover should be primary concerns right up there with short term business goals, but I also don't experience it at most companies unfortunately. There have been a few rare exceptions. – Michael Durrant Sep 21 '19 at 13:35
  • 1
    and also yes, 9 women can't make 1 baby in 1 month :) – Michael Durrant Sep 21 '19 at 13:36
33

The situation that you describe, where management requires that the team works overtime to complete all planned stories, is one of the reasons why the Scrum literature has stopped using the term "commitment." A true commitment can only be given when all uncertainty is taken out, including uncertainty about 3rd-party dependencies, how much work each item is, how much time everyone will be available taking sickness into account, etc.

One of the basic ideas of Scrum is that the team works at a sustainable pace, which essentially means working normal hours without overtime (paid or unpaid). This directly means that you are not experiencing an acceptable variation of Scrum.

What you can do about it depends on your role.

If you are a developer, then the best you can do is

  • (collectively as development team) refuse to "commit" to more work than you are absolutely certain you can deliver within a sprint. This will probably be way less than what management wants you to do.
  • keep focus on finishing work, rather than starting on new tasks. If you see that some work can't be completed, indicate this as early as possible so that the plans can be adjusted.

If you are a Scrum master, then you can really prove your worth by educating the management about Scrum. Some points that stand out to me:

  • As stated in the first paragraph, the team does not give a commitment during the sprint planning, but they give a forecast of the work they expect to have finished.
  • Although the team should avoid having unfinished work at the end of a sprint, this situation is nearly unavoidable at the beginning of a project (or after a change in the team's composition). How much work the team can realistically accomplish in a sprint can only be determined based on historical figures of the last few sprints of the team in this composition. Early in the project, such historical figures just don't exist, so a team accomplishing in the first 3 sprints exactly what the planned for each sprint is more accident than good planning. After about 5 sprints at a sustainable pace there is enough data to get a reasonable idea how much work the team can realistically finish within a sprint.
Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
Bart van Ingen Schenau
  • 71,712
  • 20
  • 110
  • 179
21

Your PM has no business being involved in your scrum.

Your PM has no business asking you for unpaid overtime.

The obvious thing to do is to estimate all tasks in such a way that you can guarantee they will be finished in time. Then you should be able to go to the pub in the second way, since clearly if underestimating a task means you finish it for free, then overestimating means you have time without work.

Kyralessa
  • 3,703
  • 2
  • 19
  • 20
gnasher729
  • 42,090
  • 4
  • 59
  • 119
  • 1
    "Your PM has no business being involved in your scrum." Under certain Agile methodologies (i.e. DSDM), they do. Pure Scrum doesn't even recognize "Project Manager" as a role that exists. – nick012000 Sep 20 '19 at 14:58
  • 3
    If the working contract says there can be unpaid overtime, the PM surely has a business in asking for it. And if the team is done earlier than estimated, that's again a "mistake" of the team so no reason to get lazy afterwards, better start estimating for the next sprint so the estimations are not as far off^^ (speaking in the PM's logic). Not that I'm agreeing with the way management handles this, but your arguments don't hold either (except for the PM being involved in scrum, depending on some constraints - as a stakeholder, he can be involved, just not in the way he currently is). – Frank Hopkins Sep 20 '19 at 20:35
  • The other logical response to being forced to commit to estimates is to schedule time to research all the unknowns before you can estimate the actual task. – Robin Bennett Sep 23 '19 at 08:12
  • I've never worked anywhere were scrum/agile is run the same way, but in broad strokes, while the PM is not recognised as a specific role, they often manage the budget and risk. The corollary of this is that they absolutely **do** have a vested interest in how well/badly the development is going and they **can** ask for unpaid overtime albeit in a good-will arrangement. How this is facilitated within the team will vary massively from shop to shop. In some places, they are strictly hands off - ceding to the scrum master, in others they participate in the stand up (contrary to accepted practice). – Robbie Dee Sep 23 '19 at 09:41
  • DSDM is not an agile methodology, its a steaming pile of **** ****** **** that *** *** ******* that managers like because it gives them a lot of involvement in the process. – gbjbaanb Sep 23 '19 at 09:44
12

There are a number of aspects to this, but at a high level, yes - the PM will absolutely want to clearly understand why the planned work has not been completed. However, this should be brought up (and resolved) in the retrospective. From the dev side, there are many factors that can contribute to sprint failures.

Some things you may want to consider:

Too much in the sprint

If you are regularly committing to too much work, then sprints will fail. The sprint velocity should be tracked over time to find out what the optimal number of points (or days) is.

Resource allocation

Ensure that sprint planning adequately accounts for non-development activities like the ceremonies, holidays, training, admin, support and other projects etc. Automatically assuming everybody is developing every minute of every hour they are physically in the office is just not practical and will immediately put you on the back foot from the get go.

Estimate variation

You're doing refinement, but are there certain sorts of tasks that always overrun? Commonly these are down to missing or vague requirements. If the requirements are woolly the story should not even make it into the sprint unless it has been adequately refined or a spike has been planned.

Velocity

If the velocity is being properly tracked, the true number of stories should become clear. That isn't to say they'll always be done in time but it should make things a lot easier.

Good will

On any project, good will is limited. If you're constantly working out of hours to deliver, morale will suffer and devs will burnout - this is a project management failure. As I've already outlined, make sure sprint planning only schedules a realistic number of stories using velocity and spikes to help you along the way.

Spikes

If an item is badly refined or just plain woolly, don't be afraid to put a spike in to provide a better estimate for later sprints. Yes, some people are just bad at estimation but most of the time, the full facts just aren't known at the time. Ideally, this should have been covered in the refinement or picked up early by the PO, but sometimes they can slip through into a sprint. Developers should be pushing back on these hard as these can easily torpedo a sprint that is otherwise going well.

Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
  • 2
    I'm not sure that "push back from the PM" is the most useful framing. The entire team, as a whole, should want to improve their process—that's what the retrospective is for—and all of the issues you've identified are great things to consider as part of that discussion, but I think it's most useful to think of it as "how can the team help ensure that the estimates provided by sprint goals are more useful in the future?" rather than a PM pushing back on the team for not accomplishing tasks. – Zach Lipton Sep 21 '19 at 00:20
  • 1
    I think you get to the heart of the problem. The PM _has_ to bring this up its vital to understand why the project is late, but the #1 reason is going to be 'estimates were wrong' for whatever reason. (and #1 reason for that would be PM didnt like high estimates!) – Ewan Sep 21 '19 at 08:50
  • To me, this is clearly the best answer so far. +1 – angryITguy Sep 23 '19 at 06:32
  • How about we refer to 'pushback' (which implies a potentially antagonistic approach) as "questions" which seems more neutral and effective to me? – Michael Durrant Sep 23 '19 at 11:05
  • 1
    @MichaelDurrant et al. Fair enough - I've modified the wording of the first paragraph. – Robbie Dee Sep 23 '19 at 11:50
10

No, this isn't a recognized form of Agile, for one very important reason: if you're committing to deliver everything, you're not doing Agile, you're doing Waterfall - and if you think you're doing Agile instead, you're probably doing Waterfall poorly, at that. I'm sure you've heard the old saw "good, fast, cheap, pick any two," right? With software development, it's more like "all features delivered, on time, on budget, pick the first or the other two". Waterfall picks the first, and Agile picks the second two.

If you're going to be Agile, you need the flexibility to drop Stories from the Sprint that you're unable to complete on time. One possible way to do this is to ask your Product Owner to assess the stories using MoSCoW prioritization. This would involve selecting no more that 60% of the Stories (by total Story Points) as Must Haves that will be completed, 20% as Should Haves that you should complete by the project concludes and the Minimum Viable Product is released, 20% as Could Haves that might be completed if you have the time, and anything outside the scope of the current release as Won't Haves. It's also important to note that when combined, the Must Haves should have enough meat to their bones to create the Minimum Viable Product without needing to include any items from the other categories.

Determining whether or not to drop items from the Sprint Backlog is the responsibility of the Product Owner, after it has been requested by the Team, and any items that get dropped from the Sprint Backlog get assessed by the the Product Owner, and then are either dropped from the project entirely, or placed on the Project Backlog in an appropriately ranked position.

In this case, I'm guessing that your Product Owner is your Project Manager, right? It would be his job to determine which tasks to drop, so he certainly shouldn't blame you for overcommitting, since it would be his job to drop the tasks to compensate for that, and it appears that he isn't doing so.

nick012000
  • 224
  • 1
  • 8
6

He is correct, that there should not be any carry-over between sprints. Scrum teams having a carry-over between sprints is an anti-pattern and not something that canonical Scrum accepts as valid result.

But, his approach is not a good one.

During a sprint, team should constantly monitor work being done and if they can keep their commitment of sprint planning about finishing selected stories. If team identifies, that it cannot finish all the stories, it should meet up with PO and select a story to remove from the sprint. By doing so, everyone STOPS WORKING ON THE STORY, and focuses on getting the remaining stories done. Remember: It is always better to finish one story fully than having two stories half-finished.

The problems of external dependencies and imprecise estimation is exactly why Retrospectives exist. During Retro, the team should reflect and identify these problems. And the team should then find and implement solutions to those problems. Estimations can be made more precise by better knowing the system and plain experience. External dependencies are harder, but not impossible to fix.

Your PM (what is even PM doing on a Scrum team??) should not be allowed by Scrum Master to force you to finish all the stories. Instead, if he has complains, he should keep them for Retrospective. And even better, should be part of solving the problems that prevented the stories from being finished on time.

Euphoric
  • 36,735
  • 6
  • 78
  • 110
  • I don't agree with the "not a valid result" comment. While it's not a _desireable_ result, scrum teams should realize that it's perfectly reasonable for some stories to not be completed from time to time. It certainly shouldn't be a normal result, if you're failing to complete more than a single story it shows you're doing something wrong, but to say its not ever a valid result is a bit strong in my opinion. I would rather have teams that pick a bit too much work to do then pick not enough. – Bryan Oakley Sep 20 '19 at 22:01
5

Is this an acceptable/common variation of Scrum I am not aware of?

No. It's completely wrong. I could maybe sympathize with paid overtime, if the PO made the mistake of giving out the estimates as facts before the sprint end, but unpaid overtime is completely ridiculous and would make me look for another job ASAP.

How do you suggest that I should act about this?

In my experience, people who are that much of the rail won't listen to logical arguments. The only way for them to see how broken it is is show, not tell. So next time when you estimate and commit, commit to the smallest amount possible. Commit to finish a small ticket by the sprint end. One you could do in a day. See how your PM reacts. Then start a discussion from there on what the system is used for and what the system should be used for.

nvoigt
  • 7,271
  • 3
  • 22
  • 26
  • upvoted, though the unpaid over-time perspective can be reasonable if the contract is formulated in that way (and the general pay is accounting for this, i.e. above average - or there are other benefits). – Frank Hopkins Sep 20 '19 at 20:39
4

Generally, at the start of the project, when we decide the velocity of the team, we go for a conservative (lower than usual) number considering the fact that it's a new team, it would take a little time for the team to settle down, understand each other, understand the functional and NFR requirements, etc. Basically, after the first few sprint's we optimise the team velocity and obviously the velocity will only improve from that point.

There is no point in committing a higher velocity at the beginning and stretching the team to achieve it.

One more thing is that, in a one-off scenario, where there is a delivery commitment which cannot be missed, project managers might put pressure on the team for stretching, working late and working over the weekends. But then that must be considered as an exception and not the norm of working. Working like that might increase the velocity on a short term, but in a longer term it would be counterproductive, as it could result in code quality issues, team burnout issues, unhappy team with low motivation, etc.

Peter Mortensen
  • 1,050
  • 2
  • 12
  • 14
Rajesh Nair
  • 41
  • 2
  • 1
    Nice! +1. At the risk of splitting hairs, you don't "decide" a velocity, it is just something that comes out in the wash after a number of sprints but yes, with sprint 0 (or however you number it) - you stack the board with as many stories as you believe are achievable. – Robbie Dee Sep 23 '19 at 09:53
2

Commitment FAQ

Is this behavior normal?

I'm not sure.

Is it surprising?

No. Some people will always understand "commitment" to mean everything in the commitment must be completed.

Is it a good idea?

No. Agile development has sustainability as a central topic: Work only as much/long/hard as you can do indefinitely. That is a sensible idea at most times. (If your management does not accept this, they should not call their organization agile.)

What should we do?

  • Explain that the sprint content is based on an estimate. "Estimate" means that it will only sometimes be correct -- and usually be wrong. When it's wrong, it is sometimes too low and sometimes too high.
  • Explain it is far easier to change estimation behavior than work speed.
  • Explain that when the team is punished for estimating too high, it will just estimate lower and management will lose a lot more progress that way than by pushing some of the content from one sprint to the next occasionally.

The nice thing is: Your project manager will know all these things already and recognize them as being right. It is only that in the short term one may prefer to ignore them.

Lutz Prechelt
  • 1,483
  • 9
  • 11
2

I don't agree with your management team. They should not have made you work overtime to finish the sprint.

However, the idea that commitments are not possible comes from a misunderstanding of software development. I have seen many teams try to predict the stories they can complete in a sprint by the number of story points they have completed in previous sprints. If that was possible it would say that software development is linear, i.e. if I work two hours I get twice as much done as in one hour.

Software development is creative, and therefore, not linear. It is a better practice for the team to tell management what they can do in a sprint and then deliver those stories. If you are consistently missing your commitments you either had no idea of the scope of the story to begin with or you are being pressured by your product owner to take on more.

In the former case, you must make sure that you understand the scope of the story before you agree to take it on. If it is the latter, you have a culture problem and there is little that can be done.

John Douma
  • 129
  • 4