50

This week at work I got agiled yet again. Having gone through the standard agile, TDD, shared ownership, ad hoc development methodology of never planning anything beyond a few user stories on a piece of card, verbally chewing the cud over the technicallities of a 3rd party integration ad nauseam without ever doing any real thinking or due dilligence and architecturally coupling all production code to the first test that comes into anyone's head for the past few months we reach the end of a release cycle and lo and behold the main externally visible feature that we have been developing is too slow to use, buggy, becoming labyrinthinly complex and completely inflexible.

During this process "spikes" were done but never documented and not a single architectural design was ever produced (there was no FS, so what the hell eh, if you don't know what you are developing, how can you plan or research it?) - the project passed from pair to pair, each of whom only ever focused on a single user story at a time and well the result was inevitable.

To resolve this I went off the radar, went (the dreaded) waterfall, planned, coded and basically didn't swap off the pair and tried as much as I could to work alone - focusing on solid architecture and specifications rather than unit tests which will come later once everything is pinned down. The code is now much better and is actually totally usable, flexible and fast. Certain people seem to have really resented me doing this and have gone out of their way to sabotage my efforts (possibly unconsciously) because it goes against the holy process of agile.

So how do you, as a developer, explain to the team that it is not "un-agile" to plan their work, and how do you fit planning into the agile process? (I'm not talking about the IPM; I'm talking about sitting down with a problem and sketching out an end-to-end design that says how a problem should be solved in sufficient detail that anyone who works on the problem knows what architecture and patterns they should be using and where the new code should integrate into existing code)

  • 26
    The first paragraph sounds like a rant against agile. The rest still somewhat sounds like a rant, only against the rest of your team. You may want to paraphrase. –  May 22 '11 at 09:48
  • 1
    +1 very interested in how people solve this in a pragmatic way that lets you take advante of the best both waterfall and agile have to offer. By the way: agile does not equal "no design", but design does tend to be the first victim in the relentless cycle of sprints... – Marjan Venema May 22 '11 at 09:52
  • 5
    Well to a certain extent I've had it up to the neck with "agile". At every turn agile seems to be stopping anyone from writing a decent line of code and it all starts with the agile premise of "we don't document," the corollary of which is "we don't plan". I don't want to hate agile, but as far as I can see so long as it encourages people not to plan their software then it is at best counter productive and at worst dangerous. –  May 22 '11 at 09:56
  • 9
    @Marjan Venema - that is my concern. I'm sure that agile never meant "no design" like "don't prematurely optimise" didn't mean "don't write efficient code". But that seems to be the mass market interpretation of it. The way agile emphasises communication is great, and a really refreshing change, but it seems to me that in agile world the software itself is more of an afterthought. –  May 22 '11 at 10:00
  • 9
    It's obvious that you're frustrated with a poor application of agile principles, but this seems like a rant thinly disguised as a question. To be clear: Agile favors "responding to change over following a plan", which means that "while *there is value in the items on the right*, we value the items on the left more." It is certainly possible to value following a plan *too little*, which seems to be the case here. – Rein Henrichs May 23 '11 at 18:17
  • 1
    Can some of the advocates give us examples of the projects they did successfully in Agile? Did you just start with the users making (only a handful of starter (doesn't agile say something like "don't think too far ahead?)) stories, and then start banging out tests and code? I'm in the camp of not understanding how that works. – Paul Sep 04 '12 at 15:17
  • Agile, but more specifically, **Sprint Zero**, generally requires either a fair amount of experience developing incrementally-developed applications, or the up-front expectations that future sprints will be hijacked by exhaustive re-factoring. Both of these are competing priorities, and both are initiated from opposite ends of the SDLC players: clients and developers. Clients want something tangible fast, and a well-initiated Agile project can take some time to get off the ground in a sustainable way. Experience helps get you sustainable quicker, without too much re-work later. – JoeBrockhaus Feb 19 '15 at 19:34
  • Re-work WILL happen, but you need to insure you do it sooner than later, and set client expectations accordingly. The biggest benefit of Agile becomes apparent once both sides swallow that pill, because iterative conflict resolution improves quality. In this particular case, it sounds like the dev team waited too long to rise issues for fear of upsetting the clients that a sprint would be deferred, and when that goes on for some time, you wind up with a spaghettified codebase, less reliable estimations, and many of the frustrations you experienced. – JoeBrockhaus Feb 19 '15 at 19:35

11 Answers11

51

I think (and I may be going out on a limb here) that ALL projects should have a bit of classic waterfall: The initial analysis and specification phase is essential. You must know what you are doing, and you must have it in writing. Getting the requirements in writing is difficult and time consuming, and easy to do badly. That's why so many skip it - any excuse will do: "Oh we do agile so we don't need to do that." Once upon a time, before agile, it was "oh I'm really clever and know how to solve this, so we don't need to do that." The words have changed a bit but the song is essentially the same.

This is of course all bull: You have to know what you are to do - and a specification is the means by which developer and client can communicate what is intended.

Once you know what you have to do - sketch out an architecture. This is the "get the big picture right" part. There is no magic solution here, no one right way, and no methodology that will help you. Architectures are the SYNTHESIS of a solution, and they come from partly inspired genius, and partly hard-won knowledge.

At each of these steps there will be iteration: you find things wrong or missing, and go fix 'em. That's debugging. It's just done before any code got written.

Some see these steps as boring, or not needed. In fact, these two steps are the most important of all in solving any problem - get these wrong and everything that follows will be wrong. These steps are like the foundations of a building: Get them wrong and you have a Leaning Tower of Pisa.

Once you have the WHAT (that's your spec) and the HOW (that's the architecture - which is a high-level design) then you have tasks. Usually lots of them.

Bust the tasks up however you want, allocate them however you want. Use whatever methodology-of-the-week that you like, or that works for you. And get those tasks done, knowing where you are heading and what you need to accomplish.

Along the way there will be false trails, mistakes, problems found with the spec and the architecture. This prompts things like: "Well all that planning was a waste of time then." Which is also bull. It just meant you have LESS foul-ups to deal with later. As you find problems with the high-level early days stuff, FIX THEM.

(And on a side issue here: There is a big temptation I've seen over and over to try to meet a spec which is expensive, difficult, or even impossible. The correct response is to ask: "Is my implementation broken, or is the spec broken?" Because if an issue can be sorted out quickly and cheaply by changing the spec, then that is what you should do. Sometimes this works with a client, sometimes it does not. But you won't know if you don't ask.)

Finally - you must test. You can use TDD or anything else you like but this is no guarantee that at the end, you did what you said you would do. It helps, but it does not guarantee. So you need to do final test. Thats why things like Verification and Validation are still big items in most approaches to project management - be that development of software or making bulldozers.

Summary: You need all the up-front boring stuff. Use things like Agile as a means of delivery, but you can't eliminate old-fashioned thinking, specifying, and architectural design.

[Would you seriously expect to build a 25-story building by putting 1000 laborers on site and telling them to form teams to do a few jobs? Without plans. Without structural calculations. Without a design or vision of how the building should look. And with only knowing that it is a hotel. No - didn't think so.]

mcottle
  • 6,122
  • 2
  • 25
  • 27
quickly_now
  • 14,822
  • 1
  • 35
  • 48
  • 11
    +1. +10 if I could. If you don't have a good idea of what it is you're eventually building - which can only come from some up-front design work, even if you modify that design later - then the development paradigm you're following is "throw crap at the wall and see what sticks". – Ant May 23 '11 at 08:35
  • 5
    -1. I dislike detailed specifications because people/clients change their mind all the time, which makes specifications and up front designs pointless, not to mention wasteful. So instead of wasting time "gathering requirements" and whatnot, you should create working software which you show to your client as soon as possible, so you can get *real* feedback for the next step. Instead of making guesses and speculations. As for "specifications is the means of communication": I prefer to *talk* to my clients and work iteratively, but hey, each to his own I guess. – Martin Wickman May 23 '11 at 20:26
  • 6
    +1. +10 if I could +1. I'm a total sucker for the building software is like building a building analogy because it just is. Yes software isn't physical, yes done correctly it can be highly modular and decoupled. But making it highly modular and decoupled is very hard; that is what takes the time and the planning. I've come to realise that there will always be two camps in software engineering: those who think planning is a waste of time and those who don't. And you know I've also realised that people don't change camps, well not really. I've worked in a highly planned environment and ... –  May 23 '11 at 23:30
  • 2
    ... I always used to think "this plan fails on line one - it's a heap of £$%&, what the hell point is planning," but now I've seen that although those plans might not have been perfect and maybe they were a bit too detail rather than big picture heavy, they were what formed the framework that allowed me to write sane code. Without that you have total chaos; hack upon hack, upon hack. Some people say "but look at the features" others say "but look at the code". It's been an important process and thanks for all the great replies. –  May 23 '11 at 23:37
  • 2
    The point about planning is that some planning is generally better than none. You can take this too far. For example, I would not advocate writing specs for how a user interface should look - its quicker and easier to mock up a prototype, or even use Agile methods to knock up a first cut and seek feedback. So it depends on what you have to do. But the PRODUCT is more than just what the user sees. And its the unseen bits that need a spec more than any other. – quickly_now May 24 '11 at 11:01
  • @The Mouth: I thought the programmer community had abandoned the "developing software == constructing houses" idea. But from that point of view I guess your arguments makes sense. It doesn't make them true, however, because the underlying analogy is plain wrong. No offense, but I think you might want to brush up a bit on your agile planning knowledge. Check out this video about agile planning: http://oredev.org/2010/sessions/agile-release-planning-from-top-to-bottom – Martin Wickman May 24 '11 at 12:00
  • 1
    @Martin: 'I thought the programmer community had abandoned the "developing software == constructing houses" idea.' only the dumb ones, lol. –  May 24 '11 at 21:20
  • 3
    @MartinWickman I think there is a compromise between the two. Not line by line specs, but a framework/general process of what it should do in the end. My wife, the contractor PM, struggles with this all the time... How do you determine a ROI if you don't have a general idea of where you are going. – Paul Sep 04 '12 at 15:13
  • @Paul: Bang on! You can't work out an ROI if you don't know where you are going. I'm AMAZED in this day and age that there are still companies out there paying s/w developers to go "do stuff" instead of asking "if we do this stuff will we get a benefit... and if so, how long will it take that benefit to pay for the investment... and if we have several things so do, which will give the bigger benefit / payoff / better ROI". This is not the ONLY means of making trade-off decisions (though accountants love it). But it is a method thats of value. Having no criteria for investing $ is just silly. – quickly_now Sep 28 '12 at 09:17
  • 6
    I AGREE with you, but what you're describing really IS agile. Agile says "no big design up-front", not "no design". This was said as a response to huge rigid mega enterprise waterfall monster projects of yore. Not as an excuse to not design and find a good architecture for your code. The point is to not spend weeks or month doing ALL design before you start coding, because you design WILL be wrong, and the further you notice and fix it, the better. Get quick feedback, iterate, improve, etc. – sara May 20 '16 at 13:34
  • 5
    Kai - I see Agile being used an an excuse to do no specification, no planning, just diving in. And that's just plain wrong. – quickly_now Jun 23 '16 at 05:29
  • "Get them wrong and you have a Leaning Tower of Pisa." - haha very good. So funny, and true. – BЈовић Jul 18 '16 at 12:50
  • 2
    IMO, the requirement to produce a runnable program at the end of every sprint pushes Agile too far in the "no design" direction. Amend it slightly to require a *functional product* at the end of every sprint, where "functional product" is understood to include things like rough design documents, test programs, even libraries and other components of the final product that can't be demoed on their own, and Agile becomes workable again. – Kevin Krumwiede Jul 18 '16 at 15:22
36

I'm still amazed that many people think that TDD means writing unit tests first. No it means writing tests you will need before you write the code. The test actually can be unit test, integration test, end-to-end test and of course performance test (well you will probably not write performance test before the tested code but it doesn't mean you should not write it at all). The needed type of the test should be visible from the definition of done for the user story. If one of acceptance criteria for the user story says that feature should provide the result within 500ms with 50 concurrent users then user story cannot be considered as completed until you have performance test which will prove that this acceptance criteria is satisfied. Once you have the automatic test you can check every day that it is passing as you add other features.

It more sounds like your acceptance criteria wasn't defined correctly and because of that you couldn't test your performance. Still it can happen that application perform badly once you deploy it to the real environment and use it under heavy load but it can be always considered either as failure of requirements (if requirement is not defined developer doesn't take it in mind when working on the code) or development team (insufficient testing against defined requirements).

Another interesting thing is that developers in your team are focusing on single user story. Agile is about communication so developers should communicate what user stories require and how it affect the rest of the application - collaboration is a must. Test should cover that as well so if one developer breaks the needed functionality for another user story it should be visible in automated tests. If you still feel that it is not enough or it doesn't work you can introduce architecture meeting where the whole team cooperates and discuses architecture of the application. It is usually enough to have such meeting once a week.

A lot of things from waterfall process are replaced with communication and automatic tests. Nobody says that you cannot write any high level documentation or design! You can and many teams use for example Wiki for that.

Ladislav Mrnka
  • 7,326
  • 1
  • 23
  • 32
  • 7
    +1 excellent answer. The story is the goal - it is the tip of the iceberg, a placeholder for a future conversation, the first step on the journey; it is not the entire journey! The test descriptions are the requirements, use-cases, and acceptance criteria combined. Design is not ignored, design is scope-limited to the story and the tests, but _do as much design as you need to_. Anyone skipping design and claiming that's the agile way either doesn't understand (go read the XP book again), doesn't want to (cowbow coding yee-haw!), or **is just being lazy**. And giving Agile a bad name. – Steven A. Lowe May 22 '11 at 14:54
16

[our product was] too slow to use, buggy, becoming labyrinthinly complex and completely inflexible.

That has nothing to do with agile, it's a sign of programmers not doing their job properly.

One basic idea of agile is that the team has complete control over the process. That means they must agree on the amount of planning, documentation, testing and how they deal with refactoring. All that power is great, but it also comes with responsibility and this is probably where your team failed. They accepted their freedom, but neglected their responsibilities.

In the end it's just a matter of explaining about code quality and trying to agree on a way to increase it and keep it that way. Normal programming practices apply, really.

Pro tip: use your "Definition of Done" to enforce this. Make sure everyone agrees on it and place it visible for everyone. Use it as a strict gatekeeper for deciding if a story is completed or not. It's even possible to add non-functional requirements (like performance) to that list as well.

Martin Wickman
  • 13,305
  • 3
  • 31
  • 66
  • 1
    "They accepted their freedom, but neglected their responsibilities" maybe should be a banner on the agile wall "Have you accepted your Responsibilities along with your Freedom?" – Andy Dent Jul 20 '16 at 14:11
  • Great answer, may I suggest adding that when you use the DoD as a contract in this way, it also becomes central in the retrospective? How did this DoD help or hinder us in adding value for our customers? – Graham Lee Mar 28 '18 at 11:37
11

Yeah. Your teammates are idiots. Your project sucked because of Agile. Feel better? Okay, let's move on. :P

I think you and your team will be able to communicate more effectively about what went wrong if you concentrate less on the names of your capital-M Methodologies and more on why the software you wrote was so slow and buggy. Don't use the terms Agile or waterfall at all. They are clearly emotionally loaded in your office.

Agile works sometimes. It didn't work for your team this time. Some people will say that's because you did Agile wrong. After all, Agile works, so if you had done it right it would have worked, right? Whatever.

It doesn't sound like anyone disagrees there was a failure, but you aren't going to win friends, influence people, or do any better next time if you blame a methodology. That probably had little to do with what actually went wrong.

Instead, concentrate on pinning down the most direct causes of the failure and work with the team to make sure they don't happen again. This will require people skills.

Speaking of people skills, you shouldn't be surprised that your team didn't appreciate you making them look bad by doing all their work and doing it better than they did. By doing this "under the radar" you may have damaged some relationships you'll now have to rebuild to be an effective member of the team.

I think the best way to look at a situation like this is to consider the long-term total output of the team as a whole. You may have saved the week this time, but you might have done better in the long run for your company by building a better team.

I'm telling you all these things, but I think you probably know most of them already and could explain them to someone else if you weren't so close to this situation.

PeterAllenWebb
  • 933
  • 8
  • 12
9

If you want to add a pithy quote to your discussions, Eisenhower had a good one:

"Plans are nothing; planning is everything."

http://www.brainyquote.com/quotes/quotes/d/dwightdei149111.html

He meant that you should expect to change your plans and so don't put too much value in the plan as it exists, but at the same time the process of planning will sharply focus your energies in a crucial way. You have to make plans before you can test them and refine them.

jhocking
  • 2,641
  • 19
  • 18
9

+1 This is the best description of enterprise agile I have read recently.

Everyone who gets along with agile points out that it was never supposed to mean this, but once everyone gets caught up in the metrics this is exactly what you get from a team that has no passion about the quality of the product and either is obsessed with test coverage above all else or with meeting their weekly deliverable deadlines regardless of the corner they have painted everyone else into because that is all that makes it up to the management level on a weekly basis.

I have never seen this fixed with anything less than a re-org. If you are a mid-grade company with nothing special to attract really passionate people then even that won't fix it unless the next management changes methodologies sometimes.

Agile only seems to work well in orgs where the dev team cares enough to make a good product in spite of the extra, uncredited work that is required. Effectively they have to care enough to make it good on their own time, fight for improvement (and be willing to burn a lot of extra time redoing tests in some TDD cases)

You obviously care about shipping a good product, they obviously care about going through a scripted set of motions and receiving a paycheck to continuously clean up the mess they make.

I would look for less irritating employment elsewhere be it agile or not.

If you are completely burnt on agile there are lots of places that still do other methodologies. Almost anything on bid or contract for example.

Bill
  • 8,330
  • 24
  • 52
  • 1
    That is actually the worst definition of agile I read. Developers do not need to do extra and uncredited work. Also, only idiots work in their own time. Time spent on testing the code is well invested time. – BЈовић Mar 28 '18 at 09:12
  • Couldn't agree more, Agile, when turned into the beast that it often becomes at enterprise levels of red tape is the worst possible agile. In a non-enterprise other tape-color environment the team would just do those things either as stories, or before committing to stories. When Agile becomes a corporate metrics indicator that flexibility is usually the first thing to go. – Bill Apr 02 '18 at 16:41
4

I tend to use a sort of hybrid. The overall plan is waterfall, but there is agile in the execution.

My guess is that if the situation is as bad as you say, your team isn't really using agile, they're just being lazy or undisciplined. Agile isn't about not planning, it's just about being flexible, and, well, agile.

richard
  • 3,325
  • 2
  • 25
  • 39
  • I tend to think that too. I'm sure that *real* agile isn't about not planning, it is just that that is one interpretation of it. –  May 22 '11 at 11:32
  • There's a world of difference between capital-"A" Agile and lowercase-"a" agile. – Aaronaught May 23 '11 at 19:58
  • Pssst...don't tell the agile people that, since that is how almost all the waterfall people do it because it works. They still believe that ALL waterfall developers build monolithic documents without writing any code until the very end and at each step everything is wrong because there is no implementation. – Dunk Oct 02 '12 at 21:39
4

We have the same problems with some staff.

Basically "I don't know until I write it" is a favourite statement. So we went the opposite way around, we worked with the client to define the deliverables with sign off points. The last one being "now write the code to do X".

If we have "write design/testing/plan etc" in the same deliverable sprint as "do the fun and interesting code" then the first part never happens ...

BUT if I place "you don't get to do any fun and interesting code until you have said what your going to build and the client has signed off" then

  • you firstly get grumbling acceptance, and a few tears and I had to do a lot of the filling in the blanks work and extra effort ...
  • then you get dawning recognition when you ask them "so how was the process" that its better to have attempted the first version on paper
  • then you have the test cases which the developers can see, and you can point exactly to the expectation of "what does done mean".
  • then clients sign off on the development has an 80% less rejection rate ...
  • plus you watch all the developers walking around holding the design documents and talking to each other over the designs ... they start to really get it.
  • Then you work out how to break down the project plan into bite sized chunks and make a process out of it.

The agile part comes when the client then changes from the plan and you can adapt within a 2 week cycle NOT fly by the seat of your pants and make it up along the way.

In our case the "big design upfront" was broken into 9 stages (actual production releases) with an average of 5 substages. The designers and developers keep pace with each other, the designers being 1-3 substages ahead of the developers ... too far and discoveries in development break too much of the design, too close and tweaks to design cost to much as they have been already implemented "set in stone" within development. Each substage is 2-4 sprints worth for 1 developer.

In smaller projects we just have the same developer do design first > sign off > invoice > develop > sign off > invoice ... in cycles.

The test naming problem

The testing has many faces, formally there is about 7 catagories of tests each with sub sections ... one of these later ones is "write automated unit tests".

Its a bad name to have "test first development" purely because of developers understanding about what "tests" means in this context they see tests as write the actual code for the test against the interface for the implemntation. When its not really that at all ... you can do this when it comes to the writing the code but its really "validate the design against use cases and user stories BEFORE starting to write the code" ... done properly this removes many of the issues normally discovered during development while its still in the far cheaper "paper that can be thrown away" version.

Robin Vessey
  • 1,577
  • 11
  • 14
3

One of the key elements of Agile is the idea of continuous improvement, and the idea that the entire team owns the project. So the correct approach would have been to review the issues with the team, and have the team decide how to correct it.

One team member going off by himself, ditching all of the principles of Agile and doing things his own way may result in a small amount of code working well but it doesn't really advance the whole project in a positive way.

Dave
  • 869
  • 5
  • 7
  • Sounds to me like advancing the project was precisely what it *did* do. "Review with the team" is not actually a solution, it's merely a way of deferring the solution and spreading the responsibility. – Aaronaught May 23 '11 at 20:00
  • The team is already responsible for the outcome. What they need is for someone to say, "We're messing up, how do we change our methodology?" Then they fix it. – Dave May 23 '11 at 20:58
  • 2
    I get the impression that what this *particular* team needs is for its members to learn how to think for themselves instead of blindly following a process they don't understand. Talking won't accomplish anything if it's already an echo chamber. – Aaronaught May 23 '11 at 21:07
2

One way to get them to work may be to make a T-Map that covers all of your sprint back-log

T-map

Each team has a thread, each sprint is a period. It all ties together (which is where your team seem to fall over). Pin this up somewhere, and get magnets to represent the pairs/subteams. They can even 'race'. But everyone knows where they and the other teams are. Put dependancies up on here if there are any.

The point here is you convey the total process but also break it down into sprints. Even if only the first sprint is put on there, and the other periods are blank this should be an excellent roadmap to finishing the project.

AncientSwordRage
  • 1,033
  • 1
  • 11
  • 24
1

You have two problems: not enough shape on the requirements and poor architecture.

Requirements

You need both an overall end goal and the detailed roadmap of how to get there.

In the waterfall world, the end goal is the functional specification, and the roadmap of how to get there is the is the project plan.

In the agile world, one way is to capture it in an epic user story. Then the roadmap are the detailed user stories that flesh out the detail of the epic.

I've never been completely happy with that story, because the epic story never captures enough meat to get across the full idea. So what I've tended to use is a high level systems requirements document in conjunction with an epic user story or two. After that, the roadmap is the detailed user stories.

The nice thing about having a systems requirements document is that then the acceptance criteria for many of the user stories can be pulled out.

Think of the high level system requirements document as a "cut sheet" that marketing might use to sell the product to a technically savvy customer.

Architecture

One of the things that the "cut sheet" gives you is it puts boundaries on the system you're designing. That lets you make informed decisions, early on, about the architecture to use.

If your team had had such a document early on, you would not have to go through the pain of re-architecting the system later.

Actually, a third problem you have is poor communication. Communication is a two-way street (or multi-way between multiple people). However, that's just a human failing and takes (sometimes) superhuman efforts to get right.

Peter K.
  • 3,828
  • 1
  • 23
  • 34