17

I've always wondered how to apply agile methods really are in large complex embedded system software (100+ engineers). Firmware development has some unique characteristics that make it difficult to do agile (ie. Hardware is not available until late in the dev cycle; Once product is released, can't easily update firmware; etc...)

The norm in this kind of development is thick documentation and grueling peer reviews. You can't get a simple code fix like renaming a variable without 2-3 signatures. (I exaggerate a little but this is typical. Additionally, a lot of people do take shortcuts and the Project Managers even approve them especially in the face of hard market deadlines.)

I would like to hear any tips or guidelines on how to adopt agile methodology for firmware development projects.

yannis
  • 39,547
  • 40
  • 183
  • 216
hopia
  • 311
  • 2
  • 6
  • I understand that the finalized hardware isn't available until late in the dev cycle, but don't you have prototype or debug hardware, a dev board, or at least a vendor's simulator to test with? Or are we starting from scratch here? – Kevin Vermeer Apr 08 '11 at 17:59
  • 3
    I was talking to an agile developer about my embedded job. "A release every 6-8 weeks!?!?" he said. "That's a really long time". "No, you misheard me," I said, "It's 6 to 8 _months_" – AShelly Apr 08 '11 at 21:19
  • see also http://stackoverflow.com/questions/4498476/agile-practices-on-embedded-software-development – AShelly Apr 08 '11 at 21:21
  • 2
    I am kind of curious what type of product has 100+ embedded engineers? – Pemdas Apr 09 '11 at 04:55
  • @reemrevnivek - usually there is an eval board from previous projects that could be used. Sometimes, they are similar enough to the new product. Even then, even though all your tests are passing on the eval board when you actually run the firmware on the final device, more often than not, tests will break because there were some new things hardware guys decided to add on the last minute or maybe didn't mention to us in the beginning.... – hopia Apr 09 '11 at 21:29
  • @pemdas - Personally, I've worked on Multi-Function Devices (laser printer, photocopier, scanner). Other examples are complex medical devices, aircraft electronics, etc... – hopia Apr 09 '11 at 21:32
  • Some would say that the best way to use 100 engineers is to assign the actual work to best five, and give the other 95 engineers tasks that will keep them out of the way. – Kristopher Johnson Apr 10 '11 at 15:33
  • @pemdas - Think big: NASA, Boeing, Airbus... – mouviciel Apr 10 '11 at 18:47

4 Answers4

10

I think two techniques are key:

  • Develop a complete simulator or test-environment for the hardware, so that you can develop the software as if you have real hardware. Don't skimp or take shortcuts here: developing a good simulator will pay off.

  • Write lots of unit tests against the simulator.

Once you have these things going, and people are confident that the simulator and unit tests will give an accurate idea of how things will work with the hardware, it will be easier to adopt other agile techniques (short iterations, relentless refactoring, etc.).

Kristopher Johnson
  • 1,719
  • 4
  • 15
  • 15
  • Also, make the chip manufacturers provide the relevant part of the simulator code. – rwong Apr 09 '11 at 04:40
  • by the time you have these things a competitor has already shipped – Bill Apr 09 '11 at 16:03
  • If you've got 100+ engineers, you can certainly create a usable simulator in less time than your competitors will ship. That's especially true if your competitors have no way to test their software until they receive the hardware. – Kristopher Johnson Apr 09 '11 at 21:12
  • Yes, I agree simulators are key. So is designing the whole system from the beginning based on how efficiently you can break up the system into testable components. Now, how to convince all of those people to go agile is another question........ – hopia Apr 09 '11 at 21:34
  • @bill That's very likely. However, that probably means they shipped an untested inferior product, and the OP's product will blow them out of water. Well, at least that's how it should be. – Julio Apr 10 '11 at 15:46
  • @Julio who ever said anything about untested... there are plenty of traditional design/testing methodologies that still apply here. My point was that by the time you build all of this virtual stuff someone could probably do proper up-front design and thorough post-dev testing. Some problems aren't agile problems. Hardware and low level OS interaction are often in this category. – Bill Apr 11 '11 at 17:01
2

The impact of Agile on a development process involving multiple suppliers can be compared to what happens when a company goes JIT.

In order to deliver JIT, each of the company's suppliers has to deliver JIT.

function deliversJIT( company ) { 
    return company.isJIT() && map( deliversJIT, company.suppliers() );
}

Likewise, if you want a complex product to be developed according to Agile methodologies, every subgroup in the chain should be able to work that way.

In terms of 'incremental' development (a.k.a. Tracer Bullets of 15 years ago), this would imply that the 'core' is released really soon to the driver guy, and that the basic driver be available to the integrator, and that the GUI would be developed, beit with one single button and an edit box, at the same time.

The hard part is convincing the hardware designers, coming from a solid think-upfront engineering discipline, to join our tinkerer society.

The second hardest part is finding a way to enable incremental development in a world where a hardware print is to be ordered 3 weeks in advance. I'm thinking emulators, fpga's here. I'm not a hardware guy though.

If you are willing to drop on incremental hardware development, you can, just like on the borders of a JIT chain, foresee a buffering mechanism that allows the Agile teams to advance.

Let's not be blind: Agile does have to deal with heavy-weight processes, too! Don't ask the Agile team to now 'refactor' their Java code base to Python in the next sprint. It is only because some of the parts are really really stable that we can dance our Agile moves on top of them.

xtofl
  • 334
  • 2
  • 10
  • +1 for agile only being possible because the underlying stuff is thoroughly designed / done. – Bill Apr 09 '11 at 16:04
1

Agile Manifesto: http://agilemanifesto.org/

"Individuals and interactions over processes and tools"

  • Meet more. Push paper less.

"Working software over comprehensive documentation"

  • Prototype and build technology spikes early and often.

  • Solve the user's real problem rather than continue to build fussy adherence to a specification. This means evolutionary solutions. The idea that we have to build it right because we can never built it again is wrong. Plan on iterating. Make it part of the marketing and roll-out strategy.

"Customer collaboration over contract negotiation"

  • Hyper-complex change-control processes are just ways of saying "no" to the customer.

  • Locking down all requirements up front and then imposing change control is another way of saying "no".

  • If you already plan on more than one release, then you can more easily defer requirements to a later release. Once the customer has the device with the embedded software, the next release will change in it's priorities.

"Responding to change over following a plan"

  • While complex integration requires a complex plan, the overall "program" (or sequence of projects) can't be cast in concrete too early.

A totally Agile methodology (i.e. scrum) may not make sense for an embedded system.

The Agile manifesto, however, provides ways to allow change to be possible without allowing simple chaos.

S.Lott
  • 45,264
  • 6
  • 90
  • 154
0

My issue with scrum in embedded systems is, there are many tasks to do, especially in the early stages, which are not demonstrable. We started with a dev board, no OS, no display, no serial comms, etc. We didn't have our display for 6 sprints.

The first 4 sprints were: Getting the RTOS up and running Creating tasks Writing network and serial drivers Writing interrupt routines for buttons, comms, etc Writing the primary database classes and methods Writing a serial debug menu

Most of these tasks are not well suited for user stories. In fact, the only interface into the whole system was the serial debug menu, built in sprint 3 so there was nothing to demonstrate at the end of the sprints. Even the serial menu was meant for internal use and not an end user.

Of course, each class we write has associated unit tests and we use a unit test framework to automate the execution of all tests.

We ended up writing "user stories" phrases like, "As a developer...", which I'm not happy with but in using Target Process (www.targetprocess.com), there is no concept of a backlog item which is a task or chore.

I'd love to hear how others have handled these situations.

Bruce
  • 441
  • 4
  • 5