How do you adapt to changes to product specifications during development?

Question

Problem

Changes to product requirements/specifications during the development cycle are inevitable in a professional setting. In particular, I'm constantly dealing with changes to the specification for an embedded system's software parameters: I/O, user settings, calibration data, logging information, etc. Are there any tools, practices, or methods that development teams commonly use to handle this more gracefully?

Scenario

Typically, we have one or more engineers/product managers responsible for the document's creation and keeping it up-to-date as requirements change. This "master" document is usually a spreadsheet and contains all the metadata and descriptions for the parameters above. As development progresses, changes to this document need to be propagated to multiple places in the product's source code (interfacing to multiple industry standard protocols among other things), automated test fixtures, other specification documents (describing how they're presented through the other protocols), and multi-language manuals/marketing materials, all of which can be owned by different people. A lot of time and effort seems wasted doing what's essentially manual data entry across multiple people, copying information into different formats. This process is also extremely prone to human error, resulting in a lot of inconsistencies.

Right Track?

My attempt to see how this process could be more streamlined in a previous project was to simply enforce a format on the "master" spreadsheet that would be easier to build a parser around. A parser was developed in Python to auto-generate the database file for the mobile app that interfaces to the product. This seemed to work well since the mobile app team would just rerun their parser anytime an update was needed. Unfortunately, this automation never extended into any of the other problem areas as this was already towards the end of the project. It's hard for me to tell if this is a reasonable approach moving forward as I have no other experiences or example cases to compare with.

Answer 1

Your parser approach is definitely a step in the right direction. To deal with what you describe, you need to follow two well known software engineering principles as rigidly as possible:

1. Single source of truth

You wrote

changes ... need to be propagated to multiple places in the product's source code

That is exactly the opposite of "single source of truth": there should be ideally just one place to change in your source code, or your database, or your interface description, or whatever depends on the change. Try to reduce the number of places a piece of information is stored in your environment, then you do not have to maintain twice.

Whenever that is not possible or feasible, make sure the information is stored in machine-readable format, maybe including some kind of meta-data, to make it possible to implement the second principle

2. Automation everywhere

For example, whenever you have an interface to implement in multiple places, provide an interface spec in a machine readable form, and a code generator which generates at least the skeleton code to implement that interface. A machine-readable spec in spreadsheet form is one of many possible solutions, other approaches I have seen were specific text formats, domain specific languages, hierarchical meta data in XML or JSON formats, descriptions in database form, machine-readable UML diagrams or whatever turns out to be adequate to make the above possible.

Of course, it is much easier to implement these ideas right from the start of a project. Changing an organization which had brought themselves into a situation where the same information has too many redundant representations in different places of the code base, which all have to be changed manually, can be hard, if not next-to-impossible. First step could be to make a checklist "if change of type X occurs in document, we have to make manual changes Y1, Y2, ..., Yn in the following places". This checklist can already be helpful for some people. The next step is to analyse why all these steps Y1,...,Yn are necessary, can you reduce the number of steps by eliminating duplicate information, or can you at least automate the step, or parts of it?

To my experience, you can often improve at least parts: if you find ways not to maintain 6 places in code manually, but only 3 when a specific kind of change request arises, it is still an improvement.

Limits

What I wrote mainly applies to your code base. When it comes to things like "other specification documents" or "multi-language manuals/marketing materials", especially when "owned by different people", it is typically impossible to reduce the manual effort to zero. Nevertheless, I made the experience that by keeping the above principles in mind, you can often get some enhancement here, too, by having clear "interfaces" between the dev teams and/or the people who write the docs, who make the translations, who write the marketing stuff etc.

For example, for our advertisement materials, we often needed up-to-date screenshots of specific parts of the UI of our product - so made this step at least semi-automatic. Assembling all the screenshots together into a new brochure is still a manual process, of course, but most of that work can be done without a dev now, and we stripped some of the cumbersome and boring parts out of it.

Answer 2

Doc Brown's answer is a great starting point, so I'll add a few other suggestions to the end of his list.

Timeboxing and User Stories

These two techniques are used together in several different Agile methodologies. Timeboxing (also known as sprints or iterations) is essentially a guarantee from both the business and the development sides that requirements that are fixed for a short period will be delivered and can then be iterated upon. These different requirements are enumerated in user stories that cover the entire life-cycle of a requirement from analysis to dev to testing to delivery.

For your purposes, you could set up a time box (maybe a week or two, maybe a month) where requirements are set. The benefit to this is that, for example, the four days that the business is waffling back and forth on a particular requirement would be invisible to the development team because they will only see the final requirement change at the beginning of the time box. If the business has not reached a final decision, it waits until the next time box. If the business changes its mind halfway through a time box, that change waits until the next time box, but because time boxes are short it's no more than a few weeks away.

When you receive requirements for your time box, you can use user stories to track everything that needs to be done for a particular requirement. The story itself outlines the particular feature or change desired, and required tasks for a user story can be broken down on the back of the note card (or in the Trello card description, or as JIRA subtasks, etc). You may even be able to generate a draft user story from the changes in your spreadsheet. Depending on the data available, you may even be able to auto-generate some of the required tasks (such as which documents to update) from the spreadsheet.

Modular, SOLID Design

When it comes to your code, you should write it in such a way that you don't have to make changes all over when requirements change. By making small, focused, interchangeable parts, you can easily write new code and swap it out as requirement changes come up. With the right kind of design, you could even swap out different parts at runtime, meaning you don't have to know exactly what it will be until the moment it runs.

By having small, focused modules, you will also be able to organize your documentation in parallel. Documentation for a particular module only needs to change if there were changes to that module. This would make it easy to track what needs updated just by doing a few queries against your version control software. VCSes are generally very scriptable, so you might even be able to wrap a small application around it that can generate notifications when certain modules change. You should already have a similar notification system around your automated tests to alert you when system behavior changes.

You mentioned that you are dealing with an embedded system, so this might not apply to you in this particular instance, but as your code gets more modular, it might make sense to break it out into completely separate microservices that can be developed, maintained, and run completely separately. If the load becomes too much, you can pass off that small application to another team to manage.

Focus Only on Valuable Documentation

It sounds like there is a lot of different documents you have to maintain. You should evaluate which of those are even useful before you do anything else. It does no good to efficiently update a document that no one reads. Documents that regurgitate standards may not be worth the effort if they don't add anything above and beyond the standard docs.

You should also look at how to make updating those documents as friendly as possible. Javadoc (if you were using Java) can be automatically converted from code comments to HTML developer docs, for example. Developers just need to comment their code in a particular simple format, and you get a large base of useful documentation on the other side. For other documents, it may not be able to be completely automatic, but you could automate generating a series of screenshots for end-user documentation and marketing materials. By outputting portable, parsable formats, you may find that some material may be reusable in several different contexts.