Branching model suggestion for same project multiple clients

Question

We have a very big project that includes several applications that acts as the base for different clients.

Every client has its own personalization of the product, different milestones, different requirements and so on, and thus each project will evolve independently based upon its own needs.

The core of the project, is similar (but not equal) in every project and the organization is made so that there are teams that handle each client independently (but with communication among them as needed). So far, I've been unable to find any scheme that suits our needs, either by searching the internet or coming up with some brilliant idea :)

So far, we've been working by getting the product as a suit all needs, with specific branches for needed changes but, though the product has a good architecture, it's slowly becoming a big problem. Here are the main problems we face:

• Different milestones for each client: Which means each team has to produce versions as different times without the rest of the commits affecting the stability or their product.
• Different requirements, which may or may not affect the core of the system in some cases.
• Large teams (20+ team members)
• Handling bugs in the system: What do you do if a team finds a bug in his project that might affect other clients?

Note: We're talking about a project that has 10+M LOC.

Note: We're using Team Foundation System, Visual Studio 2008 and C# (mainly).

Any suggestions, sources or ideas about how to address the situation? Is there any model in the market that has a similar problem?

Answer 1

When I started at my first job I worked on similar projects (but with smaller scale) and we faced the same problems. We also started with general solution handling requirements for all clients but that was possible only to same point where requirements become contradictory. We did what you suggest and started separate version for each client. Even this solved the problem with requirements and customization it becomes a maintenance nightmare for solving bugs and global changes.

Because the code in application was only similar, merging changes from one customer version to another was very complex and it required retesting each version separately (we didn't have automatic tests!). It often caused regression bugs in different versions. In your scenario this can be even worse because one team will solve the bug in their version and another team will have to merge that change from the version they don't fully understand (we were one team working on all versions).

Unless you have some shared global core you will have same problems. Before I left the company we found that our approach was incorrect. To support such development scenario we needed shared extensible core and data model which will be configurable from upper customizable application layers. This core should be used as the base for each customer specific customization and maintained by separate team. It will include some management complication because multiple project managers will need resources from the same team but it is the only way to make architecture consistent, control the whole process and keep versions in sync.

Answer 2

Actually, I would suggest that you do not need a branching model, but rather a complete comprehensive approach to deal with the multi-dimensional constraints with regard to the system without branching. In practice, I believe it will always be a problem to maintain multiple systems that have some commonalities but will evolve differently in branches, so it is better to turn everything into a single system that will evolve as a whole, but consists of different configurations. This may appear to be too simplistic, but there is extensive research in this area, with a lot of successful industrial applications.

The name for this approach is Software Product Lines or sometimes Product Line Engineering. From CMU SEI's Software Product Lines page:

A software product line (SPL) is a set of software-intensive systems that share a common, managed set of features satisfying the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way.

The key idea is that every requirement, every milestone, every feature (a key term in this domain) is part of the complete system at the highest level. The actual systems deployed at various customers is essentially a collection of features. Each feature however is not just a physical component mashed into the system, it is defined as depending on or enabled by other features (so that by picking a feature, you may automatically include its dependencies, etc.)

Instead then of maintaining all these branches, you end up maintaining one system along with a set of customer-specific configurations.

It may be difficult or even impossible in practice to migrate to such an approach with a very large system, but even then it will be useful to investigate the approaches used in SPL in order to assess what approaches used there can at least be (partially) integrated in your work.

Some additional useful links:

• Wikipedia: Product Line Engineering
• Case Studies by CMU SEI
• Software Product Lines, a site with a lot of introductory material.

Answer 3

I may be way of base, but I think that what you are facing with your core of your system is the same problem everyone faces who uses components and needs to maintain and support different releases of their software and those different release each require a different set of component versions.

For example

• release 1.0 requires Library A 1.0, Library B 2.0, Library C 5.6
• release 2.0 requires Library A 1.0, Library B 3.7, Library C 5.7
• release 3.0 requires Library A 1.2, Library B 3.7, Library C 5.8

The way we solved the component problem is to have all versions of the libraries in our version control system (we build them on source) and to make each project use the proper library version by changing their search path (reference path?).

In Delphi this is easily done through the project's configuration file (under source control) if you do not need the libraries at design time, otherwise it is still doable but becomes a bit more of a pain as you need to switch the Delphi IDE to use the correct version as well (registration (.reg) files under version control can come to the rescue here).

A solution for your core system might be to treat it as a library for which you maintain different versions. Which in essence means that you need to set up different branches for each version. A client's application can then use the proper "version" of your core system by referencing the proper core system's branch folder.

Where the dependencies for your core system would then resemble the example above, the dependencies for your client applications then "just" have an extra reference: the version of your core system.

Added advantage with multiple client application here is that you can choose when to start using a specific version of your core system and that you are not affected by changes to the core system that you are not ready to use yet for a specific client application.