How to maintain different, customized versions of the same software for multiple clients

Question

we have multiple clients with different needs. Although our software is modularized to a degree, it's almost certain that we need to adjust every module's business logic here and there a little for each customer. The changes are probably too tiny to justify splitting the module into a distinct (physical) module for each client, I fear problems with the build, a linking chaos. Yet, those changes are too large and too many to configure them by switches in some configuration file, as this would lead to problems during deployment and probably a lot of support troubles, especially with tinker-type admins.

I'd like to have the build system create multiple builds, one for each client, where changes are contained in a specialized version of the single physical module in question. So I have some questions:

Would you advise letting the build system create multiple builds? How should I store the different customizations in source control, svn in particular?

Preprocessor directives can quickly become very complicated and make the code harder to read and harder to debug. — Falcon, Mar 21 '11 at 11:36
You should include details about the actual platform and type of application (desktop/web) for better answers. Customizing in a desktop C++ app is completely different than a PHP web app. — GrandmasterB, Mar 21 '11 at 19:36
layers. build an internal api for your core, then business logic on top. Tweaking just the business logic is much easier. But are you really going to modify your build script every time you get a new customer? I think you'll need to try solving this problem a different way. — jqa, Mar 22 '11 at 01:56
@Falcon: since you picked in 2011 an answer I have lots of doubts about, can you tell us if you have got any experience with using SVN in the suggested way in between? Are my objections ill-founded? — Doc Brown, Mar 31 '15 at 11:31
possible duplicate of [Doubts about architectural pattern for customizable standard enterprise application](http://programmers.stackexchange.com/questions/277747/doubts-about-architectural-pattern-for-customizable-standard-enterprise-applicat) — gnat, Apr 02 '15 at 09:56
@Doc Brown: the branching and merging was tedious and complicated. Back then, we used a plugin system with client specific plugins or "patches" that altered behaviour or configurations. You'll have some overhead but it can be managed with Dependendy Injection. — Falcon, Apr 02 '15 at 11:18
@ohho If I would have a team-mate solving this problem according to your suggestion I'd probably be that "Maniac who knows your address" — Cristian E., Sep 25 '16 at 17:48
@Falcon: then why did you not change the accepted answer, which tells exactly the opposite of what you did? This is pretty confusing for anyone who comes here and finds this for the first time. — Doc Brown, Aug 14 '20 at 06:48

Alb · Answer 1 · 2011-03-22T00:45:43.757

44

Don't do this with SCM branches. Make the common code a separate project that produces a library or project skeleton artifact. Each customer project is a separate project which then depends on the common one as a dependency.

This is easiest if your common base app uses a dependency injection framework like Spring, so that you can easily inject different replacement variants of objects in each customer project as custom features are required. Even if you don't already have a DI framework, adding one and doing it this way may be your least painful choice.

edited Mar 22 '11 at 00:45

answered Mar 21 '11 at 23:51

Alb

3,359
2
20
24

Another approach to keep things simple is using inheritance (and a single project) instead of using dependency injection, by keeping both the common-code and the customizations in the same project (using either inheritance, partial classes, or events). With that design, client-branches would work fine (as described in the selected answer), and one could always update the client-branches by updating the common-code. – drizin Sep 17 '16 at 22:44
1

If I don't miss something, what you suggest is that if you have 100 customers, you will create (100 x _NumberOfChangedProjects_ ) and use DI to manage them? If it's that so, I would definetely stay away from this kind of solution since maintainability would be terrible.. – yakya Aug 21 '18 at 11:05
@sotn Agree with sotn; also, see my comment on Robert's answer, which also applies here: hard to tell how any code changes will affect the entire project. – Andrew Sep 04 '20 at 07:32

P Shved · Answer 2 · 2011-03-21T22:15:38.377

Store software for all clients in a single branch. There's no need to diverge the changes made for different clients. Most lilkely, you'll want your software to be of best quality for all the clients, and the bugfix to your core infrastructure should affect everyone without unnecessary merging overhead, which also may introduce more bugs.

Modularize the common code, and implement the code that differs in different files or guard it with different defines. Make your build system have specific targets for each client, each target compiling a version with only the code related to one client. If your code is C, for example, you might want to guard features for different clients with "#ifdef" or whatever mechanism your language has for build configuration management, and prepare a set of defines corresponding to the amount of features a client has paid for.

If you don't like ifdefs, use "interfaces and implementations", "functors", "object files" or whatever tools your language provides for storing different things in one place.

If you distribute sources to your clients, it's best to make your build scripts have special "source distribution targets". Once you invoke such a target, it creates a special version of sources of your software, copies them to a separate folder so you can ship them, and does not compile them.

Robert Koritnik · Accepted Answer · 2015-03-31T10:07:03.137

9

What you need is feature branching-like code organisation. In your specific scenario this should be called Client-specific trunk branching because you'll likely use feature branching as well while you develop new stuff (or resolve bugs).

http://svnbook.red-bean.com/en/1.5/svn.branchmerge.commonpatterns.html

The idea is to have a code trunk with new features' branches merged into. I mean all non-client-specific features.

Then you also have your client-specific branches where you also merge same features' branches as required (cherry-picking if you will).

These client branches do look similar to feature branches although they're not temporary or short-lived. They are maintained over a longer period of time and they're mainly only merged into. There should be as little client-specific feature branch development as possible.

Client-specific branches are parallel branches to trunk and are being active as long as trunk itself and don't even get merged as whole anywhere.

            feature1
            ———————————.
                        \
trunk                    \
================================================== · · ·
      \ client1            \
       `========================================== · · ·
        \ client2            \
         `======================================== · · ·
              \ client2-specific feature   /
               `——————————————————————————´

edited Mar 31 '15 at 10:07

answered Mar 21 '11 at 10:37

Robert Koritnik

3,511
1
16
26

7

+1 for feature branching. You can use a branch for each client too. I'd like to suggest just one thing: use a distributed VCS (hg,git,bzr) instead of SVN/CVS to accomplish this ;) – Herberth Amaral Mar 21 '11 at 11:08
3

I agree with Herberth Amaral. Branch merging in Subversion is terrible. – Htbaa Mar 21 '11 at 12:08
48

-1. That's not what "feature branches" are for; it contradicts their definition as "temporary branches that merge when the development of a feature is over". – P Shved Mar 21 '11 at 21:17
17

You still have to maintain all those deltas in source control and keep them lined up - forever. Short term this might work. Long term it might be terrible. – quickly_now Mar 22 '11 at 01:07
@PavelShved: Maybe I didn't name them correctly. I should likely just use term *branching* or *client branching*. The main idea is to keep a master brach where features are being committed and these are then committed to client branches as required. – Robert Koritnik Dec 16 '13 at 08:08
6

-1, this is not a naming problem - using different branches for different clients is just another form of code duplication. This is IMHO an anti-pattern, an example how not to do it. – Doc Brown Mar 30 '15 at 17:09
Ok let me edit my answer. I was referring to the accepted term *feature branching* here while it should practically be renamed for the sake of this answer. – Robert Koritnik Mar 31 '15 at 09:45
8

Robert, I think I understood pretty well what you suggested, even before the edit, but I think this is a horrible approach. Assume you have N clients, whenever you add a new core feature to the trunk, it seems the SCM will make it easy to propagate the new feature to the N branches. But using branches this way makes it too easy to avoid a clear separation of the client-specific modifications. As a result, you now have N chances to get a merge conflict for each change in the trunk. Furthermore, you now have to run N integration tests instead of one. – Doc Brown Mar 31 '15 at 11:16
1

-1 sorry this is a disaster waiting to happen Alb has provided the answer below. – krystan honour Feb 25 '19 at 14:56
Yeah I think more specifically why this approach is problematic is as a developer, you're typically only going to look at and work with 1 branch at a time. So you'll be looking at code and think, "Oh, I can just develop it like this," not realizing that your code will be incompatible with the other 10 branches. So for a problem like this you actually do want to have it be built into the same code and/or config files somehow, so that it is apparent to every developer how code changes will affect every corner of the project. – Andrew Sep 04 '20 at 07:30

score 9 · Answer 4 · answered Mar 22 '11 at 00:04

As so many have stated: factor your code properly and customize based on the common code—this will improve maintainability significantly. Whether you're using an object-oriented language/system or not, this is possible (though it is quite a bit more difficult to do in C than something that is truly object oriented). This is precisely the type of problem that inheritance and encapsulation help to solve!

kevin cline · Answer 5 · 2011-03-22T00:21:22.430

5

The changes are probably too tiny to justify splitting the module into a distinct (physical) module for each client, I fear problems with the build, a linking chaos.

IMO, they can't be too tiny. If possible, I would factor out the client-specific code using the strategy pattern pretty much everywhere. This will reduce the amount of code that must be branched, and reduce the merging required to keep the general code in sync for all clients. It will also simplify testing... you can test the general code using default strategies, and test the client-specific classes separately.

If your modules are coded such that using policy X1 in module A requires using policy X2 in module B, think about refactoring so that X1 and X2 can be combined into a single policy class.

edited Mar 22 '11 at 00:21

answered Mar 21 '11 at 19:28

kevin cline

33,608
3
71
142

I like this. [Strategy Pattern](https://en.m.wikipedia.org/wiki/Strategy_pattern) would work great. I'd also look at the [Factory Pattern](https://en.m.wikipedia.org/wiki/Factory_(object-oriented_programming)), which may be of good use with the Strategy Pattern instead of inheritance (which can sometimes get quite ugly, especially in c++), and the [Flyweight Pattern](https://en.m.wikipedia.org/wiki/Flyweight_pattern) for any kind of configuration or vendor/version-specific options (so you're not passing around 1,000 variables into various constructors and functions). These 3 are very powerful. – Andrew Sep 04 '20 at 07:41

score 5 · Answer 6 · answered Mar 22 '11 at 02:34

Very Carefully

Feature Branching is an option but I find it to be somewhat heavy. Also it makes deep modifications easy which can lead to a forking outright of your application if not kept under control. Ideally you want to push up as much as possible the customizations in an attempt to keep your core code base as common and generic as possible.

Here is how I would do it though I do not know if it is applicable to your code base without heavy modifications and re-factorings. I had a similar project where the basic functionality was the same but each customer required a very specific set of features. I created a set of modules and containers that I then assemble through configuration (à la IoC).

then for each customer I created a project that basically contains the configurations and the build script to create a fully configured installation for their site. Occasionally I place there also some components custom made for this client. But this is rare and whenever possible I try to make it in a more generic form and push it down so other projects can use them.

The end result is I got the level of customization I needed, I got customised installation scripts so that when I get to the customer site I do not look like i'm tweeking the system all the time and as an added VERY significant bonus I get to be able to create regression tests hooked directly on the build. This way any-time I get a bug that is customer specific I can write a test that will assert the system as it is deployed and thus can do TDD even at that level.

so in short :

Heavily modularised system with a flat project structure.
Create a project for each configuration profile (customer, though more than one can share a profile)
Assemble the required functionality sets as a different product and treat it as such.

If done properly your product assembly should contain all but a few configuration files.

After using this for a while I ended up creating meta-packages that assemble the mostly used or essential systems as a core unit and use this meta-package for customer assemblies. After a few years I ended up having a large toolbox that I could assemble very quickly to create customer solutions. I'm currently looking into Spring Roo and see if I cannot push the idea a bit further hoping one day I can create a first draft of the system right with the customer in our first interview... I guess you could call it User Driven Development ;-).

Hope this helped

score 1 · Answer 7 · answered Mar 21 '11 at 10:36

You could use your SCM to maintain branches. Keep the master branch pristine/clean from client custom code. Do main development on this branch. For every customized version of the application maintain separate branches. Any good SCM tool will do really well with merging branches (Git comes to mind). Any updates in the master branch should be merged into the customized branches, but client specific code can stay in its own branch.

Though, if at all possible, try to design the system in a modular and configurable way. The downside of these custom branches can be that it moves too far away from the core/master.

score 1 · Answer 8 · answered Mar 22 '11 at 01:15

If you are writing in plain C, here is a rather ugly way to do it.

Common code (eg unit "frangulator.c")
client-specific code, pieces that are small and used only for each client.
in the main unit code, use #ifdef and #include to do something like

#ifdef CLIENT=CLIENTA
#include "frangulator_client_a.c"
#endif

Use this as a pattern over and over in all code units that require client-specific customisation.

This is VERY ugly, and leads to some other troubles but it is also simple, and you can cross-compare client-specific files one against the other quite easily.

It also means that all of the client-specific pieces are clearly visible (each in own file) at all times, and there is a clear relationship between the main code file and the client-specific part of the file.

If you get really clever you can set up makefiles to create the correct client definition so something like:

make clienta

will build for client_a, and "make clientb" will make for client_b and so on.

(and "make" with no provided target can issue a warning or usage description.)

I have used a similar idea before, it takes a while to set up but it can be very effective. In my case one source tree built about 120 different products.

score 0 · Answer 9 · answered Mar 21 '11 at 22:30

In git, the way I would do it is to have a master branch with all of the common code, and branches for each client. Whenever a change to the core code is made, just rebase all of the client-specific branches on top of master, so that you have a set of moving patches for the clients that are applied on top of the current baseline.

Whenever you are making a change for a client, and notice a bug that should be included in other branches, you can cherry-pick that into either master, or into the other branches that need the fix (although of different client branches are sharing code, you should probably have them both branching off a common branch from master).

How to maintain different, customized versions of the same software for multiple clients

9 Answers9

Linked

Related