Splitting large C++ project

Question

Like it happens to many, our C++ project grew larger and larger, and it finally hit the point where maintainability starts to be a concern, mostly due to build times: even by using ccache, every change requires anywhere from 30 seconds to 5 minutes to rebuild and test (actually it seems that "linking" is the time-consuming phase, but we are still investigating).

The project consists mainly of a big scientific library, along with some applications which make use of it. For example, there are applications to augment simulation models, or language bindings (python) for integration into other software packages.

Now, it would seem natural to split the library into smaller libraries: the code is already organized in "modules", so this would not be an issue. The problem is with dependency management: we can think of the whole project as a big DAG, where each node is a library module, or external library (due of the project nature, there are many external dependencies on other scientific libraries).

So ideally, when working on a module, we would like to rebuild only that module, plus maybe the related test app, and link it with its dependencies. This would speed up the development cycle considerably.

Consequently, the idea is to handle each "module" as an independent project, that will be separately built into a (static) library. I have two concerns with this approach:

• Imagine that I am working on app A that depends on library B, which itself depends on library C. During development, I find that I need a new feature in library C, so I go ahead and code that. I now have to manually rebuild library C, then library B and finally library A before continuing with the development. When the dependency chain grows larger, this can be problematic.
• If the dependencies of one module change, how to manage the impact on the whole DAG (other modules that change as a consequence)?

I have been researching into several tools that might assist with my scenario. Git submodules is one option, but it does not address my concerns above. Apache Ivy seems not suitable for what I am trying to achieve. I suspect that the answer involves some powerful build system (we are using qmake), so I looked into cmake (but I found its scripting language limited and hard to learn), and then buck and pants.

I am still unsure if this is the way to go, so I would be grateful if someone could share advice and/or experiences.

Answer 1

You mention static libraries in your question. I'm guessing (perhaps wrongly) that you code a program for Linux.

You should use shared libraries when possible. Then you need much less linking time (at the expense of runtime linking, which usually is fast enough, at start time).

During development, I find that I need a new feature in library C, so I go ahead and code that. I now have to manually rebuild library C, then library B and finally library A before continuing with the development.

This should not be the case with shared libraries (as long as their API did not change).

Consider perhaps some architecture using plugins, and which dynamically loads them (e.g. with dlopen on POSIX).

it seems that "linking" is the time-consuming phase

You need to be sure that is the case (e.g. by passing -time or even -ftime-report to g++). Then you might use the gold linker and/or visibility function attribute with g++.

Be sure that the APIs of your sub-modules or libraries don't change often.

Avoid too small C++ source files, e.g. prefer having ten C++ source files of two thousand lines each rather than a hundred C++ source files of only a two hundreds lines each. A C++ source file can define and implement several (related) functions or classes. Remember that C++ don't have modules, and that most standard headers (and your own ones) are quite big files (for example #include <vector> expands to about 10KLOC on my Linux desktop); what counts for compilation time is the size of preprocessed form, and template expansions need a lot of time too. See also this.

BTW, you also need to use a good build automation tool (such as ninja, GNU make - with parallelisation enabled thru make -j 8, etc...). You could also use distributed build tools (distcc, icecream, ...)

You might also (with GCC) consider using a precompiled header. Then, you'll probably want to have a single header file including all the other ones. You might want to use the PIMPL idiom.

See also this answer to a relevant question.

---

every change requires anywhere from 30 seconds to 5 minutes to rebuild and test

BTW, a five minute incremental build time seems rather small to me. Why are you complaining? I'm old enough to have had in the 1990s, on powerful Sun workstations of that time, build times of almost an hour (for a software of a few hundred thousands lines I wrote alone during several years). And I sometimes contribute to GCC itself even these years, whose rebuild time takes several hours.

Answer 2

There are already fine answers related to build systems and shared libs, but I'll tackle this from another angle I don't see people applying so often.

To me it's useful to separate stable code (as in unlikely to ever need further changes) from unstable code (code that naturally warrants further changes as the software expands). If you cannot do this and see the entirety of your codebase as potentially moving parts, even if the interfaces are completely stable, I think you are doing something wrong.

We already tend to do this naturally for open source third party libraries. We generally build them once, create a dylib ideally to avoid static linking times, and just look up symbols and call functions in them at runtime. We don't find a need to constantly rebuild them over and over since we have no interest in touching their source code, only using the available functionality -- basically build once and use forever. The code is 100% stable in that regard (there is no reason for it to ever change, at least not in our hands).

So for third parties I personally put them in a "third_parties" subdirectory and I only build that when I add a new third party library or replace an old version with a new version which is extremely rare, like once every few months minimum. I'm not rebuilding those over and over on a daily basis.

Same thing should be able to apply to your own codebase. I have a similar thing where I have a "libs" subdirectory with my own code and rarely ever build the resulting libraries since the code is very stable, reliable, efficient, tested thoroughly with unit tests, been through static analysis, and not something I need to change in the future. It's separated far away from the unstable code outside of the "libs" subdirectory which does need to change on a daily basis which does get rebuilt repeatedly.

So separating your codebase this way where stable parts you rarely, if ever, need to change from unstable parts which constantly warrant changes can be a useful way to organize your codebase -- not only in terms of optimizing build times but in order to better separate away stable packages from unstable packages with a goal to end up with as much stable code as possible that you can confidently say will not warrant any changes in any near future.

When you do that, those stable parts of your codebase can be built separately and away from the unstable parts using them. The unstable parts shouldn't take so long to build as your stable set of libraries expand and expand while the unstable parts shrink and shrink.

However, this tends to imply some code duplication. To be able to create a stable image library, it can't depend on, say, some unstable math library or else changes to the math library will warrant a rebuild if not further source changes to the image library. So I actually find it useful sometimes for, say, an image library to duplicate some math routines in order to become independent from any auxiliary math library. In that case, decoupling the code in this way through some modest duplication of logic can actually help make those packages a lot more stable, eliminating reasons for them to have to change and/or be rebuilt over and over. If the code is well-tested and works beautifully for years to come and might hardly ever have to be rebuilt, the modest duplication required for the library to achieve its independence and stability is hardly a problem.

Further it helps to achieve minimalism in your interfaces. If your image library aims to implement every single image operation ever imaginable to mankind, then its ambitions and monolithic nature will warrant neverending changes. It can never hope to become a stable library with such goals. If it only aims to provide basic image operations or even none at all with the ability to create image operations outside of the library using what the library provides, then it can potentially achieve a state of perfect stability (finding no reason to change in any near future).

So anyway, if you are going to start splitting up a codebase, I'd suggest to start off splitting the stable, well-tested parts where you're pretty sure you don't need to change them any further from the unstable parts where you at least anticipate some possibility of a need for future changes. Building the stable parts should invoke a separate build process (rarely applied) from the unstable parts (frequently rebuilt).

Answer 3

Not sure what your environment/budget is but here are a few suggestions:

• Visual Studio 2017 especially with /MP, precompiled headers, incremental builds, and possibly C++ modules
• Incredibuild (never used personally)
• Octobuild (never used personally)
• Zapcc can be much faster than mainstream compilers

The usual advice applies for single and multiple projects: run the best CPU+RAM+SSD (or even ramdisk) you can buy, disable on-access virus scanning, and infrequently change headers where possible.

Answer 4

Just in case it’s not obvious: Use precompiled headers, and tell FCC’s to run more than one instance of the compiler.

For linking, putting code into a shared library should help. Some build systems use link-time optimisations, that might add to build times and you may be able to turn it off.

And what kind of computer do you have? If you suffer from build times, 8 cores and plenty of RAM can be an easy solution. And if it is possible to switch from gcc to Clang, that might help.

Answer 5

In a rather large project I have worked on, we had last known good versions of libs from other projects published for consumption, unless you had that project checked out, in which case it would compile it in.

However, you might need to revise your bounded contexts or your workflow to really fix the issues. Either define module boundaries to reduce the need to work on C which tickles down to B which tickles down to A, or do the work in a logical fashion so prep work gets put into C's release channel before work on B and A starts.

Answer 6

You needed to use "partial" linking trick, together or instead of libraries.

Gnu ld could combine multiple objects into a single one. It's extremely useful on huge projects because it allows two-phase linking: "preliminary" and "final". Proper using of this trick could dramatically reduce CPU resource required for linking and loading of an application.

Often it is done next way:

ld -relocatable a.o b.o -o c.o

PS: it's not a bad idea to combine described trick with other same-goal tricks.