38

There are countless war stories about how long a compile can take. Even xkcd made a mention of it.

Now, I haven't been programming for a long time and have mostly just been exposed to Java and Python (and Python is an interpreted language, not a compiled one). I realize it's possible that I just haven't come across projects that take very long to compile, but even for decent sized apps, it has either been instantaneous for me (usually handled in the background by an IDE) or taking no more than 30 seconds or so for an extremely large project. Even in a business environment (where the comic takes place), I've never had code take that long to compile.

Have I just not been exposed to projects with long compile times? Is this a relic of the past that is no longer something that happens in the modern day? Why would a compile take such a long time?

Thunderforge
  • 2,668
  • 3
  • 23
  • 30
  • 32
    Try compiling chromium. – UldisK Sep 26 '14 at 22:30
  • 2
    Grab a copy of the linux kernel. Do a complete build. See for yourself. Or Spring from source if you are a Java coder. As it is, this question has several answers that are answering the question as if it was a poll ("I've done 30 minute compiles..." type answers) which is an indication that the question itself isn't a good fit. –  Sep 27 '14 at 00:54
  • A recent large project took me 40 minutes to compile (40 000 source code files compiling with Maven). The workaround is to paralliize the compilation on many CPU cores. – Niklas Rosencrantz Sep 27 '14 at 05:00
  • The [standard for Python](http://en.wikipedia.org/wiki/CPython) compiles to bytecode and runs in a virtual machine; this is conceptually the same as Java. Other Python implementations might do something different, though – Izkata Sep 27 '14 at 07:38
  • 2
    Pick up a source Linux distribution (gentoo, LFS, ...) then spend days compiling every software you are installing. – Basile Starynkevitch Sep 27 '14 at 13:22
  • 6
    define long... To some kid fresh out of school 1 minute may seem long, to an oldtimer who's been in the trenches for decades a few hours don't raise eyebrows. – jwenting Sep 30 '14 at 07:15
  • How has no one mentioned GWT yet?!? I've deliberately stayed away from frontend projects because of the hell that is GWT compilations. (30 minutes spent compiling? Pshh.. That's nothing compared to 2+ hours) – Marco Sep 30 '14 at 20:05

6 Answers6

49

Compilation can take a while, especially for large projects written in languages like C, C++, or Scala. Compiling parts in the background can reduce the compilation time, but occasionally you have to do a fresh compile. Factors that can lead to long compilation times include:

  • Large code size, obviously. Large projects will have hundreds of thousands of lines of code.

  • C's #include preprocessor directive, which effectively causes the same code to be compiled hundreds of times. The macro system has similar issues, as it works on a text-level. The preprocessor really bloats up the code size that's actually passed to the compiler. Looking at a file after preprocessing (e.g. via gcc -E) should open your eyes.

  • C++'s templates are Turing complete, which means that in theory you can perform arbitrary computations at compile time. Nobody really wants to do that, but even lots of simple cases do add up to quite some time spent specializing the templates.

  • Scala is a fairly young language, and the compiler is horrendously under-optimized. Currently, the compiler uses a very large number of compilation passes (C is designed to require only two compilation passes). Typechecking is one of these passes, and can take some time due to the complicated type system featured by the language.

Compilation isn't the only thing that takes time. After the project has been compiled, a test suite should be run. The time spent on this can range from a few seconds to a couple of hours (if the tests are written badly).

amon
  • 132,749
  • 27
  • 279
  • 375
  • 14
    Actually, Scala's type system is Turing-complete, so type-checking can take an infinite amount of time, and it's impossible for the compiler to determine that. – Jörg W Mittag Sep 26 '14 at 23:47
  • 7
    Don't forget optimisations. A lot of optimisations that an (e.g.) C/C++ compiler will do are very expensive (e.g. so expensive that JIT can't afford to do them at all). For a worst case most tool-chains support whole program optimisation now, which is known to increase build times significantly. – Brendan Sep 27 '14 at 00:28
  • I've accepted this answer because you pointed out a number of things that I wasn't considering, specifically compiling the whole thing versus compiling in pieces and the fact that test suites might be included in this time of "compiling". – Thunderforge Sep 27 '14 at 02:22
  • 1
    not just test suites - code coverage analysis, automated packaging, automated deploys to the test system; there's a lot of things wrapped into an integrated build system these days. And if you're on hold until it gets to the dev or qa environment, you certainly have time for a little chair jousting. – corsiKa Sep 27 '14 at 06:00
  • 1
    Great answer, I would just note that the spread of possible compile times can be a lot larger. I have worked on projects where a full compile could take two to three days (yes that was horrendous!) and I imagine there are worse offenders out there. – Roy T. Sep 27 '14 at 12:46
17

It's by no means a relic of the past. One of the projects I work on requires 45 minutes for a clean build from scratch. Aside from our own code we also have to pull and build the source from several large C and C++ libraries from external repositories. Compiling and linking C and C++ code is computationally expensive. As you point out, Python is typically implemented as an interpreted language, and Java usually uses a JIT (Just in Time) compiler, so your projects are skipping the upfront compilation and linking costs altogether. The price you pay is in longer start up times and (for Python at least) slower execution speed.

When build times get that long it becomes more important to take advantage of continuous integration systems like Jenkins or TeamCity. This allows individual developers to (mostly) avoid the pain of building from scratch, while still testing that changes aren't breaking the build.

Charles E. Grant
  • 16,612
  • 1
  • 46
  • 73
  • 1
    javac isn't "*skipping the upfront compilation and linking costs altogether*". It skips a lot of optimisation costs, but it's still transforming source to bytecode and doing lots of static checks in the process. It does about as much linking as a C compiler. The real performance difference is that Java's compilation process was designed in an era when it was assumed possible to load the entire program and its dependencies into memory at once rather than having to break it down into tiny chunks and reprocess the same files thousands of times. – Peter Taylor Oct 01 '14 at 10:57
10

Large projects can take a long time. It can be an hour or more for a big enough project. There are a couple of libraries that I have to compile from source on my computer that take a very long time - e.g. opencascade. The Linux kernel itself also takes quite a long time if you have to build it from scratch.

However, there are other compilation-like processes that can take far longer. Digital circuit design (either for ASICs or FPGAs) requires a place and route step. The place and route step is where the placement of individual logic gates, flip-flops, registers, RAMs, and other components is determined along with the routing for the interconnect wiring. The software uses timing models to determine the gate and routing delays for possible placements, compares these to the limits provide by the timing constraints, and then adjusts the placement locations and wire paths to try to get the timing requirements met. Sometimes the software will even have to resize gates and add buffers in order to meet timing. This step is extremely computationally intensive and can take many hours or even days to complete. It also does not really parallelize very well. There was an FPGA design that I was working on a year or so ago that consumed about half of a Virtex 6 HXT 565 FPGA (~300k out of 565k LUTs) and took about 7 hours to complete place and route. I can't imagine how long it would take to run place and route on something like a Core i7 CPU design - probably at least several weeks.

alex.forencich
  • 209
  • 1
  • 3
4

Other answers have already mentioned that yes, code on large projects, where large means 500k lines and up, can take significant times, especially when building from scratch.

The additional point is that some projects have to be built for multiple target environments. When the machines that host those environments are not available, the build must be done by cross-compilation, in serial on the machines that you do have. This can lead to significant build times. For one project I worked on, the nightly build would take 10 hours. Woe betide that you were the one who broke it!

I would add that you wouldn't get away with any such excuse for time wasting. A professional person should be planning their tasks so that they do have something useful to do in such periods.

andy256
  • 3,156
  • 2
  • 15
  • 20
3

A little of both. C++ (and C to a lesser extent) were notorious for their slow compilation times, especially on period hardware. Around the turn of the millennium, I worked on a project that took about 4 hours to build due to macro shenanigans.

These days things are better, but 30 seconds is fairly low in my experience - especially in formal builds where things need to be checked out from source control, unit tests run, installers built, and everything sent out to some SAN somewhere.

Telastyn
  • 108,850
  • 29
  • 239
  • 365
2

It depends on the project and the environment in which it is compiled. I've worked on C++ projects that took several minutes to compile (set up as multiple projects in MSVS), which is probably enough time for a swordfight.

If you work for a large company with a huge code and data base (Proctor and Gamble, Google, etc.) or for a small company or startup focused on one or two primary products which are very complex (e.g. scientific simulation and rendering), then waiting for a big project to compile is a realistic thing to expect even on powerful machines. This can affect how you develop and debug code (as well as how often you choose to update and merge changes through versioning).

Trixie Wolf
  • 414
  • 2
  • 6