8

Builds are slow and take time. We can get MSBuild to parallelize them, but only on a single machine, not across a cluster. Why hasn't anyone come up with clustered build solutions in the .Net space? I know such solutions exist in the C++ space (e.g., IncrediBuild, Electric Cloud, etc.) but we .Net developers are forced to build huge solutions on individual machines. Anyone know if there's some unsolvable technical difficulty in this?

Just curious.

Alex Feinman
  • 5,792
  • 2
  • 27
  • 48
Dmitri Nesteruk
  • 189
  • 1
  • 1
  • 10
  • 1
    I don't know what exactly you define as "slow" and "huge". But I'm pretty certain it's peanuts compared to what the larger, very template-heavy C++ projects take. You don't need to wait for thousands of template instanciations, the associated compiletime computations, countless optimization passes, native code generation, etc. that make C++ compile times so infamous. –  Apr 22 '11 at 19:29
  • I've seen Visual Studio Solutions that take +60 minutes to compile; on a modern machine. –  Apr 22 '11 at 19:32
  • Code-weaving usually slows down compile time quite significantly. PostSharp, CodeContracts... – GregC Apr 22 '11 at 19:42
  • 2
    @Nate: **C#** code? A C# project that I worked on (~600k lines spread across around 150 projects and 20-30 solutions) built from scratch in about 4 minutes. In my experience, it wouldn't be worthwhile having an IncrediBuild-like system for C#/.NET – Dean Harding Apr 23 '11 at 08:07
  • @Dean: well... if you are actually *okay* with waiting 4 minutes, there's nothing to discuss, is there? :) – Dmitri Nesteruk Apr 23 '11 at 08:19
  • 2
    @Dmitri: Well, I've also worked on C++ projects that took over 4 **hours** to build, so yeah, I'm OK with 4 minute build times :-) That's a rebuild-from-scratch as well, just building a single project or two typically takes seconds... – Dean Harding Apr 23 '11 at 08:30
  • @Dimitri the company I work for has old C++ builder software which takes at least 30 minutes for a build the biggest .net projects they have build in maximum 5 minutes :D – Knerd Dec 21 '14 at 13:47
  • I'm happy to inform you that IncrediBuild actually supports the distribution of .Net projects (both C++ and C#). You'll get this behavior out-of-the-box upon IncrediBuild's installation when building using IncrediBuild's menu under Visual Studio or the command line (make sure you have IncrediBuild's C# solution as part of your license - IncrediBuild's trial version comes with it automatically). Pay attention that IncrediBuild can only distribute entire C# projects so in order to benefit from IncrediBuild under C# you'll need to have many non-dependent C# projects or a mix of C# and C++. Discla – Dori Dec 21 '14 at 12:13

2 Answers2

6

" Why hasn't anyone come up with clustered build solutions in the .Net space?"

If compilation were to be clustered:

  1. The compiler will have to first break the compilation into different threads which can be done on indepenent machines.
  2. Send the code to different machines for compliation.
  3. Wait till all machines have completed their task
  4. Get the results from all machines in the cluster together.
  5. Put together the results and link them up.

steps 1 and 5 are dependant on the nature of the project : Too many interdepencies and the number of distribuable tasks is drastically reduced. Step 2 and 4 depend on the status of your network. The time taken to send and recive the object files adds to the compile time and beats the benefit of clustering. Depending on the project you probably may end up with the same time for compiling on a single machine as you would for a cluster.

DPD
  • 3,527
  • 2
  • 16
  • 22
  • 1
    If you're saying that it's not worth it, I *totally* disagree :) Regarding your points, 1. is already being done locally so could be done remotely (e.g., with MSBuild), 2. is really a non-issue, I use DropBox and it does peer-to-peer on my LAN so, basically, it's instant, 3. is why we are doing it in the first place, 4. is same a 2. i.e. a non-issue, and 5. is the only really 'tough' thing. – Dmitri Nesteruk Apr 23 '11 at 07:39
  • I am saying that it wont be worth it except for very large projects. Compilation is a CPU intesive task. It would have been a geat idea in the days of yore but not today. What appears to us as "instant" may not really be for the computer. What I understand is that network transfer is much slower than single CPu compilation. – DPD Apr 23 '11 at 10:25
1

Pretty much any CI system does that. If you're talking specifically about on the dev machine: Automated Build Studio is one such example.

That said, most of the products I've worked on/with never require a full rebuild on the dev machine as the code is usually loosely coupled enough and the devs rarely need to make changes to that many different assemblies between checkins.

Steven Evers
  • 28,200
  • 10
  • 75
  • 159
  • If it's loosely coupled, you're still limited by the # of cores on your machine, not by the number of cores on *all* your machines. – Dmitri Nesteruk Apr 23 '11 at 07:40
  • @Dmitri Nesteruk: Yes, but if it's loosely coupled, an incremental build won't perform a whole rebuild, hence you only have to build a couple files/projects instead of the whole application - thus, distributed compilation isn't really necessary. – Steven Evers Apr 23 '11 at 17:34