34

There is an overhead associated with continuous integration, e.g., set up, re-training, awareness activities, stoppage to fix "bugs" that turn out to be data issues, enforced separation of concerns programming styles, etc.

At what point does continuous integration pay for itself?

EDIT: These were my findings

The set-up was CruiseControl.Net with Nant, reading from VSS or TFS.

Here are a few reasons for failure, which have nothing to do with the setup:

Cost of investigation: The time spent investigating whether a red light is due a genuine logical inconsistency in the code, data quality, or another source such as an infrastructure problem (e.g., a network issue, a timeout reading from source control, third party server is down, etc., etc.)

Political costs over infrastructure: I considered performing an "infrastructure" check for each method in the test run. I had no solution to the timeout except to replace the build server. Red tape got in the way and there was no server replacement.

Cost of fixing unit tests: A red light due to a data quality issue could be an indicator of a badly written unit test. So, data dependent unit tests were re-written to reduce the likelihood of a red light due to bad data. In many cases, necessary data was inserted into the test environment to be able to accurately run its unit tests. It makes sense to say that by making the data more robust then the test becomes more robust if it is dependent on this data. Of course, this worked well!

Cost of coverage, i.e., writing unit tests for already existing code: There was the problem of unit test coverage. There were thousands of methods that had no unit tests. So, a sizeable amount of man days would be needed to create those. As this would be too difficult to provide a business case, it was decided that unit tests would be used for any new public method going forward. Those that did not have a unit test were termed 'potentially infra red'. An intestesting point here is that static methods were a moot point in how it would be possible to uniquely determine how a specific static method had failed.

Cost of bespoke releases: Nant scripts only go so far. They are not that useful for, say, CMS dependent builds for EPiServer, CMS, or any UI oriented database deployment.

These are the types of issues that occured on the build server for hourly test runs and overnight QA builds. I entertain that these to be unnecessary as a build master can perform these tasks manually at the time of release, esp., with a one man band and a small build. So, single step builds have not justified use of CI in my experience. What about the more complex, multistep builds? These can be a pain to build, especially without a Nant script. So, even having created one, these were no more successful. The costs of fixing the red light issues outweighed the benefits. Eventually, developers lost interest and questioned the validity of the red light.

Having given it a fair try, I believe that CI is expensive and there is a lot of working around the edges instead of just getting the job done. It's more cost effective to employ experienced developers who do not make a mess of large projects than introduce and maintain an alarm system.

This is the case even if those developers leave. It doesn't matter if a good developer leaves because processes that he follows would ensure that he writes requirement specs, design specs, sticks to the coding guidelines, and comments his code so that it is readable. All this is reviewed. If this is not happening then his team leader is not doing his job, which should be picked up by his manager and so on.

For CI to work, it is not enough to just write unit tests, attempt to maintain full coverage, and ensure a working infrastructure for sizable systems.

The bottom line: One might question whether fixing as many bugs before release is even desirable from a business prespective. CI involves a lot of work to capture a handful of bugs that the customer could identify in UAT or the company could get paid for fixing as part of a client service agreement when the warranty period expires anyway.

Phil C
  • 1,956
  • 1
  • 18
  • 34
  • 13
    It can be beneficial even for a team of one. Especially if you've got a long running test suite, it is much better to get automatically results of an overnight build and test run than doing it manually all the time. – SK-logic Apr 02 '11 at 10:01
  • 3
    @Carnotaurus, local clone of remote git repository gets rid of the timeout from source control. Network issues - for building the product? Now, really... –  Apr 03 '11 at 08:12
  • 3
    @Carnotaurus red light due to data quality or infrastructure is a codesmell of [Fragile Tests](http://xunitpatterns.com/Fragile%20Test.html). See also [so: managing-the-maintenance-burden-of-unit-tests](http://stackoverflow.com/questions/208302/managing-the-maintenance-burden-of-unit-tests) – k3b Apr 03 '11 at 08:59
  • @k3b: There appears to be nothing in that link about data quality or infrastructure. Red lights can occur as a result of these, independent of code (e.g., partial data extracts, network errors, etc). So please explain yourself. – Phil C Apr 03 '11 at 09:14
  • 1
    The more a test depends on external aspects the more fragile it is. Example: If a test only succeeds if customer XY is in database the test if fragile unless the test-setup makes shure that this prerequisit exists by inserting the data itself if neccessary. – k3b Apr 03 '11 at 09:22
  • 6
    "Bottom line: Why make working products when we can get someone else to pay us for fixing it, because it isn't like they expected the software to work in the first place"? It may not meet the legal definition, but that sounds like fraud to me. – Chris Pitman Jan 02 '12 at 17:35
  • That doesn't even come close to what was said. I'm pleased you don't work for any company that I do work for. – Phil C Jan 03 '12 at 00:36
  • @Carnotaurus Your edit does not seem to be directly related to your question or CI in general. The bottom line specifically is not related to CI but to your company's policies - does your company prefer to have a good reputation and be able to work on projects that require high quality. Or does your company prefer to keep looking for new clients that haven't heard about it and then leaving them with a bad taste after deliveries and hoping potential clients don't hear about it. Also automated tests should pass on any developers machine at any time even if you are not using CI. – Danny Varod May 22 '12 at 11:06
  • FWIW, i think CI is totally worth it. But CCNet + nant in particular is not the most effective / cost effective route, as you basically start with almost nothing and have to do it all yourself. compared to say TeamCity, where you have many common things built in and you just have to define steps and plug in specific values for common things, that is so much easier to manage. – Andy Jul 09 '16 at 19:21

12 Answers12

43

Setting up a CI-engine is akin to setting up a fire alarm in a house.

In my mind the benefits do not correlate with many developers, but with a large code base. The CI-engine actively do all the boring work you do not want to do yourself, and do it everytime.

If you break a remote module which you haven't touched for a long time, you are told immediately. Not only compilation wise, but also functionwise if you have set up unit tests.

Also note that if you let you CI-engine do all the boring work, including setting up installers etc, you do not have to do it manually. You can just check in your source, and await the finished product being built in the standard location. (EDIT: The CI-engine also works in a well-defined environment, avoiding any developer specific setups, ensuring reproducibility)

This is also a part of quality assurance.

EDIT: After writing the above, I have had experience with the Maven build tool for Java. Essentially this allows us to keep the complete CI-configuration inside the project (with pom.xml) making it much simpler to maintain the CI-installation and/or migrate to another CI-engine.

  • 1
    @Carnotaurus, in that case the red light is being used for transient errors too. Sounds like a limitation in the CI-engine you are using. –  Apr 03 '11 at 07:49
  • @Carnotaurus, sounds like management did not appreciate the value of the CI-engine giving you better means. However, if you used a distributed source control system, the first thing to do was to have a local clone and let all jobs refer to that instead of the remote. –  Apr 03 '11 at 08:10
  • Ironically, it was local and accessible by LAN. – Phil C Apr 03 '11 at 08:23
  • @carnotaurus, so the clone could be done really fast then? –  Apr 03 '11 at 08:26
  • In principle it would; However, it would even timeout on the retrieval to produce the clone. I suspect that a low priority was given to the network packets... I think it's red tape all the way on this one. I'm pleased that I do not deal with chaps like this in my new company. – Phil C Apr 03 '11 at 08:33
  • +1 for pointing out that the CI-engine isn't just some fancy setup but should **save** time, rather than consume it. I've never felt so sure of the quality of what I produce that since I have a CI setup verifying *every* commit (even the little insignificant one that could not possibly break anything). In fact, we almost abuse it: for simple fixs we just submit them instead of running the full test-suite ourselves; the CI reports the issue (if any) to the patch author directly. Saves considerable time. – Matthieu M. Apr 03 '11 at 15:53
  • @MatthieuM. Having given it a fair trying, I believe that it is expensive and there is a lot of working around the edges instead of just getting the job done. It's more cost effective to employ experienced developers who do not make a mess of large projects than introduce and maintain an alarm system. – Phil C Jan 01 '12 at 10:51
  • @Carnotaurus it may be more cost effective in the short run but not necessarily in the long run. Developers come and go. –  Jan 01 '12 at 11:01
  • @ThorbjørnRavnAndersen It doesn't matter if a developer leaves because processes that he follows would ensure that he writes requirement specs, design specs, sticks to the coding guidelines, and comments his code so that it is readable. All this is reviewed. If this is not happening then his team leader is not doing his job, which should be picked up by his manager and so on. – Phil C Jan 01 '12 at 11:06
  • 2
    @Carnotaurus in my experience the work done to thoroughly document every aspect of everything regarding e.g. doing a release and then doing said releases a number of times, is larger than capturing the workflow in an ant or maven script which can then be executed by a robot any number of times. Said robot also works in a clean room environment ensuring that builds are reproducible. –  Jan 01 '12 at 11:37
  • @ThorbjørnRavnAndersen Nant scripts only go so far. They are not that useful for, say, CMS dependent builds for EPiServer... Anyway, it's not so much the software build and the deployment I have issues with, it's the writing of unit tests, maintaining full coverage, and ensuring a working infrastructure for sizable systems. It's a lot of work to capture a handful of bugs that the company would get paid for fixing as part of a client service agreement anyway. – Phil C Jan 01 '12 at 11:46
  • 1
    Note that the breaking I am looking for is not failing unit tests but that the actual programs we ship can still build. It is not as important anymore as we migrated to maven artifacts instead of compiling everything for each release, but it is still important to know that the build is fully functional. We just need the Continuous Deployment part too. –  Jan 01 '12 at 11:50
  • 2
    @Carnotaurus: I am not talking about an alarm system. If the tests fail, the patch is not integrated into the main repository (at all), ensuring that whenever you checkout/clone you get a fully working copy and can start working right away. Now, I agree with you that tests can be difficult to write and maintain... but I have worked with and without test, and I see a definite quality improvement *with* them. So the question is: automated or not ? In my experience, it takes about as long to test manually than to write the automation script (but I work in a large corporation with tools for that). – Matthieu M. Jan 01 '12 at 12:20
  • I've updated the question now. I hope this clarifies my position. – Phil C Jan 02 '12 at 11:04
  • 5
    "_It's a lot of work to capture a handful of bugs that the company would get paid for fixing as part of a client service agreement anyway_" - well in that case you have economic incentive NOT to produce software with as few bugs as possible, and in that case all this does not apply to you. –  Jan 07 '12 at 11:44
33

It's not how many developers, but how many steps it takes to get from 1 to n (inclusive), where 1 & n are ...

1: Checking out code
And
n: having installable\deployable packages

If n < 2 you maybe don't need CI
otherwise, you need CI

Update
From reading your findings I can only conclude that you approached CI from the wrong direction and for the wrong reasons.

Binary Worrier
  • 3,112
  • 2
  • 25
  • 23
  • 1
    +1 - I've quite a lot I could add to the above - but this is very close to the core of why I like CI... not only for what it does but also for what it requires you to do to make it work (those requirements being things you really ought to do anyway). – Murph Apr 02 '11 at 12:18
  • 2
    @murph: Yup, and it brings little advantages like 1) knowing what's in your repository will build, 2) single click build, 3) being able to release a version at a moments notice and So much more – Binary Worrier Apr 02 '11 at 12:51
  • So it's fair to say that it depends on the complexity of what is to be released, measured by its number of steps to deployment as being able to "test" separate layers? – Phil C Apr 03 '11 at 07:28
  • No, it just the number of steps. If your build\deploy task is simple, you can live without CI. However this ability to realease at a moments notice (to client, to QA, to stress test etc) is the core of CI. The other things, knowing immediately if the build is broken, automated running of unit test, visibility of code changes to the team, are all very valuable side effects of CI. – Binary Worrier Apr 03 '11 at 09:57
  • Actually, even if you have a trivially simple build it's worth doing fir all the other benefits, but rapid deployment is a bacon saving insurance that CANNOT be under estimated! – Binary Worrier Apr 03 '11 at 10:09
  • I still cannot buy the idea of single step CI - there is nothing easy than just deploying simple software and than doing a full regression test – Phil C Apr 05 '11 at 16:52
  • 1
    @Carnotaurus: Sorry mate, but I'm not sure I know what you're trying to say. CI has nothing to do with testing. Yes, you can - and should - run unit tests as part of the build, but they shouldn't be dependant on anything that isn't setup as part of the test. However CI does aid testing. One of the - _many_ - benefits of CI is that it allows a QA team to immediately & seemlessly pickup new code changes. **CI = The Ability To Release Immediately** – Binary Worrier Apr 06 '11 at 06:50
  • How can you say that when CI is set up to facilitate testing visibility and that just because something builds does not mean that it is ready for release or do you mean ready for internal testing? – Phil C Jun 10 '11 at 19:19
  • @Carnotaurus: "Release" for me means Release out of Dev, so yes, releasing internally to QA/Test etc. However the important thing is once that release passes QA, you have nothing else to do to _Release_ it into the wild. – Binary Worrier Jun 21 '11 at 11:42
  • @BinaryWorrier Nothing else to do? There is if the hosting environment is outside your control but that's got nothing to do with CI. – Phil C Jan 01 '12 at 10:53
  • 1
    @BinaryWorrier - I think step 1 is checking in the code ;) –  Feb 04 '12 at 12:16
  • Agree that it's not proportional to the number of devs. I once set up a CI server to solve the problem that shippable builds were extremely time consuming (essentially a full developer day). But in that instance, the typical benefits of CI (immediate notification of bugs etc) were really just a bonus. – Steve Bennett May 04 '12 at 07:55
  • @Carnotaurus: Well, d'ah. – Binary Worrier May 22 '12 at 09:40
10

It can be worth the effort even for a team of one. This is especially true when you are developing cross platform code and you need to ensure that your changes will work on both platforms. For example, Microsoft's C++ compiler is more accepting than GCC, so if you develop in Windows but need to support Linux as well, having a CI system tell you when your build breaks on Linux is a huge help.

Some CI systems are quite easy to set up, so the overhead is not really that huge. Try Jenkins or Hudson for instance.

mch
  • 201
  • 1
  • 4
  • I had not considered a cross-platform scenario. If different compilers are needed to create different builds then there is a strong case for CI - take an upvote. – Phil C Apr 03 '11 at 07:49
4

As you say there is an overhead cost of setting it up and keeping it running.

But the question of where is the break even point is not a function of how many people you have in your team, but rather a function of the length of your project.

Having said that, there is a part of the setup cost that you can use in all your future projects, so in the long term the overhead cost may approach zero.

Stephen Bailey
  • 2,236
  • 14
  • 14
  • So length of the project (size of project) is important for you CI? I have found the false alarms to be very costly. – Phil C Apr 03 '11 at 07:37
  • Yes, you need to have the time to pay back the setup and learning curve costs. In theory you should over time learn how to eliminate the false alarms. – Stephen Bailey Apr 03 '11 at 09:39
  • Yes, this is theory – Phil C Apr 05 '11 at 16:54
  • Well, no, its practice. Over time, you make a note of what tests are fragile and what tests aren't, and you don't worry too much if your fragile tests break unless they break several times in a row. You write more isolated, robust tests as you learn how, and you let legacy coverage build over time instead of doing it all at once. In practice, CI is a not a silver bullet, its a process change that takes time and eventually leads to less buggy software. – philosodad May 22 '12 at 13:45
3

I set up Jenkins this week to build a small .NET project I'm working on. I integrated it with my Git source control so that it triggered a build on every commit. I integrated the unit tests into the build. I integrated static analysis in the form of FxCop and StyleCop violations.

Now every time I check in, it checks out all of my code, builds it, increments the version number across all assemblies, tests it, analyses it for FxCop and StyleCop violations, archives the build and records the results in a number of graphs so I have visibility over time of test results and violations.

Doing that from scratch takes an hour or so (maybe a day with Google if you've not done it before). It costs nothing because all the tools are available for free.

If, as and when the project builds I have a high quality infrastructure that will grow with it at no cost. If or when new developers join the project I can get total visibility on their work and their impact on the project for no cost.

So the only scenario I can see CI not being worth while is either for a project that will take a day or so and then never get revisited, or one in a language where there are no existing / free tools available and the cost of acquiring them is disproportional to the work.

  • As I said, it is the unit tests to cover legacy code that presents a major cost. Also, we are not talking about a one man team here either... I appreciate the heads up on Jenkins though. – Phil C Feb 04 '12 at 12:16
  • 1
    For what its worth, I gained considerable benefit from running a CI server without having any tests - just from the confidence in clean builds and the availability of deployment packages. – Murph Mar 24 '12 at 14:14
1

CI is always always always worth it: The sense of security it gives you enables you to work at a faster rate than would be possible otherwise. The problem that you have seems to revolve around unit tests. I agree that unit tests are very expensive, but also think that (for many things) they are the least worst option that we have. Personally, and for the sorts of systems that I tend to work on, I swear by system-level tests operating on a combination of real-world and (possibly synthetic) pathological cases; It is cheaper than unit tests, and gets into those hard-to-reach corners of your conceptual universe: Donald Rumsfeld's "Unknown Unknowns".

William Payne
  • 1,161
  • 8
  • 20
  • I like the bit about system-level UI tests. Much of testing gets lost in a fog of unit tests of questionable quality and coverage. – Phil C Jan 06 '12 at 15:02
  • Coverage is a question of human psychology also; we have an innate tendency to write tests for those cases that we have already thought about. It is for this reason that to be certified at a high SIL level, unit tests need to be written by a separate team from that which developed the system under test, and even then, there are many cases & situations that are obvious to everybody **only** in retrospect. Rigorous checking of code coverage is needed; but extraordinarily difficult and expensive. – William Payne Jan 07 '12 at 00:01
  • ... Hence the benefit that you get by throwing the nastiest junk that you can find into the system at the top level and seeing what happens... :-) – William Payne Jan 07 '12 at 00:02
  • 1
    +1 - Totally agree. Some parts of a system are just not unit testable (i.e. the edges like at the database). Sure you can mock the db, but that leaves the code that talks to the db untested. At some point you need to write some real world tests that integrate your systems and talk to other systems. When you do that you always find bugs that you missed in the cosy clean world of the unit test and the mocking framework (LINQ to SQL springs to mind). –  Feb 04 '12 at 12:13
  • Actually, I recently discovered an interesting take on fuzz testing that might redeem unit testing: http://jester.sourceforge.net/ – William Payne Mar 07 '12 at 14:31
1

If you can verify all aspects of your project after each change, then you don't need CI.

In all other cases, it's a net win.

Macke
  • 2,576
  • 16
  • 16
  • I think this is where I'm coming from too. It's much cheaper too. – Phil C Apr 03 '11 at 07:45
  • What do you mean by verification in this case? If it is automated, happens as frequently as possible, and helps identify mental slips & oversights, then I am all for it. :-) – William Payne Feb 06 '12 at 14:28
1

Overhead is minimal. I'd say for one man projects it would be useful. Once you reach two it's invaluable.

I agree that for one man projects if you have a "one step build/verify" then you might be okay with continuous integration, but in those cases you've done most of the hard work to set up CI so might as well just put it in a formal system (CC, Hudson, TeamCity, etc).

Mike
  • 111
  • 3
1

The question of, when does CI pay for itself, is a hard one to answer on a new project.

On an existing project, it's much easier to see. If you find a critical bug at the development piece of the supply chain, you know that problem exists at the earliest point possible. The cost for fixing that is now the lowest. If that problem exists in any environments above development, your cost is higher.

Now, management might decide to release with a bug, but that's a business risk. At least now it can be mitigated through that risk assessment, rather than late night phone calls/panics, which, if they were billed at hourly rates, end up being mighty expensive.

Another thing to consider in terms of cost. What is your QA department like? Is it manual testing? If you go CI, you may be able to reduce overall QA costs by automating those scripts against your dev box. You may be able to reduce the number of QA folks supporting your project by getting them involved in the CI phase.

Mike Cornell
  • 121
  • 3
  • 1
    To replace a manual regression test with some UI automated tests takes quite a bit of time and skill. In one company, one chap wrote about 40% of the tests and left the company. Several years on, the tests still have no been written. I bet this is typical. – Phil C Apr 03 '11 at 07:44
  • No, it's not typical. – quant_dev Apr 03 '11 at 09:48
  • I haven't seen much counter-evidence – Phil C Apr 05 '11 at 16:55
  • If you write integration/acceptance tests in natural language DSL like Gherkin, you can easily translate the automated test to manual. So I don't see that as a problem. – Mike Cornell Apr 06 '11 at 13:33
  • @Carnotaurus - I think it is typical. I've seen it twice too. UI automated tests are hard. – Rocklan May 10 '13 at 02:13
0

One. Yes, one is enough to start using continuous integration.

java_mouse
  • 2,627
  • 15
  • 23
0

It is not a matter of how many developers, but of

a. How much time you save using automation and how often.

  • Time saved on building after integrations, running automated tests and creating setups * frequency of check-in = percent of time saved.

b. How many versions/branches/products you have.

  • If you have one developer working on two different branches then the time saved is doubled, since each branch would require building, testing and packaging.
Danny Varod
  • 1,148
  • 6
  • 14
0

Always use it, regardless of the size of the team. If it's just you for example, who knows, you might be coding from your laptop at starbucks then continue from your beefier system at home?

The overhead really isn't that bad.

Julio
  • 1,854
  • 13
  • 10