20

I've always launched builds after each commit, but on this new project, the architects just asked me to change the frequency to "one build every 15 minutes", and I just can't understand why that would be a good reason vs "building on each commit".

First off, some details :

  • Objective-C (iOS 5) project
  • 10 developpers
  • each build actually takes ~1 min, and includes build and unit testing.
  • the integration server is a Mac Mini, so computing power isn't really a problem here
    • we use Jenkins with the XCode plugin

My arguments were that if you build at each commit, you can see right now what went wrong, and correct directly your errors, without bothering the other devs too often. Plus our tester is less bothered by UT errors this way. His arguments were that devs will be flooded by "build error" mails (which is not completely true, as Jenkins can be configured to send a mail only for the first broken build), and that metrics can't be done properly if the frequency of builds is too high.

So, what's your opinion on this ?

Valentin Rocher
  • 303
  • 1
  • 8
  • Sure your build time will be ~1min in 2 or 3 months, with 10 devs continuously adding more code including unit tests to your project? – Doc Brown Feb 23 '12 at 12:00
  • It would be interesting to explore the architects' reasoning for requesting the change; your points are good, but do they address the actual issue? –  Feb 24 '12 at 16:54

7 Answers7

32

Fail fast is a good principle - the sooner you know the build is broken, the sooner the offending commit can be identified and the build fixed.

Build on every commit is the right thing to do.

Building every 15 minutes can be pointless if the project has a high volume of commits within such a timeframe - tracking down the bad commit would take longer and may be difficult to determine (one might also be in a situation where multiple commits have different things that break the build). There is also the possiblity that in quiet times (night time?) you end up rebuilding though no changes have been made.

If the build breaks so often that it is a problem, the answer it to re-educate the team on the importance of not breaking the build and in techniques that ensure this does not happen (frequent fetches, checkin dance, compiling and running unit tests locally etc...).

Oded
  • 53,326
  • 19
  • 166
  • 181
  • 16
    +1. The answer to annoyingly frequent "build failed" messages is to not break the build annoyingly frequently. – suszterpatt Feb 23 '12 at 15:16
  • 3
    On quiet times - Jenkins's third option, "Poll SCM", is meant just for that. It will only update/run tests when changes are found in the repository. For example, we have one job set to run every 5 minutes if there are any changes (unit tests), and a second set to run every 3 hours if there are any changes (integration tests). Both are quiet at night/weekends because no one is committing anything. – Izkata Feb 23 '12 at 19:00
5

There is literally no point in doing a build every 15 minutes if nothing has changed. But equally there is no downside, afaik, jenkins will only e-mail on fail and then success and not everything in between (eg 10 fails).

We do it on every commit. However we do poll the repository every fifteen minutes and check for changes, maybe that is what yr colleagues are referring to.

You expect your 10 dev's to be committing more than once every fifteen minutes ? That sounds like rather a high estimation. 10 dev's means that after every 150 minutes the same person is committing again, so thats 2.5 hrs. So in your average day each dev commits 3 times. Personally I do one commit ~ 2 days ... sometimes more sometimes less.

NimChimpsky
  • 4,627
  • 4
  • 26
  • 39
  • 1
    Actually, commits go pretty fast around here, so this is entirely possible, but yes, I see what you mean. – Valentin Rocher Feb 23 '12 at 09:59
  • Perhaps the answer is to switch to a DVCS, whereby your devs can commit as often as they like, but only *push* completed features/fixes to the server - triggering a build. – MattDavey Feb 23 '12 at 10:04
  • @ValentinRocher seems high to me, see my edit. – NimChimpsky Feb 23 '12 at 10:04
  • @ValentinRocher dvcs are good we switched, and now when I go back to svn its rubbish. there is a quite a hard/long learning curve but it is worth it. – NimChimpsky Feb 23 '12 at 10:05
  • I'd love to switch to git or hg, but sadly the higher-ups don't want to... – Valentin Rocher Feb 23 '12 at 10:26
  • @ValentinRocher - The higher-ups need never know! If you are using [hgsubversion](http://mercurial.selenic.com/wiki/HgSubversion) or [git-svn](http://www.viget.com/extend/effectively-using-git-with-subversion/) to access a central `svn` repo then to them it would look just like you are using `svn` normally. *8') – Mark Booth Feb 23 '12 at 11:15
  • 3
    @NimChimpsky: you do one commit every 3 days? If that's true, I suggest you should seriously reconsider your commit strategy. Whenever you are going to reset something to a previous state, you will loose up to 3 days of work! How do you describe changes of 3 full days in a few words in your change log? That sound very absurd. Personally, I do a commit whenever I have added a working feature slice to my program, typically several times a day. – Doc Brown Feb 23 '12 at 11:54
  • 2
    @DocBrown its far from absurd. I might commit to different projects and different repositories three times in a minute. On the other hand I might not write any code at all for a whole week. I suggest you seriously consider your commenting strategy. – NimChimpsky Feb 23 '12 at 12:18
  • 1
    @NumChimpsky: I was assuming you were talking about a situation comparable to the one described by the OP. We are talking about 10 devs working on the same project. If the median time between commits per dev is 3 days, then something goes very, very wrong in that project. – Doc Brown Feb 23 '12 at 12:22
  • 2
    @DocBrown wtf ? You don't know what you are talking about ... I take it you don't work on multiple projects simultaneously. – NimChimpsky Feb 23 '12 at 12:24
  • @NimChimpsky: the topic in question was not "how many commits do you expect when your 10 devs work on 10 projects". The question was "how many commits do you have to expect when 10 devs work on the same project". And one dev every 150 minutes seems to be very reasonable, if the team does it right. – Doc Brown Feb 23 '12 at 12:33
  • 1
    @DocBrown I was responding to that fact you said there was something very wrong ont he project I was working on. And my work practices were absurd. The question doesn't state whether the 10 devs are working on one project. Most places I have been working on one project would be an unheard of luxury. And 150 mins, sometimes yes - a lot of the time no. You don't spend time designing, gaining requirements, initiating new projects etc etc ? You just incessantly code new functionality in one project every waking minute - that does sound absurd. – NimChimpsky Feb 23 '12 at 12:40
  • let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/2588/discussion-between-doc-brown-and-nimchimpsky) – Doc Brown Feb 23 '12 at 12:42
  • I disagree with the first sentence of your answer. In modern professional application development where both local and public repositories are regularly utilized (not to mention web service APIs), it is next to impossible to define the state of "nothing has changed" (please see my answer as an example). If you are really used to working on purely self contained projects, I ask that you qualify your assertion so that it doesn't come off as dogmatic. – smp7d Feb 23 '12 at 15:39
  • @smp7d The configuration described above is to check SVN every 15 minutes, and do a build if there are updates. This is the sanest configuration (although we use 5 minutes), since it builds often, doesn't waste time on weekends/nighttime when there are no changes, and doesn't build immediately, for when developers do several commits in a few minutes (say, forgot to add a file to svn, or splitting up lots of changes over a few commits if they were working on a few bugs at once). – Izkata Feb 23 '12 at 17:20
  • @Izkata Public libraries can still change on weekends and depending on how your repository manager is configured, it may break your build. Sounds like this isn't a concern for your particular environment though. – smp7d Feb 23 '12 at 17:39
  • 1
    @smp7d That's a concern no matter what the build schedule is. And yes, we have local copies of specific releases of external dependencies, so that that case doesn't happen. – Izkata Feb 23 '12 at 18:03
3

It's going to flood developers with mail more if it's every 15 minutes only. That's because it won't know for certain who broke the build and thus mail more people.

As for metrics, if it's really a problem—which I can't tell because I don't know what metrics they think there is a problem with—, you can always make another job for collecting the metrics.

Jan Hudec
  • 18,250
  • 1
  • 39
  • 62
2

Perhaps the requirement shold be "build at most once per 15 minutes". This could make sense for projects with very frequent commit activity (i.e. multiple commits within few minutes) and perhaps long build times. Of course it depends also on how the builds are used. For testers it might be somewhat confusing to get multiple builds within 15 minutes ...

But I agree that it makes no sens to build if nothing has changed.

MartinStettner
  • 151
  • 4
2

Some devs want to be allowed to do commits in a way where the files belonging to one functional change are not committed in one single, atomic procedure. They need two or three minutes to do a "logical commit", which consists of some "physical commits". This is typically the case when your devs directly commit to a central repository, not using a DVCS.

For this cases, it may be a good idea to let your CI server wait some time after the last commit before starting a new build. But 15 minutes seem to be a very high number, 5 minutes or less should be enough.

Or, better(!), try to guide your devs to work only in small portions, only one thing at a time, making it much easier to do only "functional complete" physical commits. Then a build after every commit will work seemlessly.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
0

Even if you have Jenkins set up to build on a source control commit to a project or any of its dependencies, this does not prevent a developer from deploying to the artifact repository without first committing to source control. If they deploy an uncoordinated API change or a bug in a dependency to the artifact repository, this can break your build without triggering the Jenkins job. I personally would want to know about this ASAP.

Ideally you would build for every commit and also on a schedule to check for situations like I just described. This would conceptually mean you build at least once every 15 minutes.

If you have your Jenkins jobs set up to run on dependency artifact deploys (and if you do...Bravo), you can prob axe the scheduled build if your unit tests are solid (meaning they don't test the dependencies) and your integration/functional tests are not part of the Jenkins job.

As far as the "flood with email" problem, I say getting flooded with email on a broken build is a good thing. Make sure you force developers to respond to the email with a description and you will see your broken builds go way down, trust me.

smp7d
  • 4,201
  • 1
  • 22
  • 40
0

You said you can't understand their reasoning, so you have to ask them. Don't just guess at what they want and try to implement that, and certainly don't just implement what they asked for without understanding why they asked for it.

That said, to answer the general question, it depends on the development philosophy you use. If every commit is expected to work, then every commit should be tested. If you use a DVCS with the philosophy that every push should work, then test every push.

In general, it's better to tell people something they don't know, then to repeat what they do know. For example, if they want to reduce the number of redundant emails they get, tweak the email settings, not the build frequency.

Paul Biggar
  • 1,207
  • 9
  • 11