3

In a situation where there are multiple teams making changes to some Maven projects with dependencies between them (otherwise unrelated projects i.e. no super POM or anything like that), with each team working in a separate working branch on all of these projects and merging them back to trunk only after a story is complete and tested, how should one handle the dependencies between snapshot builds that each team produces?

Note that any of the teams could be making changes in any of the projects: they work on different pieces of functionality of the same system, and the projects are more like different layers.

Before the split into multiple teams, artifacts were deployed to a central Nexus repository. Of course, it would be possible to set up separate Nexus repositories for each team, and an additional one for trunk, but I'm hoping there would be a better way to deal with this.

herman
  • 405
  • 4
  • 11

2 Answers2

4

TL;DR: Use release dependencies instead of snapshot ones.


Case to consider when choosing between snapshot and release dependencies is when changes in some component (A) break a dependent component (B).

When this happens, you need to fix it, but there are two options, one of which might turn out preferable depending on your project specifics.

One option is when you prefer to fix the issue immediately, even though it may come at cost of distracting developer working on component A. Benefit of this approach is its conceptual simplicity - it relieves developers from the need to memorize, maintain and update dependencies between components, because instead their efforts are focused on simply keeping all the most recent versions of components in sync.

In this case, release dependencies don't offer anything appealing compared to snapshot ones; rather the opposite - these would introduce an extra step / effort to release updated component A to be able to test whether dependent component doesn't break anymore.

  • By the way, Maven provides support for a simplest form of snapshot dependency preference: multi-module projects, where handling snapshot dependencies between modules is made as easy as it gets (though this comes at the expense of simultaneous releases, which is sometimes undesirable - eg when components / modules tend to change at very different pace).

An opposite option is when you would prefer to postpone integration fixes in the component A, for example when you believe that in your project it is more important for developer to strongly focus on the changes made in it and switch their efforts to integration later. This could be the case when dependencies between components are "loose", small, carefully designed and slowly changing interfaces.

  • For example, if you already have carefully crafted, thoroughly tested, clean and convenient interface between components A and B, you may prefer to postpone messing in B while you're polishing changes roughly drafted in A, expecting that it will take less effort to integrate after interface at A side gets clean back again.

In cases like this release dependencies allow you to totally isolate dependent component. You don't need to worry about component B at all, until you are done with A and modify dependency in B to refer desired release. Snapshot dependencies can not offer anything like this: if changes made in A break dependent component, you'll have to put an effort into bringing things back to shape.


Now, let's see how above applies in your case,

  • unrelated projects
  • each team working in a separate working branch
  • split into multiple teams

All of the above whispers, says, cries that team is moving in a direction allowing more independent, more isolated work on the components of the system. So that John working on a project A, Paul working on B, George working on C and Ringo working on D have a leeway in planning and implementing changes within these components without permanently having to look at what happens in other components.

Now, let's look from above perspective at "dependencies between snapshot builds that each team produces". The way how Maven works, this would mean a huge step back into what team is supposed to move away from. the very moment John makes an intermittent change in A and builds a snapshot, Maven immediately picks that snapshot and "broadcasts" it into projects of Paul, George and Ringo!

Now, three guys working in other projects have to adapt to changes made by John and, if these changes cause some serious trouble in other projects, most likely John will have to break his flow and urgently adapt their project to... oh by the way to what is he going to adapt? Paul, George and Ringo, they all work like John, their projects are permanently changing, are they supposed to explain to John what exactly he needs to take into account for today, and what he possibly will have to account for tomorrow?

You see, if team is supposed to move to more independent development of different components / projects, using snapshot dependencies essentially totally breaks this movement and returns you back to the days of monolithic approach.

If you want to keep moving to stronger separated components / projects, you better have to define "communication points" at which these projects would "declare" their readiness / obligation to integrate with others. In Maven, releases naturally serve this purpose.

Maven releases provide stable, fixed "checkpoints" allowing project developer to safely communicate with outer world.

  • Release 2.15.23 is most recent one ready to be integrated; we no longer support / accept bug reports against releases earlier than 2.14.999; releases 2.15.24 and later ones currently suffer from a critical issue and are by no means intended for integration until further notice.

Worth stressing that release dependencies denote stable "checkpoints" between components, not between people. Whether components are maintained by separate developers or not isn't really relevant.

Even if you have, say, John working in both components A and B or maybe John and Paul together working on these two components, it still makes sense to have release dependencies, to allow programmer focus on work in currently chosen component, without being permanently distracted with worries about how this could impact another.

Given above, you also need to decide between teams how exactly are you going to schedule releases to make integration as smooth as possible. Here your options are really broad. A natural "schedule" that comes to mind is that each team decides when they're ready to offer a "checkpoint" of their work for other teams to integrate.

However natural ad-hoc releases sound "from within" a single team, if you look at it from the perspective of other teams, you may find that they would prefer to have a more regular way to get to "integration checkpoints" - from this perspective, you may wish to consider some predictable ways to "promote" component updates for other teams attention - monthly, or weekly, or maybe daily releases.


Now that your system became more complicated, specifics and purpose of snapshot dependencies are worth taking a closer look at. You can find a pretty decent explanation of this in Maven: The Complete Reference at Sonatype (bold font in below quote is mine):

...when you deploy a snapshot, you are not making a release of a software component; you are releasing a snapshot of a component at a specific time.

Why would you use this? SNAPSHOT versions are used for projects under active development. If your project depends on a software component that is under active development, you can depend on a SNAPSHOT release, and Maven will periodically attempt to download the latest snapshot from a repository when you run a build. Similarly, if the next release of your system is going to have a version "1.4", your project would have a version "1.4-SNAPSHOT" until it was formally released...

If a project depends on a SNAPSHOT, it is not stable as the dependencies may change over time... SNAPSHOT versions are for development only...

As you can see, above stresses temporary, intermittent, unstable nature of snapshot dependencies and limits their applicability to times of "active development". In your case, you can think of temporarily switching to snapshot dependency when you need to do series of closely correlated, synchronized, tightly coupled changes in different components. After this "active" phase is over, you can do "checkpoint" releases of involved components and switch back to release dependencies.

Note by the way that if you discover that switching to snapshot dependency happens too often, this might indicate a need to rethink your design. You may think of ways to better isolate components, or of opposite, that is, of "merging" previously separate components back into monolithic, synchronously developed and released module.

gnat
  • 21,442
  • 29
  • 112
  • 288
  • Thanks for an elaborate answer, but my question may not have been clear enough: each team focuses on pieces of functionality that potentially involve changes in all Maven projects. So it's not "John working on a project A, Paul working on B, George working on C and Ringo working on D". Where e.g. project A is a web GUI, project B is a client for some web services that is used by A, project C are some custom widgets also used by A, etc. – herman Feb 17 '14 at 19:32
  • I realize this was not apparent from the question - I'll update it. – herman Feb 17 '14 at 19:34
  • @herman it doesn't really matter here whether different projects are maintained by different or same people - I updated the answer to expand on that – gnat Feb 18 '14 at 12:37
  • I guess what I mean to say is that we're not moving towards more independent development of projects/components, but to more independent development of functionality/features, which run across projects (which can to some extent be considered as layers). So *before* a stable checkpoint can be made, *before* anything is merged to trunk, in other words during development of different functionalities by different teams, each team will normally be using snapshot builds (for their own internal use, not to share with other teams). – herman Feb 18 '14 at 13:31
  • @herman you don't need a particularly stable checkpoint, you don't need to wait until trunk merge - just _some_ checkpoints, some _intermediate_ releases in branches would suffice. Otherwise, your split doesn't make sense - simply because snapshot dependencies will keep projects coupled. If you stick with snapshot dependencies _and_ feel comfortable doing this, reconsider the very idea of splitting – gnat Feb 18 '14 at 13:35
  • In any case, I don't want to create a checkpoint (release version) for *every* build that gets deployed to the CI server (e.g. nightly or even on SVN commit). Splitting into multiple projects makes sense for different reasons, like e.g. reusing one project in multiple products or preventing circular dependencies between layers. @sora's solution with a simple reactor-build POM seems to make sense, but I'll need to do some more investigation. – herman Feb 18 '14 at 13:49
  • @herman why not? Maven releases are cheap: as long as you don't (and you really shouldn't) force other projects to immediately switch to latest release, they merely provide a convenient [coordinate](https://maven.apache.org/pom.html#Maven_Coordinates) system to communicate about state and progress of the related projects. If done right, these are as lightweight and flexible as snapshots, only without limitations mentioned in Sonatype reference I quoted – gnat Feb 18 '14 at 14:05
  • I think this would get confusing: team A builds version 1.2.3 from its feature/working branch, with feature X half complete. Then team B builds version 1.2.4 with feature Y half complete but without any changes for feature X. Then team A builds 1.2.5 with completed feature X and none of feature Y. Then team B builds 1.2.6 with completed Y and none of X. Then 1.2.7 is built from trunk with both features in it. One would expect increased versions to contain increasingly more features. – herman Feb 18 '14 at 14:24
  • @herman I see, this is the price you have to pay indeed, and it takes time to get used to. Anyway, you can try solution suggested in another answer and see if it feels simple enough (I also tried various approaches before picking the one based on "dumb" release dependencies), it's little harm of trying as long as you feel free to change when you find it's not good enough – gnat Feb 18 '14 at 14:31
4

Source code repositories have branches, binary ones don't. So any part of your development plan that relies on branching can't rely on binary repositores for things that are in the scope of branches.

(One exception to the above is ClearCase, which does do both, and is best not spoken of.)

If the teams are to be independant, you really do need to manage the dependencies in terms of released versions. This will be painful; what you are doing is hard. Probably have to get used to planning and discussing changes in advance, abandoning ad-hoc refactoring.

On the other hand, perhaps the teams are really only nominal, e.g. everyone is all working together to a single overall product release/delivery schedule. Then just keep to a single overall version number that gets bumped any time a feature branch is merged back to master. All local dependencies then simply use {project.version}.

Note that this is still not a snapshot, as you never update it without changing the number.

And then within a team share code not binaries. Some extra recompilation, but if that takes more than 15 seconds, get faster PCs. And anyone not using a multi-module IDE like eclipse will have to write a trivial reactor-build POM to cover the set of modules they are currently working on.

See:

http://books.sonatype.com/m2eclipse-book/reference/eclipse.html

http://axelfontaine.com/blog/final-nail.html

http://ahoehma.wordpress.com/2010/12/22/intermodule-dependencies-now-better-working-with-maven-3/

For how to set up an IDE, CI build server and reactor build for this kind of situation.

Or, you could try reimplementing ClearCase with some scripting on top of git+Nexus (don't do this).

soru
  • 3,625
  • 23
  • 15
  • I'm not sure you're understanding the question: if a member of team Y performs changes in projects A, B and C (which have dependencies between them) in team Y's own working branch, then a build needs to be made to deploy to team Y's own CI server (e.g. to do some testing) *before* changes are merged back to trunk (since things may not be working correctly yet). Surely, you don't want to bump the version on *every* build (and even then, teams would have to make sure not to use the same version number). – herman Feb 18 '14 at 09:10
  • The latter counts as sharing within the team, which means it should be done in source form, not via a non-branch-aware binary tool. Which is why you need a multi-module reactor build to get the dependencies to resolve without needing a repository. Or use an IDE which can resolve dependencies across opened projects (Eclipse with M2E can do this, I'd guess the other comparable tools can too). – soru Feb 18 '14 at 11:02
  • We're using Eclipse and M2E so indeed if everyone on each team has all projects local then for local builds there is no problem. I'll look into this reactor-build for deployments to CI server or QA (each team has it's own QA people who need to verify the story before merging to trunk). But note that currently the Maven projects have no relation to each other (other than one being a dependency in another). – herman Feb 18 '14 at 13:31
  • If I understand correctly, this solution with reactor-build POM (from outside Eclipse, e.g. for continuous builds from Jenkins) requires all projects to be in the same SVN repository so that there can be a POM with `` at the top of the directory structure, referencing the modules in each subdirectory. Or can this also be applied if each project has it's own repository location (as is the case here)? – herman Feb 18 '14 at 17:37
  • It works as long as the directory structure is fixed, which you get for free with single repository, but can arrange by scripting, or just documentation, if you really must use multiple ones for some reason. Modules of a rector build are located by path, not name. So they can be something like ../repoA/module1, ../repoA/module2, ../repoB/module3, .... – soru Feb 18 '14 at 18:01
  • 1
    I'm going to accept this answer as it seems to be the standard way of dealing with this kind of setup. I was first afraid that projects could only be specified in a single reactor-build POM, but then learnt about Maven project aggregation vs project inheritance. – herman Feb 20 '14 at 14:47