9

My company* wants

  1. to move from using long-running feature branching (~up to a few weeks) to continuous integration with trunk-based development, and
  2. to break up our monolith into microservices.

We're unsure which to do first to best achieve both: 1) moving to trunk-based continuous delivery, albeit dealing with a several-hour time delay for our automated testing in the meantime; or 2) splitting up our monolith to shorten build and test suite times, then moving those to trunk-based continuous delivery.

Is there consensus on the appropriate order of operations here? If not, what are the pros and cons of implementing one first while trying to move towards the other?

*For context, our main application is a large .NET monolith released weekly, with a lead time for changes of about 2-4 weeks to accommodate long feature branches and end-of-sprint QA testing, a robust unit testing suite that takes ~2 hours to run fully, and a longer-running set of integration tests maintained by a separate QA team.

EDIT: We seem to be running into some terminology issues.

By continuous integration, I mean a practice whereby all developers working in a common code base are getting each other's changes all together at least once per day. The use of feature branches lasting longer than a day is incompatible with CI in this sense.

By continuous delivery, I mean CI combined with version control for all production artifacts, automated build & test pipelines, and prioritizing fixing the build over other work if it breaks, so that we aim to always keep our main branch in a deployable state.

By continuous deployment, I mean continuous delivery combined with automatically deploying releases passing all pipeline checks to production. This question does not ask about continuous deployment.

This is how the terms are used by Jez Humble and Dave Farley in their 2010 book Continuous Delivery first popularizing the topic, in Martin Fowler's bliki, in the DORA research group's State of DevOps reports, in Accelerate, and in Dave Farley's Continuous Delivery YouTube Channel.

Jacob Archambault
  • 209
  • 2
  • 7
  • 2
    A critical nuance to consider: Are your unit tests so entangled that they will become simpler under your microservice plan? This will tell you whether CI will become faster under microservices, which will tell you whether microservices-first is worthwhile. – Corbin Jun 04 '23 at 16:37
  • 2
    I would recommend to replace "CI/CD" by "CD/trunk based development" in the title's question, assumed that this is what the question is about. This would make clear if the currently [top-voted answer](https://softwareengineering.stackexchange.com/a/445881) is on track or missing the point. I suspect it is the latter, but you should really clarify this. – Doc Brown Jun 04 '23 at 21:01
  • 3
    CI/CD doesn't mean no feature branches. Just to be clear; are you suggesting short term (days) feature branches and frequent merges into trunk with code review (fine) or literally every commit goes into trunk and straight out to prod (which feels like it needlessly ties your hands) – Richard Tingle Jun 05 '23 at 11:36
  • 4
    Note that you may be able to simply parallelize the unit testing. How big is the development team? Microservices is often an antipattern when there's only one team. – pjc50 Jun 05 '23 at 11:48
  • @DocBrown is correct about the top-voted answer. Our company already makes extensive use of automated pipelines for running builds and unit testing. The question is asking about adapting our existing build and testing practices to the practice of CI in the sense since clarified in the edit, to achieve continuous delivery in the sense clarified in the edit. – Jacob Archambault Jun 05 '23 at 13:23
  • 1
    Have you seen Dave Farley's YouTube videos about microservices? – benxyzzy Jun 06 '23 at 05:24
  • 1
    @JanHudec CI is generally recognised as a requirement of CD, not something incompatible. There's are whole books about it, but in brief you can only merge changes that are thoroughly tested and do it within a day by breaking changes down into very small parts and doing them one at a time. Plus the trunk doesn't have to be literally always ready for release, it just has to be known when it is and isn't ready and repaired immediately any time it's not ready. – bdsl Aug 16 '23 at 08:25
  • @bdsl My experience is that most meaningful changes can't be broken down to steps that can be done in a day and the added code tested at the end of that day. So I always understood CI to mean you integrate the changes as they are done, and also integrate them into the other work in progress, but there is no limit on how long the work in progress might be outstanding. – Jan Hudec Aug 17 '23 at 12:37
  • @JanHudec This is Fowler & Humble's CI Certification requirements: https://martinfowler.com/bliki/images/ci-certification/sketch.png CI literally means you *Integrate* the changes *Continuously*. If it takes several days before the changes are ready to integrate that isn't continuous. – bdsl Aug 17 '23 at 13:37
  • @JanHudec I suspect when you say your experience is that these things can't be broken down you mean no-one you've been working with has found a way to break them down. And I would guess that's because they didn't try very hard to break them down that much, maybe either because they thought it was impossible to do or thought it was not very useful to do. – bdsl Aug 17 '23 at 13:39
  • @bdsl, ok, I clearly misunderstood the thing. Basically “CI” appeared back when everybody was still using centralized version control, and committing to the same trunk, but continued to be called that when everybody switch to feature branches. And most were happy to do so, because until then the trunk was broken quite often. And when it takes two hours to just build it, the test suite runs overnight and when it breaks, it's another two hours rebuilding it locally just to debug it, no, it will take more than a day to fix. That's typical values I've seen. CI like this isn't for such projects. – Jan Hudec Aug 17 '23 at 15:01
  • @JanHudec Right - to do continuous delivery you need to have a test suite that runs very much faster than that. – bdsl Aug 17 '23 at 17:45

5 Answers5

27

Consensus is you want CI/CD first and anyway, independent of your application's language, design or architecture. Whether you deliver trunk-based or use feature branching is also independent of CI/CD.

CI/CD is just automating your build steps, testing and packaging which is convenient to have for any software product under development. You typically want to set that up before you do any development, right after you created the first empty module in your application.

So it seems you are long overdue with that. But again, it is not really an A vs B question, they are independent.

Martin Maat
  • 18,218
  • 3
  • 30
  • 57
  • 6
    To some extent, I'd say "build tests, testing and packaging" is CI but not CD; for it to be CD, you also have to be releasing on every commit to `main` or similar. – Philip Kendall Jun 04 '23 at 07:35
  • 1
    @PhilipKendall I was unsure about this. "Main" sounds like a database centric type of system to me by the way. Would you consider automatically creating an installer not to be CD? Would you say it had to include some kind of deployment to a usable environment? – Martin Maat Jun 04 '23 at 08:20
  • 2
    I'd personally say CD requires deployment to a production environment, but I admit that's a view which is biased towards my recent jobs which have been SaaS-type things where you can do that. – Philip Kendall Jun 04 '23 at 08:46
  • 8
    The issue here is that some people use CD for Continous Delivery, some other mean Continous Deployment. – Doc Brown Jun 04 '23 at 09:09
  • 5
    I guess the OP already uses CI in the sense of this answer. They use CD as a synonym for "trunk-based development", which makes most parts of this answer quite pointless. – Doc Brown Jun 04 '23 at 10:15
  • @MartinMaat: "Release" is defined based on how your medium is released to consumers. If the installer is immutable and can be given to the end user, yeah creating the installer is a release and therefore CD. Same for e.g. a Nuget package, its release is defined by when a new version is made available, not when it gets adopted by consumers. When dealing with SaaS, releases are defined as the service being updated which means users see it (regardless of whether it's a new versioned route to the API or if the existing route was altered) – Flater Jun 05 '23 at 05:18
  • @MartinMaat [..] "Release" is a very literal term, it is the moment where the dev team _releases_ control over the product in its current state/version. A release is a version that's set in stone and cannot (or at least should not) be changed _within that same version number_. – Flater Jun 05 '23 at 05:20
  • I'd add that with CI/CD setup, you can join the microservices to the same CI/CD infrastructure for a single project - which allows for the testing to be coupled between versions for integration tests of passed unit tests or build/packaging steps for the individual microservices successfully pass within it. – Alexander The 1st Jun 05 '23 at 05:37
  • @DocBrown is correct. This answer is off track. Our company already makes extensive use of automated pipelines for running builds and unit testing. The question is asking about adapting our existing build and testing practices to the practice of CI in the sense since clarified in the edit, to achieve continuous delivery in the sense clarified in the edit. – Jacob Archambault Jun 05 '23 at 13:17
  • 1
    CD (in either sense) doesn't mean releasing on every commit to main - you still have tests (automatic and/or manual) and don't release that which fails tests. It means being ready to release almost everything you've done so far at any point, where the "almost" means generally no more than the most recent day's worth of code written at most does not go into the release, and ideally much less code than that gets left out. – bdsl Aug 13 '23 at 20:24
12

Trunk based development with continuous develivery and microservices are mostly orthogonal concepts. However, for a huge monolith, the delivery cycles which usually require a certain amount of testing will typically last longer, which will work againts the idea of CD.

In theory, having smaller microservices which can be developed and deployed independently can shorten those cycles. In practice, this will only work if you manage it really up to the point where a new release of microservice X has a low risk of breaking the whole system, and you can indeed reduce the necessary amount of integration testing for all other microservices here. For this, X must be strongly decoupled from other parts of the system, and have a certain tolerance against changes of those parts, and those other parts must have also be robust against changes of X.

What does that mean for your order of transition? I suggest you try the following:

  • Identify a part of your system you want to make a (micro-)service of its own. Create the necessary interfaces around it and rework it up to the point to be able to deploy this independently of the rest of the system. This includes a certain backwards and forwards compatibility of the versions of both parts now.

  • This new service gets a repo of its own and you use trunk based development for it (instead of feature branching).

  • Repeat this until the remaining core part of the system has become so small that you perceive it as a microservice of its own, and you can switch to trunk-based development for it as well.

Of course, in reality, things will probably not turn out to work so straightforward. But the gist of my recommendation here is, try to approach this iteratively and observe where it leads you. Breaking your architecture down into micro services can embrace trunk based development, but trunk based development does not necessarily require your product broken down into microservices in full.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • The likes of Dave Farley would say that having CI in place first will make the breaking down into microservices process quicker, because automated tests can catch regression etc. WRT microservices, the world is long into the post-hype disillusionment stage now, majority opinion says you'll tend to end up with even more unmanageable complexity. So you will be very grateful for the tests (+documentation of contracts, interfaces etc. to the extent that they achieve this) when you are mired in the transition – benxyzzy Jun 06 '23 at 05:21
  • 1
    @benxyzzy: the OP has already those CI techniques in place, the question's title is misleading (and the top voted answer missed that completeky). My answer is about trunk based development and microservices. – Doc Brown Jun 06 '23 at 06:26
3

Moving to trunk based development while still being on the monolith could also work. You would need to ensure that dev branches are short-lived and merged to main & integration tested often (ideally within a few days).

Note that splitting up the monolith into microservices could take time (i.e several months to over a year).

Arjun
  • 51
  • 2
  • 1
    Yes, that last point is important. Adopting continuous integration is something that can be done relatively quickly... not overnight, but it's simply a process change. Splitting up the monolith is a much harder and much more time-consuming task... depending on the size of the monolith, "several months to over a year" may be highly optimistic... – Simon Geard Jun 05 '23 at 21:54
0

Consider refining your CI/CD approach by building the first microservices.

In my experience working with "legacy systems", it can be very hard to retrofit a better build process and add on automated tests. That depends on how old your system is, and how complex, of course. You may find that your new approach to automated testing, or a new way of scripting the build, doesn't work for reasons that are hard to figure out (undocumented dependencies, for example).

On the other hand, your team is probably taking some training or reading some books about microservices and CI/CD, and working through little examples as part of that learning. I suggest that the next step would be to put your new skills/ideas into practice by building a very simple but usable microservice, with the automated tests, build process, and CI/CD process you want. Experiment and tweak until everyone is satisified, then make the old monolith consume that microservice instead of doing that functionality itself.

The benefit of this approach (doing both new things together, but in parallel with the existing monolith until it's ready) is that you can come away from it with a DevOps model that everyone on the team is happy with and can reproduce many more times.

On the contrary, if you attempt microservices first and then later attend to the build/test/deployment pipeline, you may find that your microservices design wasn't quite right and needs to be re-engineered. Or if you develop your build/test/deploy pipeline first and then try to split it up into microservices, you may find that you need to backtrack and adjust the first thing.

workerjoe
  • 261
  • 1
  • 6
0

In a way these two aims are contradictory, or at least going in different directions.

Trunk based development is about frequent integration of the work from different developers - integration meaning merging together and then running automated tests and builds on the combined part.

Moving to microservices is the opposite - it separates out work from different developers so that they never have to be integrated. They are deployed separately, and the first time a specific pair of versions meet may well be in production.

I think the most useful thing is to move towards Trunk Based Development and Continuous Delivery within the monolith.

If you still find you want or need to move towards microservices, start small by just splitting off a single microservice from the monolith. Once you've had time to evaulate that decide whether or not you want to create a second one.

bdsl
  • 2,365
  • 1
  • 14
  • 14