8

I am working on creating a set of microservices in Docker (using .net core and AWS RDS as the backbone, but that is not relevant).

As part of the deployment, the old container and new container co-exist for a short period of time (under 30 seconds usually) to ensure there is no downtime.

However, this means that if my schema is incompatible with the previous version of the microservice (or the old schema is incompatible with the new version), one set of containers will not function correctly.

My thoughts are:

  1. Do backwards-compatible migrations only. Meaning, there is no delete column migrations. I am not impressed with this approach as it would result in some clutter accumulating (granted, with the size of the microservice it won't be too bad, but it will be an unsightly schema in a couple of years time)

  2. Obsolete-remove two step deployments. Meaning in v2 we remove code consuming the schema, but not the schema itself and in v3 we remove the schema. The issue here is that either we need to do two consecutive deployments in a row, which seems stupid, or make sure developers remember to checkin those removals into the next version, which is potentially human error prone and will result in clutter accumulating again

  3. Have two types of deployments - breaking and non-breaking; with breaking deployment doing the old school stop service, migrate, bring up new service. The issue here is that there is downtime for the end users.

What would you go with, and why? (or, if you already have something else, please do tell).

zaitsman
  • 384
  • 4
  • 13
  • 2
    With proper process put in place, there should be no reason why #2 isn't feasible. – Jack Sep 24 '17 at 12:57
  • There was a detailed blog post by the Nick Craver of Stackoverflow last year: https://nickcraver.com/blog/2016/05/03/stack-overflow-how-we-do-deployment-2016-edition/#database-migrations – Patrick Oct 24 '17 at 21:08

2 Answers2

4

A change should only change that which is necessary. Removing unused columns in a DB schema isn't a necessary change, it's a non-functional, maintainability kind of change. So there's no reason to lock in those changes to occur in the same iteration of any service (or, for that matter, be part of a service deployment at all).

While you certainly could work out a system that enforces a migration every deployment after a given 'breaking change' kind of deployment, this strikes me as a bit of over-optimization or YAGNI.

The way I'd handle it in one of my apps is to simply add the step to my change control process, wherein any microservice deployment has to be non-breaking as in your #1, but I'd keep track of which columns are removal candidates, and do a quarterly (or so) cleanup (with buy-in from the business users), possibly on a rolling schedule such that any proposed removed column has to go some amount of time before we actually remove it, b/c you know in many business settings the thing we're sure we'll never need again is actually a feature we see requested in six months or so.

Paul
  • 3,277
  • 1
  • 17
  • 16
1

  1. What is your "error budget"? i.e. How much downtime can you afford to have and still be within the agreed SLA? If your SLA's uptime is 99.95% and your service is serving 99.999%, then you are able to do so.

  2. If there are 30 seconds of downtime, how much of SLA does that reduce? If it reduces significantly, then there cannot be constant releases but if you're planning a monthly release, 30 seconds of downtime out of 30 days will not harm the agreed numbers.

  3. Is it really 30 seconds? If the application is erroneous, how long does it take to rollback? Is there a blue-green deployment strategy?

  4. From above, is zero downtime necessary? If so, then "obsolete-removal" step is necessary. The good thing is that you're separating concerns at each step so you can handle each fail case (i.e. rollback strategy)

The concept of SLA and error budget came from this Google's SRE book

In my experience, this is one way of doing it:

  1. Version your API so that you can deprecate/remove that code - then you can DELETE columns

  2. Each code set has a minimum migration version so that that code cannot be deployed unless the minimum migration version is deployed.

  3. Migration is always backward-compatible. I understand your concern about "obsolete-remove twp steps" but don't most software handle backward compatibility (i.e. deprecated, removal)?

Taku
  • 111
  • 1