How to manage ongoing data migrations?

Question

I work in a SaaS project in which each user can have a representation of some common data at a given point in time. For the sake of example, let's consider that each user's data is metadata about them persisted separately on disk as a JSON file:

{
  "name" : "Bob"
}

As new requirements come along, this underlying representation may change (for example, in the above JSON, let's say that name attribute is changed to firstName and another attribute lastName is added). In order to support artifacts that were created at different points in time, we are forced to handle this in one of two ways:

1. Migrate all data from old representation to new
2. Follow different code paths/conditional logic depending on underlying representation:

.

if (oldRepresentation) {
  ... do this ... 
} else if (middleRepresentation) {
  ... do that ... 
} else {
  ... do something else ... 
}

Both seem problematic in that one is a big bang approach (and we all know that migrations are rarely smooth) and the other creates some amount of dissonance in the source with all the conditional logic. I'm curious how other projects handle migrations of this sort? Do most of you use one of the above approaches? How does your architecture account for this? Do you have a separate service to do migrations on demand?

Answer 1

The best way is to store a version number in each of the persisted files. Then you can read any of them, from any time and if the version is not the current one - send it to a migration routine that updates it to the current format and then continue processing it as if it was always the current format.

This allows you to perform the migration on an as-needed basis, rather than a lengthy conversion-of-all-records process. It also allows you to maintain records in offline or disconnected models. Your codebase is significantly simpler as it only had to manage the current format, though you will have to continually append rules to a migration routine as you add new formats.

To make this work, you MUST store a version. Trying to guess that format the file is after 2 or 3 revisions to the format will not be fun. If you current do not store the version number, that's fine - all files without one are simply v0 by default. Migration should add one.

Answer 2

Converting your data files from the "old" formats to your latest and greatest one is probably the best option. Save the reformatted data after converting it. That way, you're data gets upgraded gradually, over time, instead of a "Big Bang". Every time a file is accessed, it gets upgraded to the latest version (whichever that is today) and nobody's any the wiser. It's a bit like Oracle Database; it only checks for block corruption as it reads a block in from disk.

One possible problem is if you have to back out an application release. You then have to be able to "downgrade" your files to the previous version, which could be something of a challenge.