Micro-services and data replication

Question

I am building a new application and was reading about micro-services architecture. The architecture itself makes lot of sense from a development, deployment and life cycle management point of view. However, one issue that came up was with regards to how to handle master data.

For example, I have 2 apps: say Sales app and a Ticketing app. Assume that both of these apps are built as independent micro-services. However both of these apps, when deployed (assuming that they are deployed separately say Sales uses MongoDB and Ticketing uses MariaDB), would need to have access to the same master data instances e.g. Accounts, Products. This would mean that there would be an owner app for a given master data entity (e.g. for Accounts it might be the Sales app) and an interested party (e.g. Ticketing app would need to have information about Accounts).

There are multiple ways in which this can be achieved:

• Data replication from master to interested party
• Synchronous read from interested party to master (sync dependency is not recommended by the micro-services architecture paradigm)
• Own centralized repository

Also even within Accounts, there can be a core part which is common for both Sales and Ticketing (e.g. account name, address etc.). However some aspects of the Account might ONLY be relevant for Sales and others ONLY relevant for Ticketing.

Any thoughts / best-practices / opinions regarding any of the above mentioned options ?

Answer 1

I was part of a team that successfully built a microservices architecture using a service bus.

Initially, we believed that microservices and an event driven architecture would enable us to fix the underlying shared data ball-of-mud database.

What we learned was that microservices and an event driven architecture required us to get rid of the underlying shared data ball-of-mud database.

I believe that shared data is incredibly hard to do well with microservices - for me it is prohibitively hard. I recommend not allowing services to see each others data. If you can't query it, you can't accidentally introduce a dependency.

If you do share data, certainly only one service can ever OWN a record; it is the only service that writes to the record, and any other users of the same data should have read only access.

Unfortunately, even this small managed amount of sharing introduces significant coupling between your services. What if one service doesn't want the data in that shape? Perhaps it wants an aggregation. Do you get your "owner/write" service to write aggregate data for the benefit of another service? I'd advise not.

What if the owner wants to persist the data in a different shape? Then every reader service needs to be updated. That's a maintenance nightmare.

The best solution we had was a significant duplication and denormalisation of our data. Micro services maintained their own copies of data they cared about.

Messages were often published with enough accompanying data to process them.

For example, you might imagine that a postal despatch service might need to keep track of changes to a customer address, in case it needs to post something. But if your "item ready for despatch" message includes the destination address as part of the message data, then the despatch service no longer needs to track changing addresses related to customers at all; just point-in-time addresses related to items as they are despatched.

I can't suggest how to architect solutions with synchronised data. Our data solutions were built around the idea of "eventual consistency".

So when a customer updates their address, the Address service processes the initial command from the UI. Once its data is correct, it publishes an event to notify all other interested services "Customer Address Updated" - with the complete address attached as data. Those services will update their own data stores with the parts of the data that they care about.

The idea is that any time a service has to take an important action, it should have a copy of the information up-to-date-enough to act correctly, either tracked independently, or as part of the message it is responding to.

Answer 2

Shared data storage would go against the microservice architecture. The point is if there are Accounts, there should be a service to handle them and there should be no other way of interacting with these Account than thorough the service. Your microservices would not be independent if they shared common storage and any change to the storage mechanism, validation or other constraints would need to be implemented in both services. In practice this never happens and soon leads to both applications being prevented from any serious changes.

So, use a service as the only way of accessing your data, e.g. Accounts. It may happen that implementing a feature in some other service requires a change to the Accounts service, and that's OK. Just remember that logic specific to any service should be in that service and as little specific stuff as possible should go into the Accounts microservice.

Answer 3

would need to have access to the same master data instances e.g. Accounts, Products

Not the same. Each micro service should do their own mapping for Accounts, Products, etc. We are assuming that each micro service will have a different way to work with the entities.

In Domain Driven Design this is common, where each bounded context (in the same app!) can map the same entity in a different way.

If you need more information, I recommend the book Building Microservices: Designing Fine-Grained Systems

Answer 4

Have you considered the concept of skeleton records based on a master set?

For example one microservice handles accounts and the other handles products. A third may keep records from both for its domain specific purpose.