We are replacing a legacy application that runs on a single Oracle DB, and the decision has been made to replace it with a microservices-based application running on Azure SQL.
Do we do data migration from Oracle via the microservices, or do we do direct data migration from Oracle to Azure SQL DBs?
Advantage I see with doing it via the microservices, is that the business logic in the microservices takes care of data transformation and load to the target DBs. Whereas if you do direct migration to the target DBs, then you have to develop separate transformation and load processes.
Disadvantages with using the microservices for data migration:
- you need to talk to the microservices via their API - so you'd have to extract data from Oracle, then convert it to a format that's compatible with the API (eg JSON). I don't know if that would be fairly easy or extremely hard to do.
- you're dependent on microservices being built before you can run or test data migration. ie, you can't run or test data migration processes independently of the development of business logic.
Other questons: maybe migrating data via microservices is the only option if there are complications relating to how the relevant data can be persisted. Eg, the new system may say that you can't create a customer without an address, whereas in the old system, this was allowed, so you may have source address records that aren't related to a customer, which is valid in the old system but won't be allowed in the new system.
I'm very much leaning towards building separate transformation and load processes for the Oracle data, as it just seems a lot easier than trying to 'feed' all the data in the right format to a very large no. of microservices.
I should also say that I'm thinking that we'll migrate from Oracle to Azure SQL DBs using SQL Server Migration Assistant, wherever possible. But the desing of the new data model/data schema is very much 'up in the air' at this stage, so not sure to what extent we'll be able to use SSMA.
On the other hand, maybe it just doesn't make any sense NOT to do the data migration via microservices. Because it's only when the granularity of the microservices is finalised that you know exactly how many DBs you'll have and what data they'll store. So, it's only at this point that it makes sense to start building and testing data migration?
Edit: thanks v much for the reply, that recommends either of 2 approaches: historic data store (HDS) or ‘clone and cut’.
• So, with HDS, each microservice has 2 DBs: the HDS (which is a copy of the Oracle DB, migrated to 1 Azure SQL DB) and its own, separate Azure SQL DB. Each microservice needs to get the ‘historic’ data it needs from the HDS, and this could be done either on-demand, or all the data it needs could be extracted from the HDS and loaded to the ‘new’ Azure SQL DB in 1 ‘hit’. So, each microservice would need to ‘understand’ how to ‘talk’ to both the HDS and its ‘new’ Azure SQL DB – the structure of the data in each may be very different, so isn’t this hard to do? Also, presumably, the general logic would need to be: if a user wants to read/update/delete an existing record, then first look for it in the HDS and if it isn’t there, then look for it in the ‘new’ Azure SQL DB? (this would apply if the microservice queries the HDS on-demand.)
• ‘Clone and cut’ means that each microservice has 1 Azure SQL DB, which would initially be a copy of the Oracle source DB. As the microservices are developed, you keep and remove the DB entities, as appropriate, for each microservice. There is the issue of what happens where those developing different microservices decide that their microservices both need the same table. In these cases, there needs to be coordination to decide which microservice has direct access to the relevant table, and which microservice/s will need to access that table via another microservice. You seem to assume that the source Oracle DB can be ‘cloned’ as many Azure SQL DBs (1 for each microservice), and that no further data migration is needed once this happens. However, the legacy system will be (and needs to) be kept running until cut-over to the new system. I expect that development of the microservices will occur over many months – so the basic question is, how do you migrate all the data that is created/updated/deleted between the ‘cloning’ and cut-over?
