0

Let's say there is a user table and has field called coins to represent the coins available for that user in the system. At this stage a certain user has 50 coins in his account. Same user do the following requests almost instantly, but in the following order.

  1. Buy something for 20 coins
  2. Insert a promo code that doubles the coins

If the 1st request fails, in a properly written monolithic application, the user will end up with 100 coins.

But in a microservices based application where users are maintained in the separate microservice with a separate database and orders are maintained in a separate microservice with a separate database which utilize SAGA pattern to handle transactions, the following scenario can happen.

  1. At some stage of the SAGA, 20 coins will be removed from the account. Now user has 30 coins.
  2. Promo code will be applied. Now user has 60 coins.
  3. At later stage of the SAGA, buying the item fails. So it initiate a rollback.
  4. User will be given 20 coins at a stage of the rollback.

At the end, user will own only 80 coins, which is wrong.

Is there any way to fix this? Any suggestion?

Ramesh-X
  • 109
  • 4
  • 2
    This can be solved in many ways. First way would be to add a status that indicates that there is a ongoing transaction and in the meantime it is not allowed do alter the coins. Another approach is to add roll back logic. For example when the purchase fail the twenty coins are added back and the promo code is rerunning again. Another approach is to rethinking your microservices if they are really split well enough. And off course you can simple don´t use SAGA pattern and instead make a synchronous transaction if needed – Darem Feb 11 '22 at 11:19
  • 1
    Another way to tackle this is to keep the value of the users coin balance in the order service and use events to update it when it changes. (Duplicated data).. That way the order service can manipulate its local coin value and fire an event that the user service uses to update the balance. Saves having to do the SAGA. But, traditionally, SAGA is the way to go with this – Robert Perry Feb 14 '22 at 23:45
  • What exactly is it that you're trying to fix, and on what terms? For example, is it acceptable to use a monolith to fix the error of having started out with microservices? Or are you trying to maintain the separated microservices but somehow gain transactional consistency between them? Or can it be fixed by eliminating the rule about "doubling the current balance", and instead using a different rule? – Steve Mar 12 '22 at 05:24
  • @Steve What I mentioned in my question is more simplified example the actual problem. Really what is need is a generalized answer. So going to monolith and changing the rule will not work in my case. The best solution I got so far is to lock every entity related every microservice in a workflow, until it completes successfully. – Ramesh-X Mar 12 '22 at 06:27
  • 2
    @Ramesh-X, I have to say, the locking will make the various microservices functionally into a monolith - an organised whole that has to be reasoned about in a total way - but will instead homebrew (and likely do badly) the things that are normally done by world experts as part of database software that incorporates transactional consistency. Effectively, microservices (in the sense implying locality of reasoning or of concerns within a larger system of parts) are incompatible with transactional consistency (a property of a whole system which requires a total organisation of the whole). – Steve Mar 12 '22 at 08:58

1 Answers1

1

You have to operations/events:

  • (A)20 coins were removed from an account and then added back
  • (B) balance of an account was doubled

And in your monolith application you guaranteed to execute them in some order - either A,B or B,A.

In either order of events, the outcome is 100. The property to help you to do this is serializability - transactions gets executed in some order - but one by one, not at the same time.

Clearly you run in a problem if you do truly execute two transactions at the same time. This problem may even arise is a monolith application if an improper isolation level is enabled b/n transactions.

So what do we need to "fix" in a distributed system to get same behaviour? There are several options. The one I would use in your case is locking. A system which operates on a user's balance has to get a lock to execute. In that case the order will be either A,B or B,A but the outcome will be the same.

If we take your example to a bit higher level, we can say that there is causal dependency between A and B, since an outcome of A is needed for B to work correctly. Before doubling coins (B) you have to know if purchase (A) succeeded. This means that your operations are not concurrent, where concurrency is not a time thing, but a part of concept "happens before".

So to resolve your specific problem in practice, you could use zookeeper to manage distributed lock on account balance. Zookeeper is actually pretty known for being suitable for these types of tasks.

We could take the solution to the next level and introduce a consensus based system, but that will be much more complicated to understand and implement.

AndrewR
  • 186
  • 3