12

I'm trying to flesh out an eCommerce system using microservices (.NET Core and Kubernetes), event sourcing (Kafka), and CQRS. The particular use case I've been thinking about is as follows.

There is an inventory microservice designed to use CQRS. Updates are fed into Kafka as events, which the inventory microservice consumes to update its materialized view, and reads are executed directly against the materialized view. The issue I'm trying to work through is how to handle orders. In my current design, orders are created in an order microservice which then emits events which the inventory microservice would consume and deduct the inventory that was part of the order. However, there's a race condition here. It's possible for another buyer to buy the same products before the inventory from the previous order can be deducted.

How does one handle this type of transactional operation that spans microservices? I've read that one should verify stock against the stream instead of the materialized view (as the stream is the actual source of truth), but I'm a little fuzzy on how that would be practical (as the stream might be huge). If I'm going to do that then why even have the materialized view if I can't trust it?

Raymond Saltrelli
  • 1,344
  • 2
  • 12
  • 16

4 Answers4

12

In my experience, most questions about transactionability and microservices are caused by the following two reasons:

  1. The transactional data is placed in different microservices: This is wrong by definition. Data that should be modified transactionaly belongs to the same service. Grouping the right data in the right service is very tricky and you need to put a lot of thought into it. If you just create microservices based on entities (Orders, Users, Inventory, Products, etc), then you will find the same problem over and over, plus you'll end up with N services depending on N other services to perform their purpose. Therefore: put all data that changes transactionally together.

  2. The operation is not transactional in the first place: You are trying to solve with technology something that should be solved by business rules. This is your scenario in my opinion. Orders and Inventory are not transactional. You obviously don't want people buying products out of stock, but you have to accept that there will be always a slight chance that this happens (what if your inventory numbers are correct, but someone at the warehouse drops a box and breaks the last item?). Businesses have been dealing with these sort of problems far longer than transactions existed. Therefore, talk to business and ask what you should do when an order comes in and you are out of stock. If proper stock management is in place, you'll probably know already when new items will be available, so you can email the customer with the new delivery date and offer a refund if she doesn't accept (as an example).

As a side note, your sentence "an order microservice which then emits events to deduct inventory from the inventory microservice" sounds wrong. It might be just the sentence wording, but just in case, I want to point out that the wording suggests that what you do is emitting a Command, not an Event. A command is imperative, you tell someone to do something (DeductInventoryCommand). An Event is something that has already happened (OrderPlacedEvent).

Ideally, you don't want one microservice telling another one what to do, as this effectively breaks the autonomy of the service that receives the command. Instead, develop your services to be able to perform their business capability based on the information they "hear" from other microservices, but autonomously. A Logistics service can listen to the placed orders (and cancelled orders) and take care of all the logistics process without being told what to do. Billing should be able to do its job without being told what to do either, and so on.

Francesc Castells
  • 906
  • 5
  • 13
  • Thank you for your answer. RE: the wording around order creation, yes that could be more clear. My intention is that the order service simply emits an event indicating that the order was created and it would be the responsibility of the inventory microservice to consume that event and update inventory accordingly. I think my issue is a mix of 1 and 2. Stock that is physically missing is always a problem and falls clearly into case 2. But I also don't want to introduce this issue unnecessarily by allowing buyers to buy things that are known to be out of stock. – Raymond Saltrelli Sep 05 '18 at 14:13
  • 2
    OK about the events. I agree that you want to limit scenario number 2 as much as possible, but given that you cannot avoid it completely, you have to deal with it. I could think of ways to minimise the "race condition" (which as another answer points out it's not a real race condition, as business processes don't really work at the seconds level), but I would need to know more about the implementation, sorry. – Francesc Castells Sep 05 '18 at 14:39
2

Review Race Conditions Don't Exist, by Udi Dahan.

It's possible from another buyer to buy the same products before the inventory from the previous order can be deducted.

Yup - so what does the business do when that happens? There's probably a protocol, which might include ordering more stock, or offering to a customer a refund and an apology or... there are many possibilities.

Pat Helland's Memories, Guesses and Apologies is another important read

Business realities force apologies. To cope with these difficult realities, we need code and, frequently, we need human beings to apologize. It is essential that businesses have both code and people to manage these apologies.

The computer, after all, is not looking at the real world, but its own internal simulation. "The best a computer can do is make a guess."

VoiceOfUnreason
  • 32,131
  • 2
  • 42
  • 79
  • `There's probably a protocol, which might include ordering more stock, or offering to a customer a refund and an apology or ...` these operations (protocols) are what we often refer to as [compensating transactions](https://en.wikipedia.org/wiki/Compensating_transaction). Just a note, these transactions are not necessarily automatic (programmable), they might involve a sort of human interaction as well. – Laiv Sep 05 '18 at 14:31
1

It's possible for another buyer to buy the same products before the inventory from the previous order can be deducted.

This problem exists regardless of the technology you use. It's inherent to temporal nature of the problem. There are two solutions: optimistic or pessimistic.

pessimistic:

You create a transaction that allows a company to reserve stock. This must be done before confirming the order. The problem with this is that you could have customers reserving things and then failing to order or release their reservation. In the meantime, you could lose out on sales because customers are unable to reserve those items.

optimistic:

Customers issue an order and you check and update the stock at the time of the order. If someone issues a request for an item that cannot be fulfilled, you return a response telling them that. You will probably want to allow for the request to specify whether the order should be fulfilled in full only or fulfilled partially.

Either way, you cannot separate the request to order and the management of the inventory. In the systems that have been around for decades, these order requests and the responses are typically asynchronous. This allows for requests to be ordered and rationalized.

Community
  • 1
JimmyJames
  • 24,682
  • 2
  • 50
  • 92
0

Generally in eCommerce you don't check the stock level. It just slows things down.

However, you have some bigger problems.

You can't really handle the stock level reservation with a traditional database transaction. Because it's a human clicking through the order process, the transaction would simply be open too long.

This leaves us implementing a stock reservation system instead. There's no real problem with this. You can have a stock microservice in front of your stock database, or 'materialized view' in kafka speak, which can reserve stock with an RPC style message and know when items are out of stock.

A user attempting to buy an item would first attempt to reserve the stock via the StockMicroservice and be told that there was none available at that time.

I think your main problem here is trying to squeeze business events into kafka streaming database events.

A business event doesn't always result in a state change.

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • This is technically possible, but I would strongly suggest to check with business if they agree on this procedure. Reserving stock before completing the purchase is very risky for high value, low stock products. Imagine you have 5 MacBook Pros in stock and 10 users starting the purchase. 5 users book the stock and 5 get an out of stock message and leave. Then the initial 5 don't complete the purchase. You have lost several sales. – Francesc Castells Sep 05 '18 at 14:35
  • @FrancescCastells totally agree. see first sentance – Ewan Sep 05 '18 at 14:36
  • I don't think I would implement this as "reserve the stock the second it enters my cart." I think it would be more appropriate to reserve it first thing when the user clicks the Create Order button. If the stock can't be reserved then don't create the order. – Raymond Saltrelli Sep 05 '18 at 14:38
  • @RaySaltrelli hard to judge, the OP might be selling next years model lamborghinis or fizzy drink cans – Ewan Sep 05 '18 at 14:41
  • 1
    True Ewan. @RaySaltrelli, eitherway, check with business. This is not a technical decision and there are more possibilities. For example, you could allow to place all orders and only book stock after payment (if payment is done after placing an order). Just to show that there is no right/best solution. – Francesc Castells Sep 05 '18 at 14:43
  • @FrancescCastells Hmmmm. Good point about payment. We have an in-store pickup and pay later use case that would affect this flow as well. – Raymond Saltrelli Sep 05 '18 at 14:45