5

I am currently building a microservice-based system to learn CQRS/ES, Docker, AMQP and all the other goodies that goes with it.

I have never asked a question online before as I am usually pretty good at finding answers to my queries by reading what that others have asked.

This time, I am stumped by what seems to be very simple.

Business documents such as invoices, purchase order, credit memos, etc. usually have an identifying number. (i.e. Invoice No.: 5707)

My question is how would I produce a sequential number for each of these types of documents in an event-sourced system?

I'm concerned about race conditions where an invoice number may inadvertently be duplicated or skipped. What is the best practice for this?

Thank you for your time.

EDIT: Here is what I tried:

I have an aggregate root (AR) called Invoice that has an associated saga called newInvoiceInitialization. The Invoice receives a command CreateNewInvoice. The Invoice AR processes this command and emits an event called NewInvoiceWasCreated.

The newInvoiceInitialization saga handles the NewInvoiceWasCreated event and starts the saga to initialize the invoice AR. The newInvoiceInitialization saga in-turn sends a CheckoutNextInvoiceNumber command to another AR called SequenceNumberGenerator with value objects: CorrelationId and ResourceTypeName.

The SequenceNumberGenerator AR works with a saga called SequenceNumberCheckoutSaga. These two will "check-out" the next InvoiceNumber in the sequence and supply that to the Invoice AR via a SequenceNumberWasCheckoutOut event.

The newInvoiceInitialization saga receives the SequenceNumberWasCheckoutOut event and sends the InvoiceNumber to the Invoice AR with a AssignInvoiceNumberToInvoice command.

When the Invoice AR completes the newInvoiceInitialization saga successfully, it emits the event InvoiceNumberWasAssignedToInvoice. This event triggers the SequenceNumberCheckoutSaga to offers the command FinalizeNumberCheckout to the SequenceNumberGenerator AR which ends the SequenceNumberCheckoutSaga and finalizes the usage of that number.

If the Invoice fails to accept the InvoiceNumber or has other problems it sends the InvoiceNumberAssignmentFailed event which causes the SequenceNumberCheckoutSaga to command the SequenceNumberGenerator AR with a ResetNumberCheckout command to roll-back the checkout-out sequence number and ends the SequenceNumberCheckoutSaga.

This all just seems overly complicated to produce invoices with numbers that aren't missed or duplicated. I am probably missing something.

Keith03
  • 49
  • 3
  • You need to have one and thread safe source of truth for the number. You can use a database with incrementing Id and use id as a number. You can use single queue which will create orders or invoices. – Fabio Jun 30 '19 at 21:23
  • Thank you @Fabio for your comment. I was planning on using an instance of Redis to store the actual sequence numbers. My question (I guess) is more about how to incorporate such a service into my domain model. The closest thing I have found on building the number sequence generator is here: https://www.akaes.com/blog/how-to-implement-robust-auto-numbering-using-transactions-in-microsoft-dynamics-crm/ – Keith03 Jun 30 '19 at 21:44
  • What did you try so far? Right now your concerns lead us to a fairly broad question: *How could I do X?* The answer depends on many things. Overall in such an env as the one you are working on (highly decoupled, distributed, DDD, etc). What would you say is the main concern? Making this to work with concurrency? What would you say is more important for you: Consistency? Performance? Availability? Let us know the "ideal" picture you want to get with this exercise. – Laiv Jul 02 '19 at 11:14
  • @Laiv I have updated my question describing what I am trying. I have not finished it yet but this is the direction I am going. It really seems like there should be an easier way. I may, as others have suggested, just use some other kind of invoice numbering system. This is not a school project or a work project. It is a pet project that I am making as a personal enrichment to learn about cqrs and event-sourcing within a microservice architecture. Thank you very much for you input and your time. – Keith03 Jul 02 '19 at 19:31
  • What does "missing number" means in this context? Are you worried about no sequentiality in the invoice numbers? What's the problem with the missing number in this case? – Laiv Jul 03 '19 at 09:31
  • Yes, what I am attempting is to make sure there is no gaps in the sequence of numbered invoices (I know there are better ways to number invoices). If I retrieve the next invoice number in the sequence and something goes wrong (i.e. malformed data, communication problem or exception) That number could be "lost" causing a gap in the sequence. I may have figured it out as described in my edit. I have much more work to do to actually implement all of this. Should I now delete my question? I am new to this forum system. Any advice is very welcome. Thanks for your input on this. – Keith03 Jul 03 '19 at 17:05
  • This page has some good information about invoice numbering. I am attempting to use the sequential number approach with some prefix system possible. https://online-accounting-software.bestreviews.net/how-to-properly-number-invoices/ – Keith03 Jul 03 '19 at 18:19
  • Honestly, we could achieve this in so many ways that after a while trying to write down a decent answer, I came to the conclusion that what you need is perspective. Stop programming and look at the problem with perspective. You are trying to enforce strong consistency among processes, That's fairly hard and costly, it's not exempt of complexity since it's required one more element in your IPC protocol and choreography a new set o events made in-expresso for the matter. – Laiv Jul 04 '19 at 07:09
  • That said, I would suggest reading about [Eventual Consistency](https://softwareengineering.stackexchange.com/questions/354911/microservices-handling-eventual-consistency/354927#354927) and [compensative transactions](https://en.wikipedia.org/wiki/Compensating_transaction). If eventual consistency makes things too complicated, then I would research about [linearlizability](https://en.wikipedia.org/wiki/Linearizability). Both approaches have trade-offs. The former trade consistency in favour of performance (IMO) and the latest trades performance in favour of consistency. – Laiv Jul 04 '19 at 07:14

2 Answers2

3

The TL;DR is: yes, there are technical challenges in preventing collisions, but it's really something you should ask your business department.

Best practice from a business perspective isn't a sequential number, but one that contains some structure. If the structure is chosen well, it may help avoid most race conditions.

One example is to use timestamps for numbering, usually in reverse order to make simple file name sorting provide a good oder: YYYYMMDD_document_title.

Another thing I've seen in use is to number documents according to cases and business years. A case number might be a customer number: <case-or-customer>/<year>/<sequence number>.

Any sequence may additionally be prefixed by a namespace identifier, i.e. one might see invoice numbers for a customer be numbered independently from correspondence.

Each business department chooses the system they work best with - so it's really up to them to tell you how to create this number.

That said, even if you do create numbers in such a way, there could still be collisions. You mention REDIS as part of your solution; REDIS can be used to create atomically incrementing counters.

I suspect that the best pattern for you would be to:

  • Choose a prefix generation method as I outline above.
  • Use INCR to generate unique counter values.
  • Concatenate your document name from those components and the title, e.g.: <prefix>-<counter>-<title>.
Jens Finkhaeuser
  • 161
  • 4
  • 1
    Thank you very much for your answer. While your answer does not really address the programming question that I had it does exemplify the nativity of my problem approach. I completely agree with you in that the business department should ultimately dictate how best to number such documents. The whole purpose of domain-driven-design is to reflect the business domain in code. In my case my project is a personal enrichment exercise to learn more about this method of programming. My project is contrived to the point where I **am** the business deptartment. LOL – Keith03 Jul 01 '19 at 23:06
  • All of these alternatives make the technical side easier, but IMO it's still a legitimate question. What if the simple sequential numbering is an immutable legal requirement, with no option of compromise? – Sebastian Redl Jul 03 '19 at 06:36
  • @SebastianRedl I have read in various places that some countries and/or jurisdictions require invoices to be sequentially numbered. This is part of my reasoning for the numbering method I am trying. The other reason I chose to do this is to figure out how to do it in a CQRS/ES microservice environment. Thank you for you great input. – Keith03 Jul 03 '19 at 17:15
  • @Jens Thanks for your additional input regarding Redis. My implementation currently is: `WATCH key; MULTI; INCR key; array = EXEC; UNWATCH;` This is using the Redis transaction method. Also, in my above edit you can see that I am actually planning on "checking out" the incremented number in a larger transaction using sagas with the option of "rollback" (decrement) upon transaction failure. i hope this makes sense. – Keith03 Jul 03 '19 at 17:39
1

I want to strongly discourage directly using DB id's as user readable identifiers of any kind.

Every unique identifier should only be beholden to one system. If you publish your DB id's where users see them they'll find some reason to mess with your system. Steve Jobs famously insisted on a having a different employee id number when he learned he was number 2. Rather than take number 1 from Waz he demanded to be employee number 0. Just don't let users see these numbers and this noise stays out of the low levels of your system. A simple mapping could have solved this problem and kept DB id's from being beholden to users.

An invoice number sounds like it could just be a DB id. Here's why not: you need to understand that uniqueness is only meaningful within a scope. A DB id is unique to a DB table. To a DB. But an invoice is unique to a business. Some businesses have more than one DB. Some get other DB's as they expand and grow. Some get other DB's as they buy other companies.

Don't even pretend that you can predict this kind of change. Maintain the uniqueness of your table by hiding the ID from anything that doesn't have to have a direct relationship with your table. Make the other things like invoices go through something that understands what those other things are. That translation is a business rule. Keep that business rule out of your DB.

This way your DB will work if you buy other companies or if other companies buy you because your DB doesn't care.

If you feel like so far I've just dodged the problem you're right. I have. That was the point. But if you want me to lay out a plan for the business rule consider an invoice number that has the ability to grow as more factors need to be added to ensure uniqueness. Then, much like a composite key, as conflicts arise they can be deconflicted by adding tags that place the rest of the invoice in it's previous scope. This is wildly different then trying to predict what these conflicts will be because no structure is imposed on what these new tags will be until we know what they are. All we've done is recognize that the invoice number may be required to expand. This may be as simple as using letters for former companies, allowing a delimiter like a dash, or a new field like "group" (that's what some insurance companies call it).

If you're thinking: Oh so I just expose the structure of the company you're really missing the point. You don't have to show how anything works under the covers. You just have to allow what you show to change as your needs change even while what's under the covers doesn't change.

So long as your design allows for this expansion, it doesn't have to predict it. Predicting what form change will take and getting it wrong does more damage than having left it alone. Just keep your direct exposure minimized and the work change causes will be minimized as well.

candied_orange
  • 102,279
  • 24
  • 197
  • 315
  • Thank you very much @candied_orange. I am not planning on using ID numbers directly from the DB. In fact, my aggregates, including the invoices will be identified in the repositories using UUID/v4 numbers. This UUID will never be displayed to anyone. You are correct though. Exposing such numbers allows a little too much inside baseball to external entities. For example, I recently used my online purchases from a website to roughly estimate their gross income by looking at my order numbers over the last few years. My project is more of an exercise for learning complex programming techniques. – Keith03 Jul 03 '19 at 17:24
  • Also, even though I am attempting to use sequential numbers for invoices, I could use sequences per customer too. I hope that I have understood your points correctly. – Keith03 Jul 03 '19 at 17:31
  • Obfuscating the business isn't the point. This is about not binding your design to destabilizing concerns without good reason. People being able to estimate your business income is just yet another reason not to make changing what is presented force a change in your model. – candied_orange Jul 03 '19 at 17:41
  • I'd generally agree with this sentiment, but in the REDIS example, it's not so much using DB IDs as it is using the DB's implementation of a thread-safe, strictly incrementing counter. Using the counter value does not imply using the counter value as a primary ID for documents. – Jens Finkhaeuser Jul 05 '19 at 08:56
  • 1
    @JensFinkhaeuser true. how you achieve uniqueness it isn't at issue so long as the number only exists for one reason, and thus doesn't have multiple meanings you won't have conflicting reasons to change. – candied_orange Jul 05 '19 at 13:18