19

I'm considering a project to migrate part of our WCF-based SOA over to a service bus model (probably nServiceBus) and using some basic pub-sub to achieve Command-Query Separation.

I'm not new to SOA, or even to service bus models, but I confess that until recently my concept of "separation" was limited to run-of-the-mill database mirroring and replication. Still, I'm attracted to the idea because it seems to provide all the benefits of an eventually-consistent system while sidestepping many of the obvious drawbacks (most notably the lack of proper transactional support).

I've read a lot on the subject from Udi Dahan who is basically the guru on ESB architectures (at least in the Microsoft world), but one thing he says really puzzles me:

As we get larger entities with more fields on them, we also get more actors working with those same entities, and the higher the likelihood that something will touch some attribute of them at any given time, increasing the number of concurrency conflicts.

[...]

A core element of CQRS is rethinking the design of the user interface to enable us to capture our users’ intent such that making a customer preferred is a different unit of work for the user than indicating that the customer has moved or that they’ve gotten married. Using an Excel-like UI for data changes doesn’t capture intent, as we saw above.

-- Udi Dahan, Clarified CQRS

From the perspective described in the quotation, it's hard to argue with that logic. But it seems to go against the grain with respect to SOAs. An SOA (and really services in general) are supposed to deal with coarse-grained messages so as to minimize network chatter - among many other benefits.

I realize that network chatter is less of an issue when you've got highly-distributed systems with good message queuing and none of the baggage of RPC, but it doesn't seem wise to dismiss the issue entirely. Udi almost seems to be saying that every attribute change (i.e. field update) ought to be its own command, which is hard to imagine in the context of one user potentially updating hundreds or thousands of combined entities and attributes as it often is with a traditional web service.

One batch update in SQL Server may take a fraction of a second given a good highly-parameterized query, table-valued parameter or bulk insert to a staging table; processing all of these updates one at a time is slow, slow, slow, and OLTP database hardware is the most expensive of all to scale up/out.

Is there some way to reconcile these competing concerns? Am I thinking about it the wrong way? Does this problem have a well-known solution in the CQS/ESB world?

If not, then how does one decide what the "right level" of granularity in a Command should be? Is there some "standard" one can use as a starting point - sort of like 3NF in databases - and only deviate when careful profiling suggests a potentially significant performance benefit?

Or is this possibly one of those things that, despite several strong opinions being expressed by various experts, is really just a matter of opinion?

Community
  • 1
Aaronaught
  • 44,005
  • 10
  • 92
  • 126

2 Answers2

7

On the topic of "every attribute change"

I think you missed the point. Mr. Udi Dahan is saying you should capture the user's intent as a command. An end-user is concerned with being able to indicate that a customer has moved. Depending on the context that command could contain a customer identification, the new address (split up into street, streetnumber, zipcode, ...), optionally a new phone number (not uncommon when you move - maybe less so with all these cellphones). That's hardly one attribute. A better question is "how do I design commands?". You design them from a behavioral perspective. Each use-case, flow, task an end-user is trying to complete, will be captured in one or more commands. What data goes with those commands comes naturally, as you start reasoning about them in more detail. The thing to watch out for is data that gets interpreted as "logic flow control" on the server side. That might be an indication that you need to split up the command. I hope you never find that standard with regard to command granularity. Good question though!

Yves Reynhout
  • 236
  • 1
  • 2
  • This definition still feels very arbitrary to me; a CSR's conceptual model may lump preferred status and martial status together the same way you lump address and postcode together. I don't mean to split hairs, it just seems to me that in order to really understand if they're different behaviours, you have to be able to predict the downstream effects, and OTOH the whole idea of ESB and CQS and pub/sub is that you aren't supposed to know or care what happens downstream. Thank you for your answer, I do appreciate it, although I can't say I feel any more enlightened so far... – Aaronaught Apr 08 '11 at 18:51
  • @Aaronaught: The definition **is** arbitrary. The granularity of a Command should be *whatever makes sense for your particular scenario*. There's no one size fits all. There are several guidelines, such as matching commands to use cases or tasks or actions available in the UI - another is to prefer more granular commands over less granular commands (particulaly as Yves said being wary of data that is interpreted as logic control flow) - but no hard and fast rule. Is there an actual scenario where "one user potentially updating hundreds or thousands of combined entities and attributes"? – quentin-starin Apr 08 '11 at 19:18
  • That's the whole point! Don't lump together. Divide according to behavior! Don't put data in the command that does not align with the intent of the command/end-user. And that's not about downstream systems. – Yves Reynhout Apr 08 '11 at 19:20
  • @qes: In our systems there are several such scenarios, very real and very necessary. To state it as simply as possible, they need to modify entire sequences of data and these sequences only make sense as sequences. Of course they don't usually make these changes record by record, they apply some algorithm to the bulk of it and then fix a few exceptions. Maybe this just isn't an appropriate scenario for CQS to begin with, but that decision is just a subset of my broader question. – Aaronaught Apr 08 '11 at 19:24
  • @Yves: I'm not really sure that that *is* the point, given how user models are often myopic, fragmented, and inconsistent. Your *business* may have very clear and distinct rules around both marital status and preferred status, but those rules may have been invented long after your original use case analysis, and your CSR may have no awareness whatsoever of what "preferred" means (it's just the output of some policy they follow). To your business, it may be very important that the user doesn't accidentally override changes to preferred status due to stale data. Whose model do you follow? – Aaronaught Apr 08 '11 at 19:28
  • If the operation is really a single logical operation, then a single command can be appropriate. And there's no dictation of how that command achieves its result. If the command handler needs to execute a stored procedure to affect a set of data, so be it. What we like about CQRS is that it feels less restrictive in this way. We model our systems mostly through Commands and Events, as these are the language of our domain so to speak, and how exactly we implement that - whether its ORM backed entities, event sourcing, or DataSets - are less important implementation details. – quentin-starin Apr 08 '11 at 19:29
  • And we do mix implementation techniques inside a single application - especially since we apply CQRS in refactorings. In some places our read and write sides share a relational DB and our handlers use transaction script style code with SubSonic, others we use EF entities, and yet others we've applied event sourcing. What's important is what you want the system to do and what has happened in the system (commands & events), imho. – quentin-starin Apr 08 '11 at 19:32
  • 1
    @qes: Fair enough, and that's an answer in and of itself. I certainly understand the concept of a logical operation (that's how the existing services are modeled), I guess I've just been concerned that CQS seems to change a few of the rules around how you *ought to* define an operation. "Traditional" SOA seems to start from the coarsest possible definition and move down the abstraction ladder if necessary; my understanding of CQS so far seems to indicate the opposite, start from the most granular definition possible and abstract if it looks too much like RPC or control flow. – Aaronaught Apr 08 '11 at 19:36
  • It makes suggestions about how you ought to define an operation. But like many aspects of software development, it isn't one size fits all. Sometimes you need a command to result in a lot of action, I know we do. One other note, when we do have commands that are batch operations, we still strive to raise all the appropriate events for individual entities involved – quentin-starin Apr 08 '11 at 19:43
  • @Aaronaught Mind you that neither commands nor events are static. They evolve - schema-wise - as your software evolves (whether you upgrade the past events is an implementation detail). Migration happens in "other" software development as well, no different in CQRS/ES. On another note, if your end-user does not understand your system, that's hardly the fault of CQRS/ES or any other tech/arch style for that matter. That sounds more like an organisational problem where proper training is in order or your app needs to change to be more user-friendly/-guiding. – Yves Reynhout Apr 08 '11 at 20:15
  • @YvesReynhout (I know it is after 3 years, so maybe you don't answer.) What do you suggest by event granularity? I mean by coarse events you can have problems later, for example by adding a new attribute to an event, you have to create a new class or add an event version, etc... How do you solve this problem? Do you make event versions, or very small events, or replace the events with the new ones in the event storage? It is hard to find a best practice, and I hate to fix issues later... :S – inf3rno May 23 '14 at 23:41
2

The message Udi is trying to get across is that CQRS is more than just CRUD. Why did I create this record? Why am I changing this record? Why is it being deleted/marked as deleted?

Commands should correspond to actions/use cases that the user makes with the system and express the intent of the action rather than just saying change this. Also, it may seem like it's finer grained but it could be much coarser than it first appears. For instance the upgrade to gold status might involve a change to several attributes and a number of other aggregates might even respond and change in response to the corresponding event.

CQRS is about capturing the language of the business in the Service layer so the UI doesn't have to worry about what happens when I make that gold status upgrade or the shipment has been marked as undeliverable by the carrier, or the employee has been promoted to manager of the technology group. Well technically I'm talking about Event Sourcing now but you get my drift. There are more distinct messages but they are not necessarily more fine grained than standard CUD.

Michael Brown
  • 21,684
  • 3
  • 46
  • 83