Re-inventing system design for Scala

Question

Many, many, moons ago, I did my masters in Object Orientated Software Engineering. I covered everything: project initiation, requirements, analysis, design, architecture, development, etc, etc. My favourite IT book of all time was Developing Object Oriented Software, an Experience Based Approach (IBM-1996). A book created by a group of true experts of their time. It describes a work product centric approach to object oriented analysis, design and development methods.

I designed and developed and was happy and at the top of my game, but I started to feel a little dated: Agile movements became fashion of the day and re-branded some well known iterative and incremental approaches with new hip words. Suddenly inexperienced developers started frowning when I said "requirements" or "architecture" as if these things were superseded by magic.

Design and development lost it's fun and I was about to leave the whole IT industry behind me.

Then I discovered Scala. Oh, like soft rain on a dusty road. Everything just became clear, the air became sweet again and the little light of my hard-drive would flicker deep into the night, merrily keeping me company in my world of discovery.

I like system design. I'm an architect at heart, more than a programmer. I love analyzing, designing, thinking, arguing, improving - I just love simple, clean and crisp designs.

How do we design pure Scala solutions?

Surely conventional sequence diagrams, interaction diagrams, object diagrams, etc, could all be replaced or improved for the goodness of blending imperative and functional object orientated systems. Surely there is an opening for something totally different!

I'm not looking for bloated complexity - I'm looking for flexible simplicity. Much like Scala is actually simple and easy (albeit different), but can be extended to fit your requirement like a pair of Yoga pants. There must be a design system for this lovely language that will start simple, and can be extended to fit your requirements and your domain. Surely UML can not be it!

So how do we design pure Scala systems? Imagine for a moment, you have the luxury to design a complete system from scratch, and know that you will only be using Scala - How will your models look? What types of diagrams would you have to describe structure and behaviour? How would you model Options, matches, mixins, singleton Objects, etc. without being drawn into the complexity of extending existing modeling techniques rather than a fresh, light-weight, innovative tool-set.

Does such a design/process/solution exist, or is it time to invent one?

Answer 1

I can't really give an answer to this question, but let me offer some thoughts.

I am a computer engineering student in a university, and last semester me and a group did a large software project in which our primary language was Scala. We did our design the traditional way: use cases, a domain model, sequence diagrams, UML class diagrams; the usual load. And, as you are already aware, they are a poor fit. Here are some reasons.

First, consider sequence diagrams. The feel of a sequence diagram is of a few stable, long-lived, semi-autonomous actors talking to each other. In Scala, objects tend to be created for a short time. Furthermore case classes encourage "dumb" objects, with only data and no code, so the concept of passing messages is out of place. But the hardest challenge for sequence diagrams is that they have no good way of incorporating first-class functions; in an OO design using only a callback here or there it can be made to work, but if that is your primary means of transferring control you will find the sequence diagram reflects little of your actual code.

UML class diagrams are more amenable to Scala; yes, mixins don't come through well, but that is something that could be "tweaked" rather than overhauled. But I think that might be missing the point.

Consider again why Scala has "dumb objects". If you write a case class with no methods:

case class NewInvite(user: User, league: League)

you don't need any methods, because the types of the members promise something about what you can do with them. In fact, they promise everything. And, in every scope within the program, the variables within scope and their types promise something about where the code can go at that point.

So, maybe, if we step back a bit, and, instead of thinking minutely about types per se, design our program in terms of what contexts our code will live in, and what is promised to the programmer (and, eventually, the user) in each of these contexts, we will find a description more appropriate to Scala. However I am only guessing here, and I think you are right that much more needs to be explored in this area.

Answer 2

UML emerged out of the "Bubble Wars" of the late 80s and early 90s. It was always a compromise for notation, not a design method. Many of the calumnies attributed to UML are secretly the result of a design method that says "Step 1: draw a class model."

I suggest that we don't need new design methods as much as we should revitalize some old ones. Start with defining and allocating responsibilities, then move to understanding their implementations, and follow with good notations and representations to communicate those ideas.

Responsibilities

CRC works as well for Scala as it does for any other OO language, which is to say that it works very well! Modeling the allocation of responsibilities by anthropomorphizing actors and understanding their collaborations still works well.

At a large scale, you should still be designing with objects. Object boundaries are encapsulation boundaries. Keep mutable state and implementation details inside those object boundaries. Case classes make good interface objects, so do vanilla collections and data structures.

Understanding implementations

Pass these data structures across object boundaries rather than other objects, and you'll find a) it's easier to keep an object's guts hidden inside, and b) functional implementations of object behaviors make a lot of sense.

You won't want to treat the large scale structure of your program as "a vast pool of small functions". Instead, you'll naturally gravitate toward narrow interfaces between isolated parts of the system. These look and feel a lot like objects anyway, so go ahead and implement them as objects!

Representation and Notation

So, within this design structure, you still need to get the thoughts out of your head and share them with someone else.

I suggest that UML can still be useful for describing the large-scale structure of your system. It will not help at the lower levels of detail, though.

At the fine grain of detail, try literate programming techniques.

I also like using minidocs. A minidoc is a one or two page document describing some specific facet of system behavior. It should sit near the code so it gets updated, and it should be short enough to enable just-in-time learning. It takes some practice to hit the right level of abstraction: it needs to be specific enough to be useful but not so specific that it gets invalidated by the next code change.

Answer 3

How do we design pure Scala solutions?

I think you're taking the wrong angle with this question. Your overall design for solutions should not be tied to a particular language; rather, it should correspond to the problem at hand. The great thing about Scala is that it is flexible enough to not get in your way when it comes time to implement your design. Java, on the other hand, enforces some design decisions (primarily, the everything-is-a-class mentality) that will make it feel a bit clunky.

When designing, try to model state explicitly. Immutable data and explicit state lead to designs which are much easier to reason about. Most of the time, this will translate nicely into a functional programming solution, although occasionally you might find that it is more efficient or more straightforward to use mutation and an imperative style; Scala accommodates both quite nicely.

Another reason Scala is so great at getting out of your way is its ability to create DSLs that suit your needs. You can make your own little language that solves a certain problem the "normal" way, and then simply implement it in Scala.

In summary, my main point is that you shouldn't be trying to "design pure Scala solutions". You should design solutions in the most natural and understandable way possible, and it just so happens that Scala is an incredibly flexible tool that can help you implement those solutions in a variety of ways. When all you know about is a hammer, every problem starts to look like a nail, so be sure to explore the various approaches there are available to you. Don't try to close the design process to steps X, Y, and Z, lest you miss out on possibilities A through W.

Answer 4

You're right that OOD is limited and outdated. Its main advantage is that it is so well specified and ritualised, with fancy names for each and every trivial pattern and design trick. You won't find any comparable system for the more modern and rational approaches to the systems design.

I can only list a couple of things: a semantic approach to design, a subset of which is called Language-Oriented Programming (and is heavily used by the Scala community) and Domain-Driven Design. The latter is an oversimplified and OOPised view on a more generic semantic design approach, you'll only be able enjoy it if you like OOD.

I'd also recommend to learn a trick or two from the other functional programming communities - e.g. Haskell and Scheme, not all of their design know-how had already been transfered to Scala.

Answer 5

The question was: How do we design pure Scala solutions?

Answers like,

1. We use XYZ because ....
2. We use ABC but like so ....
3. We started using XYZ, but then found ABC to be better because ...

would have indicated some maturity, or generally accepted existence, of such a tool-set or solution.

Comments relating to whether it's appropriate to consider a Scala specific design tool-set is a different matter. My opinion on this is that Scala is revolutionary in the way it combines imperative and functional programming. Scala captures an amazing breath of development practices, concepts and principles, and so, in finding a solution that is the perfect fit for Scala, we will probably discover a solution for which a subset will serve many other languages very well also.

Is it safe to say then, that we know the answer to the fallback question:

"... is it time to invent one?"

Could it be yes? (Where 'yes' does not prescribe the means to the solution. It could be achieved either by extending an existing solution, or creating one from the ground up. It does admit though that there are significant shortcomings in existing design options)

You are welcome to vote my answer down if you disagree. I'll take it like a man.

Answer 6

I wouldn't differentiate between languages, I'd do it the same way in java. However I'm from the OO school rather than the functional (algebraic Haskell one or the lisp one). Haskell app design is really different from Scala or Clojure. Also a Scala.js app design is different due to the stateful DOM - it is hard to fight a stateful part of your app that is mandatory. Over time I got used to doing it the OO way, while trying to eliminate state and mutability as much as possible. If I were a typelevel or scalaz fan, my apps would probably look quite different.

1. Deciding on technology stack and the major design decisions like client/server responsibility, distributed nature - do we really need data persistence? What suits our application the best? (certainly not libraries or frameworks yet)

2. Starting with just a single App.scala file where I define the key types (traits and case classes) and roles - a lot of contemplating and thinking while being AFK. It is so easy to play with it while it is in one file. Even the prototype might come into being if it is rather a simple application. I dedicate a lot of time to this phase because nothing is more important than having the most correct and mainly transparent and sensible prototype possible. It all helps to make our further contemplating more accurate and it increases probability of success.

3. Now when we have proven correctness of our solution, we have clear vision and all potential problems are revealed, it is time to think what kind of third party libraries or methodologies to bring in. Do we need play framework or just a few libs? Are we gonna make these stateful actors, do we really need Akka's resilience, .., .. or maybe not? Do we use Rx for these streams? What do we use for UI so we won't go insane after it has 10 interactive features?

4. split the App.scala file into multiple ones because new traits and classes appeared and got bigger. Their interface should tell about their roles. Now it is also time to think about utilizing type classes and ad-hoc polymorphism where possible. So that whoever calls your interface, it must be flexible for them. The point is to implement only general functionality, leaving the details to the caller. Sometimes what you are trying to implement might be Turing-complete-like so you should provide a way for callers to implement specifics by themselves instead of piling up feature requests on GitHub. This stuff is hard to add later. It must be thought through at the very beginning. As well as designing a DSL.

5. thinking the design through because the application will grow and it must be ready to withstand new features, optimizations, persistence, UI, remoting, etc. - stuff that was not implemented yet or was somehow mocked or abstracted over. Remember that while it's all in one file it is quite easy to change things. Human brain starts loosing it after it gets spread across 10+ files. And it is like a tumor that grows, that's how overengineering starts. I noticed that my applications end up with a few long partial functions that form some sort of a backbone of it. I find it easy to work with. I guess I became infected by Akka actors.

6. Now that the first actual features exist, not just the core functioning, it is time to start writing tests. Why so late? Because during this time you probably went through major rewrites and you'd have wasted time maintaining those tests. One should not write tests unless one is really sure there won't be any major redesigns. I prefer integration tests that easily withstand continuous refactoring and I won't spend more time testing that actually developing stuff. It happened to me a few times that I spent way too much time maintaining tests. Fixing potential bugs would certainly pay off more rather than bad tests. Bad tests or tests written from the very first moment can cause major pain. Notice that I'm certainly not a fan of TDD - IMHO it's bullshit.

7. Refactoring, making and keeping the application design readable, component roles clear. As the application grows there is no way it stays that way unless you are a genius programmer and if you read this answer you're probably not:-) Neither am I. Usually some stateful things exist because you didn't know how to avoid that - that might cause problems, try to eliminate them. Also be sure to eliminate code smells that have bothered you for some time. At this point your project manager (if you have one) probably starts shaking his head. Tell him it will pay off.

8. Done, is the app well designed? It depends: Do the have components well-defined roles that actually make sense even outside of your head? Could you take some of them and open source them without much effort? Is there clear separation of concern between them? Can you add a new feature without worrying it will crash elsewhere (sign of you know exactly what you are doing)? Generally speaking, it should be readable as a book with straightforward pragmatic solutions to problems. Which is IMHO the major sign of good design.

All in all, it's just a hard work and time you probably won't get from your managers. All I described here takes a lot of time, effort, coffee, tea and especially thinking AFK. The more you give it, the better the design. There is no general recipe for designing applications, common sense, pragmatism, KISS, hard work, experience means good design.