31

I'm beginning a new project at work, and will likely be nearly the sole developer on the project, though one or two other developers will need to integrate existing applications or simple scripts into the main project. The project needs to handle small-scale bulk and streaming data ingest/processing, and both event-driven and on-demand code execution. Some parts of the framework will be heavily CPU bound, and some parts could be heavily I/O bound; most of the data must live on a single machine, but we're able to create a cluster and connect VM's to increase available compute power. There will probably be one or more small web applications that depend on services provided this core framework. The main language will be Python for just about everything.

My question is whether or not I should take a microservices approach to an effort like this or stick with a monolithic application, given that I'll be doing most of the development by myself. My thought is that microservices (using Nameko) provide a natural separation between elements of the framework that have different execution models (data pipelines, event-launched, on-demand, web applications, etc.) and a clear way to distribute the workload and communication across multiple processes. My concern is that I'd probably end up with a Kubernetes cluster to manage (I'm familiar with Docker, but still fairly new to Kubernetes), multiple services (rabbitmq, redis, etc.) required just to facilitate running the system, and potentially a lot of small chunks of code to actually implement all the necessary capabilities that we'll need in the framework.

For a project with little more than a single developer, do microservices still simplify developing and maintaining a complicated system like this? Are there methods/systems/frameworks I should consider using instead, or to reduce the overhead involved in designing the system this way?

scnerd
  • 421
  • 4
  • 5
  • 11
    You use microservices when you need the benefits that microservices provide, and those benefits outweigh the costs. Personally, I don't see why you would need microservices in an individual application written by one person, unless you were teaching yourself or you had a long-term outlook for a larger application. – Robert Harvey Oct 05 '18 at 14:53
  • To update this a couple of years later: I ended up with most capabilities centralized in one "large" project, then used Nameko to bind in services that had extremely different and specialized dependencies (e.g., one service required specialized licenses and needed to run on Windows; another was a service written in Rust for efficiency and wrapped in Python for ease of use). These services were useful to be able to easily scale up. However, accessing these from development instances of the main application became an extra burden and complexity, and developing them independently was tricky too. – scnerd May 06 '20 at 13:10

1 Answers1

50

Microservices are generally undesirable because they turn your software into a distributed system – and distributed systems make everything a lot more difficult. But a service-oriented architecture has some important benefits:

  • different services can be developed and deployed independently by different teams
  • different services can be scaled independently

As you will be the sole developer, you do not need the flexibility to develop the services independently.

But you note that some parts may be CPU-bound. So it might be desirable to scale those independently from the rest of the application. If that is the case, that doesn't mean that you have to turn the whole project into a microservice-architecture. You only need to move that CPU-intensive part into its own service, and can keep the rest in a convenient monolith. Along which lines the system should be split up is difficult to tell, but in general the DDD idea of “bounded contexts” is a good guideline.

Note that monoliths are not bad. Monoliths do not equal a huge messy unmaintainable project. Where you could split the system into different microservices, you can also split the system into different components within a monolith. The separation between these components is just more visible and more clearly enforced in a service-oriented architecture. This also means that for a well-designed system, it should be fairly easy to turn a component into a service at some later point. So you don't have to decide right now, you can move to microservices if and when a monolith has proven itself unsuitable.

Consider also Martin Fowler's concept of the Microservice Premium (2015): microservices introduce substantial complexity of their own, in addition to the base complexity of your system. You have to pay this “premium” in terms of reduced productivity. This means that for simple projects, microservices make you less productive. This changes for more complex projects: whereas a monolithic solution might become increasingly difficult to work with, a microservice architecture scales much better and requires roughly constant effort. You have to know whether the extra initial effort of microservices is worth it given your software system. Since you are asking this question, the answer is probably “no”. Fowler continues:

So my primary guideline would be don't even consider microservices unless you have a system that's too complex to manage as a monolith. The majority of software systems should be built as a single monolithic application. Do pay attention to good modularity within that monolith, but don't try to separate it into separate services.

amon
  • 132,749
  • 27
  • 279
  • 375
  • 32
    TL;DR version: Only add complexity when it solves problems you actually have. – jpmc26 Oct 05 '18 at 19:48
  • 1
    Architect it in such a way that you could more easily migrate to a microservice architecture later. E.g., separation of concerns vs. a big ball of mud, and it will be easier to maintain now and easier to move portions out to separate services later. You can still have the microservice-inspired division of duties/domains without making calls/sending messages to other services. – ps2goat Oct 05 '18 at 20:04
  • 1
    Fowler's First Law of Distributed Objects: Don't – K. Alan Bates Oct 06 '18 at 01:34
  • 1
    There is a third benefit, although it is similarly useless to the OP: If you have to build a distributed system anyway, and if you already have or plan on building good tooling for your distributed system, a microservice will integrate with that tooling more easily and allow for finer grained control in many aspects. This can simplify troubleshooting, profiling, integration testing, tracing, etc. But that obviously depends on those tools existing and supporting your microservice architecture. Without this supporting ecosystem, microservices are just added complexity. – Kevin Oct 06 '18 at 03:29
  • 4
    Yep, good answer. People seem to forget that microservices actually _add_ complexity -- so unless they remove _more_ complexity it's not worth it. And in nearly all cases a single dev won't have enough benefit of doing MSA. – enderland Oct 06 '18 at 16:18
  • @jpmc26 or expect to have in the (relatively near) future – Drakarah Oct 06 '18 at 17:26