Deployments, another unsolved problem in software engineering

Question

Why is it so hard to package and deploy software? I want to understand what exactly is the inherent complexity of creating self-contained binaries? Currently it seems that golang is the only game in town when it comes to almost completely solving the deployment problem but even go has some issues with versioning dependencies. Why aren't more languages solving this problem at the compiler/language level?

Note that I'm not talking about library code. I'm talking about an actual application or a service that needs to get to a production environment as quickly and as reliably as possible.

Answer 1

• Dependencies. Take a very simple Python application. I can distribute it as a script. It will work on some machines, and won't work on others, because the script is using Python 3 and those machines have only Python 2 installed, or no Python at all.

  So now, I need to make sure, during deployment, that a specific version of Python is either available or will be installed during deployment.

  How would you solve that at compiler/language level?...

  See also: Has anyone nailed dependency management?

• Configuration. Take a very simple service which receives calls from the outside. It works on my machine. I deploy it on the server, and it stops working. Nothing happens: the service receives no calls, no matter what I do.

  This is because the server has a firewall which blocks the port used by the service.

  Which means that my deployment script should open the specific port. If I have total control over the servers and know that every server uses UFW as the only firewall, the task is relatively easy. If I don't have such control and has to support multiple firewalls, the task becomes more complex.

  There is nothing a compiler/language can do for me there.

• Infrastructure. Take an example of a web app which uses MongoDB. If there is no MongoDB server, the app won't work. This means that database should be deployed as well, probably on a different server. Thus, I should create a dedicated virtual machine for the database, deploy MongoDB, and configure the deployed web app by setting the proper connection string.

  But then, it appears that I need a failover, so now I need two database servers, and I need to change the connection string accordingly. More importantly, I need to configure the failover so the data replicates as expected after one of the database machines is deployed.

  Then, I don't want to lose all the data if the VMs which host the database are destroyed, so I make backups. Now, the deployment script should also handle the recover from the latest backup.

Those three points are just a small subset of all the challenges you may have during deployment.

In all those points, neither your compiler, nor the programming language can help. This is because most of the deployment challenges are not inherent to the language/compiler/framework you use.

But wait, there are instead a bunch of tools which are specific to deployments:

• Package managers. Have you used npm in Node.js, pip in Python or NuGet in C#/Visual Basic? Their handling of dependencies is particularly exciting, and they make it quite easy to deploy a small software product. Have you used apt-get? The sole fact that you can simply type apt-get install something and let the tool do all the job, no questions asked, is nearly magic.

• Configuration managers/infrastructure automation tools. Tools such as Chef, Puppet or OpenStack do a great job of making it easy to automate deployments. It doesn't mean that you can say "Make me a MongoDB failover" and watch the result (although, it does exactly that for simple cases), but at least, you can have a reproducible, automated process which makes your life much easier.

Answer 2

Deployment is easy. You package up your binaries and reference any dependant projects it needs. Then you deploy the package and binaries to a well-nown location and anyone can install it, and whatever dependancies they do not already have, with a single call.

Of course, if you're not using a Linux repo then you'll have to use whatever (inferior :) ) deployment mechanism your OS supports.

But that still doesn't handle per-installation specific things, such as adding users which is something you cannot deploy automatically as it changes per site. Sure, you can make this easier and a end-user can build deployment scripts using a tool such as puppet or chef and deploy a customised setup.

Even then, there are advances in deployments - look at virtualised application systems such as Docker. this allows you to build up a system (eg pre-deployed) and then ship it in its own little world, isolated from the rest of thr OS so it doesn't conflict with any other application that is installed.

Answer 3

The root of this problem is that your program doesn't live in isolation in your system hard drives are shared, libraries, etc.

In this realm Snappy Ubuntu makes an important contribution, from their website:

Ubuntu Core provides transactional updates with rigorous application isolation. This is the smallest, safest Ubuntu ever, on devices and on the cloud. We’re excited to unleash a new wave of developer innovation with snappy Ubuntu Core! (Mark Shuttlewort)

What that means:

• Transactional: A new installation succeeds completely or fails completely. If it does fail then it rolls back and leaves the system exactly how it was. No more crap from files of a partial update.

• Application Isolation: It doesn't matter what specific version of a library your program uses depends on, that library won't be overwritten by any other application. Nor can your application delete other apps files.

• Small: This basic version of Ubuntu is around 70 MB.

Answer 4

I don't believe this is a language problem. This is more of a software architecture or run-time architecture or operating system architectural concern (note, I don't say problem).

If every software deployment was a self-contained binary then every software package would be an absolutely HUGE install. You would need to replicate every dependent library and run-time piece of software needed by the software in question.

So really, unless you wanted to ship a 6 gig version of solitaire, external dependencies are a good thing.