2

The Background

I have a friend (no, really!). This friend works in the layer between IT and end users. Something like a business analyst or consultant. This friend does not have a technical or IT background. They have gained a high level understanding of software and systems, but, for example, are not able to read/write code or converse on the low-level details of software.

My friend's company has Continuous Integration (CI) / Continuous Delivery (CD) pipelines in place to build & deploy their software, but these pipelines must be manually triggered.

My friend is being asked to take responsibility for triggering the automatic deployment pipelines, both for non-production and production environments. The process for doing this is technically simple and quick:

  • Login to automation software
  • Select appropriate CD pipeline job
  • Choose a couple configuration values (environment to deploy to, software artifact to deploy)
  • Hit the button to trigger job

To deploy, or not to deploy

My friend does not feel comfortable accepting this responsibility, even though the process itself would not be difficult for them to perform.

I am a software engineer responsible for triggering similar automated deployments at my company, and I would not feel comfortable handing this responsibility over to a less-technical user, like my friend. However, we are both having difficulty in articulating what about this makes us uncomfortable, and we're not sure if we're being unreasonable.

What's the big deal, anyways?

When I, as a software engineer, perform the process to trigger an automatic deployment, I do more than just trigger the job.

I familiarize myself with the code changes between the artifact that is already deployed, and the artifact I am deploying (if I am not already familiar). I assess the scope and risk of these changes. I assess the affects of these changes on the system and dependent systems. If, during the above steps, I identified any areas of risk, I prepare myself to respond to those specific items, if need be. Using my knowledge, I make a final judgement call of whether to proceed with the deployment.

Once I trigger the deployment job, I monitor the status. If the job fails, I first confirm that the automated rollback was successful and that the application / system is not impacted. I then begin initial analysis & troubleshooting of the job failure, and whether another attempt is warranted or if action needs to be taken first.

After the job completes successfully, I perform some sanity checks of the deployment process and software health.

I have intimate knowledge of the code, of each step performed by the automated deployment process, and of the system infrastructure & architecture. This knowledge informs my decisions.

Now, I will be the first to admit - a lot of this stuff could be either automated, or built into the change management process before it gets to me pushing the button. In an ideal world, the whole process could be entirely automated and continuous - no humans necessary.

That said, I do not believe my friend's company currently has a change management process or automation that would handle all of these.

TL;DR

What is needed to allow non-technical, non-IT users to manage automated deployments of software artifacts (both in non-production and production environments). What safe guards must be put in place to make this process safe and reliable? Or is this just a bad idea no matter what?

Johndt
  • 207
  • 1
  • 5
  • 1
    Even if this was a push one button deal it’s still technical. Just knowing when to push it is technical. Reporting what you did before the system spontaneously tore itself apart is technical. Even knowing what responsibilities you’re taking on here is technical. You can automate a lot of things but the moment a human is involved they need to understand what you’re making them do. TL;DR It’s not a self driving car if I can’t sleep in the back seat. – candied_orange Dec 04 '20 at 16:27
  • 1
    Unarticulated responsibilities (such as the responsibilities you listed), unmitigated risks (your friend's workplace isn't implementing all the best practices that your workplace does), and unarticulated **political** risks (such as your friend being used as a convenient scapegoat, fired, in a fire pit, in case something goes wrong). – rwong Dec 04 '20 at 16:58
  • Does it have to be manually triggered? Can you not fully automate the process to deploy silently on a regular basis? – John Wu Dec 04 '20 at 18:26
  • i am going to plus one this simply for the opening statement :-D – simbo1905 Dec 04 '20 at 18:56
  • 1
    @JohnWu - I think that is pretty much where I am. If you as a company are confident enough in your automation & processes to allow someone with no SME to "press the button", it's not much of a leap to fully automate the process and have *real* CD. I think the reality is that the company is not that confident in their automation. – Johndt Dec 04 '20 at 19:06
  • My suggestion would be to mitigate the risk via a promotion process, e.g. fully automatic deployments to a lower environment on a daily basis, then weekly (or so) pushes to upper environments. – John Wu Dec 04 '20 at 19:59
  • Why does it have to be a human that does the deploy instead of an automatically triggered deploy? – Thorbjørn Ravn Andersen Dec 05 '20 at 13:31

4 Answers4

6

It is actually more than just triggering the pipeline job to start.

If the job fails how should we know if it's the bug from your artifact, resource shortage, or some transient error from the dependency services?

What if the deployment succeeds but some time later the actual calling to the service fails? How should we target to the issue, and how should we know which part of the job we need to rollback?

Those are all technical parts.

lennon310
  • 3,132
  • 6
  • 16
  • 33
3

Your question contains parts of the answer. The deployment by that non-technical person must be monitored and rolled back automatically. Basically, it means automation of the workload you do while monitoring the deployment. Of course, this automation won't cover all possible scenarios but increases the certainty of success. It also involves additional costs to deliver such a solution. One might want to estimate possible risks on historical data, like "this system breaks every two months due to that subsystem". That estimation will tell how easy or hard the automation to implement and rely on. Also, one may want to write an escalation policy to back up that person.

Toshakins
  • 77
  • 3
3

You need some sort of approval system in place so technical people and business people can understand what will be deployed, when, and what impact it has on the wider technology ecosystem. Your friend should only need to "approve" the deployment. And once all approvals have been recorded (and approved) then the automated build system should start the deployment at the scheduled time.

Deployments, even if they are push-button, require intimate technical knowledge. It's not that pushing a button requires expertise, but as you said, the failure scenarios are complex. A multitude of checks need to happen before and after a deployment. Some are manual. Most can be automated.

This is why many DevOps-style tools have approvals built in to their push button deployment systems. Sure I can push a button. Sure I can roll things back, and I know that this will not affect other systems. Do end users want me to deploy the changes right now? The approval system provides this transparency to everyone involved.

The bottom line is: you cannot make it safe for a non technical person to deploy software by themselves. You need communication to technical and business people. Approval systems give you this communication channel.

Greg Burghardt
  • 34,276
  • 8
  • 63
  • 114
1

Classic deployments of non-web software with no "CD" process in place are usually organized for people who don't know the code of the software and all of its dependencies, what you may call "non-technical persons". I call them admins, and they usually have a lot technical knowledge, but that does not include intimate software development knowledge about the components which are installed.

For someone who is deploying a new version of a piece of software, which is actually a black-box for them, the following things are required:

  • a robust setup-routine and/or installation manual

  • a reliable change-log, written from the end-users perspective, so someone can decide about the risk and chances of deploying or not deploying the newest version of a component

  • useful logging options in case the installation/deployment fails (which can be send back to the developers)

  • an uninstaller or roll-back mechanism, maybe combined with backup/recovery mechanisms

  • a hotline at the software manufacturer which can be called in case a problem occurs

In principle, the same things are required in a CI/CD based system. However, in such systems, the frequency between deployments is usually much higher than for non-web software, because of higher automation, smaller development cycles and less technical overhead for doing the actual deployments. Hence, the classic organization around several parties (devs, users, admins, hotline) maybe simply not effective enough any more for this kind of process.

That's where the "DevOps" role comes in, which actually combines some of the formerly separated responsibilities of an admin, developer and hotline in one person. This role requires usually intimate knowledge of which features can be rolled-out safely, and maybe some knowledge about the code as well to fix problems without having to call a separate support-hotline first.

So let me finally try to answer your question: for making the CI/CD process something which can be triggered manually by your friend - and not by a person which has a DevOps qualification - the organization requires the same things like mentioned above:

  • a robust deployment mechanism
  • high-quality change logs
  • deployment logging
  • rollback mechanisms
  • people who can be called by your friend in case something goes wrong

and an organization which is willing to accept that this kind of process will not achieve the high frequency of CI/CD like Facebook, where they deploy several dozen times a day.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565