-3

We have some linux(ubuntu) based VMs running in the cloud in a private network and they do not have external/public IP address. We have some apps running on those servers and everything is working fine. My question is: are there any arguments for doing regular updates on those machines? How about security patches only?

The counter arguments I came up with are:

  • Updates would create some downtime for our service
  • updates can break things
  • time spent updating + testing things still work after update can be spent somewhere else
  • these machines do not have internet access so we need to enable it during the updates which is itself a security risk.

PS: Of course, bastion hosts will be fully secured/patched.

enter image description here

Caner
  • 95
  • 3
  • 3
    Any infected machine with access to both the internet and your internal network can be a gateway for infection. Internal network isn't automatically safe. – whatsisname Oct 08 '18 at 14:49
  • 3
    Several "closed" systems in the past were attacked by other means than the internet (for example, by letting some prepared USB sticks lying around in the lobby, it is not unlikely someone finds it and sticks it into their internal PC to check the content - guess what happened then). – Doc Brown Oct 08 '18 at 15:10
  • @Doc Brown, these servers are in the cloud, pls read the question carefully – Caner Oct 08 '18 at 15:22
  • @whatsisname, if internet facing servers are getting hacked/infected internal VMs will get hacked/infected too. Updating them wont help. – Caner Oct 08 '18 at 15:28
  • 3
    @Caner: well, I read your question carefully, but you missed to read my comment as what it was: an invitation to start thinking about the physical isolation of your system, and other potential attack areas than the "outside IP address". USB sticks is just an example (but from the real world). And if your servers were completely isolated from the outside world, they were probably not very useful to anyone. At some point in your private network, someone has probably potential access to them - if you can guarantee all those access points are 100% safe, then you won't need any updates. – Doc Brown Oct 08 '18 at 15:32
  • 2
    @Caner, not necessarily, the internal and external machines could be different OS's with different vulnerabilities – whatsisname Oct 08 '18 at 15:37
  • @whatsisname good point, but for our case they are the same – Caner Oct 08 '18 at 15:39
  • After your edit, this all boils down what is really happening behind those blue arrows in reality. – Doc Brown Oct 10 '18 at 08:41

2 Answers2

3

Yes, every VM connected to a network should be updated!

Let's flip your arguments around:

  • security breaches would create some downtime for our service
  • security breaches can break things
  • time spent cleaning up after a security breach can be spent somewhere else

Take a look at some recent security breaches and you'll see that in some cases the attackers spent months inside a network searching for weak links.

Imagine how excited they will be when they find oodles of unpatched VMs on the network, giving them lots of opportunities for exploits.

Only installing security patches is a decent start, but sometimes the best security patch is updating to a new version. Software vendors don't release security patches for old versions forever.

...patching the security hole was labor intensive and difficult, in part because it involved downloading an updated version of Struts and then using it to rebuild all apps that used older, buggy Struts versions.

Failure to patch two-month-old bug led to massive Equifax breach

Community
  • 1
Dan Wilson
  • 3,073
  • 1
  • 12
  • 16
  • please explain how hackers will have access to the network? if network is accessed by hacking internet-facing servers then internal VMs can also be hacked, updating them regularly won't help – Caner Oct 08 '18 at 15:30
  • Attackers access networks *by any means possible*, including external VMs and [security appliances](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-0101). – Dan Wilson Oct 08 '18 at 15:38
  • physically network is provided/protected by cloud provider. External VMs are of course always will be regularly software updated. – Caner Oct 08 '18 at 15:48
  • 3
    Using a cloud provider doesn't make your network more or less secure. AWS and Azure have teams of engineers and sophisticated tools, but they are also much larger targets of hacking attempts. "The cloud" is still just a rack of machines and appliances sitting in a data center that is connected to the Internet. Plus you are also vulnerable to a junior admin inadvertently making your internal VMs external because a business user really needs to access their app from a coffee shop. I have seen this happen more than once. You're one simple mistake away from exposing unpatched VMs to the world. – Dan Wilson Oct 08 '18 at 15:56
  • 1
    You don't know how many times I log into to AWS and see that "someone" has added 100+ security groups all with one IP address, and full, unblocked access. When I ask why managers and owners always say "Well, that's probably when I needed access for the hotel or something." – coteyr Oct 08 '18 at 16:49
  • 1
    +1, well written and concise. Hackers don't always have to come in through the internet also, even aside from your thumb-drive scenario there are plenty of cases where 'partner' companies coming in on trusted networks (Target) have been the attack vector, as have hackers who have gotten private network access (e.g. company laptop) or for that matter disgruntled employees of the company with legitimate access who decide to use that legitimate access for nefarious purposes. It's just head-in-the-sand thinking to believe you only have to protect your public facing servers. – Paul Oct 08 '18 at 17:03
  • how can I update instances without internet? – Caner Oct 11 '18 at 10:16
2

Updating properly does indeed take a large number of resources. You have to test it, apply it, and deal with any downtime, or issues the update may have caused. For the truly concerned you also need to check/know what issues are fixed and what new issues may be exposed. It can be very tempting to skip updating.

There is one case that I can think of where updating may not be worth it. That is when your updating short-lived worker instances, that are network isolated. A great example of this is an app I worked on. It would need to process images and video. When it was time to process video, it would spawn an instance, with the video file in a location, boot the instance, process, shove the file to another server and then turn off the instance, and delete it. The entire instance was up for maybe an hour a run. We never updated "those instances" and instead would update the image they came from. But even then we didn't update frequently, because the risk just wasn't there.

That said, normally any network (not just internet) connected machine should be updated. This is because any other machine on the network may have been infected. There are a lot of "worms" that do just that. They enter a protected network via a compromised machine then "go to town" because the other networked machines don't have any protections.

Another really good reason is that unless your really lucky (or unlucky) Your not the only key holder. Who knows what dumb, silly, and just plain stupid things the other guy will do. Remember the days of Windows XP. You would boot the machine up and because of this newfangled cable modem thing before you could boot you would get a virus or two. 1 idiot setting the wrong permissions for 30 seconds could totally bork your network.

Always wear your raincoat. The End.

coteyr
  • 2,420
  • 1
  • 12
  • 14