5

I was arguing this point with my boss the other day. He claims that the larger the code base the riskier the deployment.

I argued that this is not true, though I see why he might think that. In my experience the size of the code base doesn't matter if you follow solid principles and have good tests.

I tried to give examples of modern cars and how reliable they are even though there are thousands of moving pieces. This is because each piece does its job and only its job and it does it well.

Am I crazy, or is this just one of those counter intuitive things that some people have trouble accepting?

Kain0_0
  • 15,888
  • 16
  • 37
John S
  • 159
  • 2
  • 5
    Does the likelihood that every component was developed with solid principles and good tests *increase* or *decrease* with the number of components? – Jacob Raihle Dec 05 '19 at 11:15
  • 2
    See [Greg Wilson - What We Actually Know About Software Development, and Why We Believe It’s True](https://vimeo.com/9270320). In particular from minute 39 second 30. However, I still recommend the whole presentation. – Theraot Dec 05 '19 at 12:10
  • 2
    This sounds like one of those arguments that everyone and no one can win because "riskier" is a word that everyone thinks they agree on but really don't. In other words, when you say "riskier" what are you actually talking about ? Security? reliability? updatability? etc? You can make arguments for each one individually (and some in concert) but not all of them as a whole. – Matthew Dec 05 '19 at 12:57
  • 1
    What, exactly, do you mean by "riskier the deployment"? Do you mean the risk that you build the right thing? Risk in number of defects? Risk that the deployment will, for some reason fail? Something else? – Thomas Owens Dec 05 '19 at 13:45
  • @ThomasOwens I think he means not the act of the deployment process itself, but that a new release of a large project is riskier than a new release of a smaller project (i.e. change management, verification, validation, and actually running the new version without problems). If so, I agree with his boss. – Andres F. Dec 05 '19 at 15:16
  • 4
    @Theraot that video is interesting, thanks! For the lazy: the speaker references a study that shows that no metric predicts flaws in code better than simply counting lines. So lack/presence of comments, descriptive function names, modularization, etc, none are better correlated with bugs than simply doing "wc -l your_source_file". And this is backed by statistics! – Andres F. Dec 05 '19 at 15:24
  • Which is riskier to build, a skateboard or an aircraft carrier? – whatsisname Dec 05 '19 at 22:52
  • Can you do a Progressive rollout/deployment/release? I mean, start deploying to a small portion of users, monitor them closely, fix bugs. Then, after a while deploy to more users (I suggest double the ammount) and repeat until you have migrated all users. That is way to both start releasing now and handle the risk that a manager is likely to agree with. – Theraot Dec 06 '19 at 01:30

5 Answers5

7

When a typical software product grows in size over years, there will be definitely more places where bugs can hide, and occasions where more complex, unforeseen interactions can happen - this is where your boss is right.

On the other hand, countermeasures against degrading quality involve often even more code in form of

  • automated tests,

  • automated deployments,

  • more in-product validations,

  • improved configuration management,

  • code generators

  • redundant functionality

  • additional security layers

and so on. And you it also requires process improvements like code reviews, manual tests, documentation etc.

So the risk of a deployment (whatever that means) will to some degree depend on the size of the codebase, but not alone - it depends also on if your system has found the right balance between code which is necessary for certain features and code for supporting those other measures to keep the overall quality high.

"Following solid principles and have good tests" - as you wrote - is a good start, but far from being sufficient in the long run. The more features a system contains, the more important it will become to apply all known means of quality assurance and quality management.

So in short: both of the two perspectives you described contain some truth, but both look to me as if they are oversimplifying things.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
  • "The larger a system gets, the more important it will become to apply all known means of quality assurance and quality management." I would say this supports the assertion that larger systems are riskier, so the manager is right :) edit: in fact, I'd say your whole answer leans to the manager's side! – Andres F. Dec 05 '19 at 15:07
  • @AndresF.: they are both right and both wrong. But I changed my wording a bit, maybe that makes it clearer. – Doc Brown Dec 05 '19 at 15:08
  • Of course, you know best what you meant, but my reading of your answer makes me think you're saying the manager is *more right* ;) "Does not simply depend on the size", for example, implies it *does* depend on the size -- just that it's not the only dependency. – Andres F. Dec 05 '19 at 15:11
  • @AndresF.: I definitely disagree with the statement in the question *"the size of the code base doesn't matter"*, at least literally, even though it is accompanied by *"if you follow solid principles and have good tests."* – Doc Brown Dec 05 '19 at 15:14
  • 1
    On the part where the boss is right, I wouldn't use a "codebase size" metric, but rather a "how many things have been altered since the last succesful release" metric. Seems to be more correct, no? – Flater Dec 05 '19 at 23:10
  • @Flater That is not more correct if the code is fragile, or the change is a sensible part of the code. Then a small change could mean a lot of new bugs. Of course we would not want to have fragile code, however that is not being captured in the metric. – Theraot Dec 06 '19 at 01:08
  • @Theraot: The odds of a release succeeding correlate to the amount of _sources_ for a bug to be located in, not the amount of impact a given bug can have. Whether a bug creates one issue or one hundred - it's a failed release anyway. – Flater Dec 06 '19 at 09:35
  • @Flater: that is oversimplifying as well. Some changes have only local effects - there it is true what you said. Some, however, have clearly non-local effects, and some *look* as if they have only local effects, but it may turn out as wrong. In general, the potential of getting unwanted side-effects grows with the total size of the code base, even if only one line is changed. As I wrote, there are counter-measures to this effect, and the solution is usually not to write less code. – Doc Brown Dec 06 '19 at 09:44
  • @DocBrown: OP specifically claims that size is negated by SOLID principles and **good** tests. These tests will catch those issues, otherwise they're not good tests. And regardless of that, if a release failure is defined by whether it has (breaking) bugs or not - that depends on the _existence_ of the bugs, not the _scope_ of a given existing bug. – Flater Dec 06 '19 at 09:47
2

Let's say you have a 1% chance of any given module having a critical bug. If you have two modules that both must work in order for your system to work, you now have a 1.99% chance of your system failing. If you have three such modules, you now have a 2.97% chance of your system failing. That's basic reliability engineering.

I oversimplified, but the main takeaway is that the reliability of the system is worse than the reliability of each individual critical component, and gets worse with each additional critical component. If you want to increase the reliability of a system, you must specifically design for a failure of one component to not affect the system as a whole. For example, by running redundant services on physically separate machines.

Karl Bielefeldt
  • 146,727
  • 38
  • 279
  • 479
2

The Example is poor

A Modern car is complicated but highly reliable for the same reason that Lego can make complicated structures but still hold together reliably.

They are both restricted to 4D interactions.

There are of course a few more principles at play such as:

  • Well defined interfaces
  • Mass Produced
  • Quality Assurance of components

Compare that to software code. The 3rd, 400th, and possibly all lines between 2001- 69876 can influence variable A which then influences...

The dimensionality boggles the mind.

Now if you write code and restrict yourself to the low tens of dimensions, and implement other principles like: well defined interfaces, mass production, and quality assurance you can indeed approach levels of reliability found in cars.

In fact these systems exist, such as the flight avionics in planes. These are modular with well defined interfaces, each module is itself redundant with each duplicate produced independently by competing teams, and quality assurance is thrown at every component and assembly.

Risk

Your boss is correct in assuming that a larger code base is riskier, because that extra line can push the dimensionality of software higher. Higher dimensionality makes a larger result space that must be quality assured.

As there is only so much time in the world, only so much of that result space can be verified. The greater that difference the riskier that code base is, because it is less likely day to day operations will stay in the verified region.

You are correct in that applying good engineering practices allows you to reduce the internal dimensionality and as a result the size of the result space. It also allows you to verify regions of the result space much more reliably and in a more timely fashion.

I think you can also agree that the most reliable code, is the code that does not exist.

Community
  • 1
Kain0_0
  • 15,888
  • 16
  • 37
1

With two otherwise equal programs the larger one obviously has more risk.

But when do you ever have two equally well programmed programs?

What you are attempting to argue is that your large program has been created smarter than the average program and lacks a proportional amount of booboos.

Which is probably not a convincing argument to a manager. I suggest you argue for measuring the risk by recording deployments, changes, bugs, roll backs etc

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • I agree with your last statement and that is part of it. I dont think there is a cumulative effect in a robust mature design though and that is why I argue that size doesn't matter at least not nearly as much as other things – John S Dec 05 '19 at 18:58
  • 1
    So you believe releasing two small programs is as risky as one small program? – Ewan Dec 05 '19 at 19:09
1

The popularity of Microservices and the other answers here suggests that your boss is correct in "the typical case". Your argument seems to be that the typical case is not absolute and that exception software engineering can deliver extremely high-quality software in a single large code delivery.

If I were you boss I would smile and tell you that while I hope for exceptional software I plan on the occasional worst case of average software. If your boss wants you to spend more time breaking down your delivery into smaller incremental features to reduce risk then, well, they are the boss.

Your boss appears to be indicating that they want to see smaller releases or feature flags. You are then saying "Actually, no, as I am delivering superior engineering through A, B and C". This is probably not a great career move, as your boss is saying they think things are actually too risky, they want you to push out smaller releases, and you are making a general argument that they may be wrong in some cases.

The correct way to think about this scenario is not pondered "who has the correct view of software engineering?". The correct way to think about this is the cost/benefits to yourself. Consider that you take the advice or ignore it:

(A) you get to push out a big code drop, or

(B) you do as your boss says and break things up into smaller code drops

Then some possible outcomes at the next release:

  1. there is a bug that has a negative business outcome, which includes superficial reputation damage as well as actual financial damage.
  2. the code is great. no significant defects. no negative business outcomes.

There is likely to be no win for you if you ignore your boss:

  • {A,1} a big code drop and a negative business outcome. You ignored your boss and thing went bad. You should get fired.
  • {B,1} you did smaller changes. A subtle bug led to a really bad outcome. Your boss knows you did your best and took their advice. You don't get fired.
  • {B,2} you did as your boss asked. Nothing went bad. Win-Win.
  • {A,2} a big code drop and no negative business outcomes. Yet you have actually ignored your boss on matters of risk which is their responsibility. This is a Win-Lose. You are a gambler who now needs to win every time as the first time you slip up your boss is going to pound you.

My advice is that when it comes to risk assessments then it is the boss's view, where they may know of political risks they cannot share with you, that is paramount.

When I have been the boss in such cases where there I flag up risks and the developer pushes back it is usually because of developer fatigue:

Dev: "This code is ready to ship."

Me: "Hum, this looks risky, because of ..."

Dev: "Nah, I have had enough of this code and I want to ship it. Things will be okay because of X, Y, and maybe Z."

Me: "Let me tell you a story, I did this feature and things went bad because of ... but I didn't get fired as my boss and my team stood by me."

Dev: "Er. Well, if you really want me to do more work to {write tests, simply the code, add a feature toggle, break it up into smaller releases, change the error handling to fail-fast, use a library not my customer algorithm, do rigorous performance profiling ...} (delete as appropriate) then I can.

Me: "Thanks. I really appreciate it."

In general "I don't want to do more work" isn't really a great basis to argue with your boss ;-)

simbo1905
  • 442
  • 3
  • 7