2

I've worked on a few codebases without a great automated test suite, where changes that concern the whole platform have to be thoroughly tested by developers and there would be a high risk that a degradation is introduced with each release. How do you prevent this from happening?

The less than perfect answers from my experience are:

  • Test thoroughly yourself - completely unsustainable on large projects
  • Organise group testing sessions - can be effective, but a bit chaotic and a pain in the ass for everyone involved
  • Manual release QA tests - soul sucking affair where some poor sod has to trawl through a spreadsheet of manual tests, limited by their knowledge of the platform and existing bugs

What would be yours, save for improving the testing suite or hiring people whose job is QA?

styke
  • 145
  • 3
  • 1
    Very careful and through code reviews. – JacquesB May 19 '21 at 18:45
  • Unfortunately the two things you don't want to do (improve the test suite and hire a QA tester) are precisely the things you need to do. In fact, I would argue for hiring a dedicated QA tester even if you have good test automation. Testers think differently than developers, and will catch errors that developers don't think about. – Greg Burghardt May 20 '21 at 16:15

2 Answers2

6

Tests prevent regressions. There are other tools which would prevent regressions, but unfortunately they are either less effective (example: code reviews), or extremely expensive (example: formal proofs), which leaves us with tests as the best option.

So you find yourself in front of a codebase which should be tested, but is not tested yet, or not enough, and you have to make a change. The solution to prevent regressions is to progressively increase the test coverage.

When you touch a small part of the codebase, check if it's tested. If you are confident enough that most if not all the logic is tested, that's great. If not, start by adding tests. The great thing is while you do it, you will:

  • Understand better the part that you wanted to modify.
  • Have a better view of how well this part of the code matches the business requirements.
  • Find a bunch of errors in the logic of the code, such as the branches which can never be reached.
  • Get a clear picture of the change you were expected to do.
  • Imagine some possible refactoring to do.

The elements in bold would reduce the risk of causing regressions by a change. Most regressions come from changes where developers misunderstood the code they were changing, or misunderstood the change itself.

Once you tested it:

  1. Write the test which will check for the change you're about to do.
  2. Do your change.
  3. Re-run the tests.
  4. Refactor.
  5. Re-run the tests.
  6. Release.

If you're not under time pressure:

  • Make additional effort to refactor more, if needed. Sometimes, there are opportunities to simplify the code tremendously, eventually replacing dozens of lines of code by a call to a method, or removing a duplication, or making a challenging algorithm easy enough for a junior programmer to understand. Since the code is tested now, you can refactor aggressively, without the risk of creating regressions while doing so.
  • If during testing, you noticed discrepancies between code and business requirements, go see the product owner.
  • If you still have free time, test code that surrounds the piece you already tested, especially if it allows to expand the refactoring further on.

Make sure you also read I've inherited 200K lines of spaghetti code — what now?

Arseni Mourzenko
  • 134,780
  • 31
  • 343
  • 513
  • Thanks for the comment, especially the link is extremely insightful. I largely agree with your assertion about the benefits of tests - the issue is that sometimes testing is not an option for x or y reason (usually pressure from product or technical blockers) - it is for these cases that I am looking for a short term fix so that people's lives are not spent in misery until the team can get around to writing tests. – styke May 19 '21 at 21:53
  • 3
    You're in a plane and your flight is delayed because of a safety issue. You're in a hurry: in three hours, you have to be in LA for the important meeting. Would you prefer the pilots to take off and see if the plane crashes or not, or you prefer to wait a bit until the safety issue is resolved? Software development is exactly the same. It belongs to *you* to explain to non-technical persons that you can release soon and the app may crash, or you may take the time you need and do quality work. If the stakeholders decide that they want to take the risk, so be it. But at least be clear with them. – Arseni Mourzenko May 19 '21 at 22:02
  • Also see [an article](https://blog.pelicandd.com/article/187) I wrote on the subject. – Arseni Mourzenko May 19 '21 at 23:39
  • @styke: To sum up Arseni's first comment, never tell them "no" but *always* inform them of the consequences. And if you aren't comfortable with the consequences, you can always find another job. Don't put yourself in a position to be their scapegoat. – Greg Burghardt May 20 '21 at 16:19
  • I wouldn't even bother informing the stakeholders about the consequences, nor give them an opportunity to decide whether I should test my code or not, *unless* I have a very good reason to think that taking risks by skipping tests could make business sense. It doesn't belong to non-technical persons to decide *how* I should do my job: otherwise, they would be writing code in my place. It is essentially exactly the same thing as a pilot who would delay a flight for safety reasons, disregarding what the passengers want or think being the best option for them. – Arseni Mourzenko May 20 '21 at 23:54
  • I would just like to point out that while I agree with the importance of writing tests, I have been in situations where it was actually unfeasible from a business point of view to write them. Like you suggested, all I could do was say that our options where to either address the issues preventing us from writing those tests or not write those tests at all. Also while I like your analogy, writing software is very different from flying a plane and I don’t think the same level of importance applies for all cases. Unless of course you’re writing software for flying a plane. – styke May 21 '21 at 18:05
  • @styke: I would like to hear more about those unfeasible situations (if you want to drop me a note in private, my email address is in my profile). Regarding the analogy, it's not about writing software vs. flying a plane: it's about, in both cases, following procedures, taking risks, taking difficult decisions, and not delegating responsibility to someone else. Also, when a multi-million projects fails after a release, granted, it's not the same as a plane crash, but it's not an enjoyable situation either. – Arseni Mourzenko May 21 '21 at 18:23
1

Writing tests really is the way to go. but assuming you can't I would go with "more logging"

If you are logging all your errors then you can tell if a release increases or decreases the number of errors of various types.

If a feature release decreases errors, you are good. If it increases you have introduced a new problem.

Coupled with a canary release, where you release a new version to a subset of users you should at least be able to gauge if the new release is better than the last, even if you can't tell if it's working correctly or not.

Ewan
  • 70,664
  • 5
  • 76
  • 161