8

Forward

I've read a lot of things before asking this question, including many relevant questions right here on SE:

However, I can't help but feel that the itch hasn't been scratched yet after reading for help.

TL;DR

How do I write unit tests for legacy code that I can't run, simulate, read about, or easily understand? What regression tests are useful to a component that presumably works as intended?

The Whole Picture

I'm a returning summer intern again as I'm transitioning into grad school. My tasking involves these requirements:

  1. For a particular product, evaluate whether our software team can upgrade their IDE and JUnit version without losing compatibility with their existing projects.
  2. Develop unit tests for some component in the existing Java code (it's largely not Java). We want to convince the software team that unit testing and TDD are invaluable tools that they should be using. (There's currently 0% code coverage.)
  3. Somehow, end the days of cowboy coding for a critical system.

After obtaining a copy of the source code, I tried to build and run it, so that I might understand what this product does and how it works. I couldn't. I asked my supervisors how I do, and I was issued a new standalone machine capable of building it, including the build scripts that actually do. That didn't work either because as they should've expected, their production code only runs on the embedded system it's designed for. However, they have a simulator for this purpose, so they obtained the simulator and put it on this machine for me. The simulator didn't work either. Instead, I finally received a printout of a GUI for a particular screen. They also don't have code comments anywhere within the 700,000+ Java LOC, making it even harder to grasp. Furthermore, there were issues evaluating whether or not their projects were compatible with newer IDEs. Particularly, their code didn't load properly into the very IDE version they use.

My inventory is looking like this:

  • NetBeans 8, 9, 10, 11
  • JUnit 4, 5
  • Their source code for a particular product (includes 700,000+ Java LOC)
  • Virtually no code comments (occasionally a signature)
  • No existing tests
  • A physical photo of a GUI window
  • A software design document (109 p.) that doesn't discuss the component in the picture

I at least have enough to theoretically write tests that can execute. So, I tried a basic unit test on this said component. However, I couldn't initialize the objects that it had as dependencies, which included models, managers, and DB connections. I don't have much JUnit experience beyond basic unit testing, so follow me to the next section.

What I've Learned From My Reading

  1. Mocking: If I write a unit test, it likely needs to have mock variables for production dependencies that I can't easily initialize in setUp.
  2. Everyone here liberally suggests the book "Working Effectively with Legacy Code" by Michael Feathers.
  3. Regression tests are probably a good place to start. I don't think I have enough weaponry to attempt integration testing, and regression tests would provide more instant gratification to our software team. However, I don't have access to their known bugs; but, I could possibly ask.

And now an attempt to articulate the uncertainty I still have as a question. Essentially, I don't understand the how part of writing these tests. Assuming I don't receive any further guidance from my supervisors (likely), it's in my ballpark to not only learn what this component does but to decide what tests are actually useful as regression tests.

As professionals who've worked with projects like this longer than I have, can you offer any guidance on how to write unit tests in this kind of situation?

  • 14
    `How do I write unit tests for legacy code that I can't build, run, simulate, read about, or otherwise understand?` -- You can't. You have to at least know what the expected output is for a given input. – Robert Harvey Jun 19 '19 at 15:35
  • 2
    Have you read Michael Feathers' book yet? – Robert Harvey Jun 19 '19 at 15:38
  • @RobertHarvey It’s slight hyperbole. Obviously I can take the time to read code to understand it, but that’s typically a last resort for something so big. And no, I haven’t yet, since I just found the book this morning. –  Jun 19 '19 at 15:59
  • If you have legacy code that you can't even build, are you sure that you even have the right version of the code? You need to resolve this first before worrying about automated tests -- is the code that you're trying to build even the same as whatever your users are running? The first step in all of this if you did have a running and working system would be to try to understand it from the user's point of view; it can help to read user documentation and maybe look at user data if possible to try to figure out what it does – Ben Cottrell Jun 19 '19 at 16:01
  • @BenCottrell To make a minor distinction, it does build. It just can't run because it's not meant to run on a laptop. To make a further distinction, my employer is one of those three-letter agencies. Most questions that go like "well, if you had x, couldn't you do y?" end in "we don't do x here". I do have the most recent version of the source code, but being on a standalone machine, there's little hope of getting more resources than I already have. The user documentation is their design document. I don't have any "data" to look at either. –  Jun 19 '19 at 16:18
  • another somewhat related question: [Is it a good idea to write all possible test cases after transforming the team to TDD to achieve a full coverage?](https://softwareengineering.stackexchange.com/a/381998/134647) – Nick Alexeev Jun 19 '19 at 16:42
  • You have been given clearly impossible task. If you are unable to even run the code means you have no ability to change it. Really. Just give up and find better place that doesn't shit on it's employees. – Euphoric Jun 19 '19 at 17:03
  • "It just can't run because it's not meant to run on a laptop." This is a puzzling statement. Is there a check in the code that tests to see if the machine is a laptop or is it just simply an issue with the hardware not being capable of running the application? – JimmyJames Jun 19 '19 at 17:09
  • I appreciate the feedback so far, and would appreciate if someone would explain why they downvoted. I've been making edits in response to confusing language I used in my post. I can't change my tasking, but I can attempt to ask a better question if this is unclear in some way... –  Jun 19 '19 at 17:20
  • OK, it's still not clear why you can't run it. Isn't there an emulator available? – JimmyJames Jun 19 '19 at 17:24
  • @JimmyJames Sorry, I guess I don't understand why that point is so critical. Suppose the software is designed for a plane and relies on all of its sensor data---if those components aren't there, it doesn't run. As for the simulator they have, that doesn't run for some reason unbeknownst to me. It just didn't work when they brought it over to my computer (probably a configuration issue, and they didn't have the time or patience to fix it). –  Jun 19 '19 at 17:27
  • @JoshuaDetwiler "Suppose the software is designed for a plane and relies on all of its sensor data---if those components aren't there, it doesn't run" not sure if you read the news but a failure in a sensor causing software controlling a plane to fail is [a really bad thing](https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-developer). That aside, to be able to run unit tests, you will need to be able to execute at least some part of the code. – JimmyJames Jun 19 '19 at 17:34
  • 2
    If sensor data is crucial to execution, mocking the sensors (or whatever the external dependencies are) seems like a good place to start. – JimmyJames Jun 19 '19 at 17:35
  • @JimmyJames Fair point..that's probably a bad example, but that's the reason (and this product failing would have [a similar outcome](https://en.wikipedia.org/wiki/USS_Princeton_(1843)) too, honestly). The hardware it needs isn't there, so it can't run on a laptop. And as for execution, I hope to make use of mocks to separate that dependency. My hope is to eventually get a test that can at least create an object. –  Jun 19 '19 at 17:47
  • As pointed out by @RobertHarvey, Michael Feathers suggests [characterization testing](https://en.wikipedia.org/wiki/Characterization_test) in his book ["Working Effectively with Legacy Code"](https://amazon.de/Working-Effectively-Legacy-Robert-Martin/dp/0131177052) to deal with legacy code. However, if you can't even "build, run, simulate" the SUT, you are doomed… – beatngu13 Jun 19 '19 at 19:04
  • @JoshuaDetwiler it sounds to me like persevering with getting that simulator running is your best starting point. Do you have the source code for the simulator to be able to see why it doesn't work? There must surely have been a developer before you who had been able to run that and use it to develop the system, otherwise from what you say here I can see no way that the code could ever have possibly made it to its end-user. That would seem like the most likely way to be able to understand what the SUT is doing (and may even be useful for writing further automated tests too). – Ben Cottrell Jun 19 '19 at 19:45

3 Answers3

9

To a first approximation, the stakeholders of a test suite are the code developers/maintainers. You're going to need some of their time. Insist on this.

Ask them about problems they're facing.
Ask them about bugs they've fixed recently (in the past couple of years, assuming this is a long slow project).

I get the impression that you're not expecting them to be amiable to your work; maybe you're right. But I don't think you can build anything useful without their help.

They're going to hold your hand through writing the first test. Maybe you'll be dragging them, but you'll be holding hands either way.

Once you have one test, hopefully it'll be clearer how to write the second. If you've managed to build any kind of rapport, it will certainly be clearer.

ShapeOfMatter
  • 404
  • 3
  • 8
  • Thanks for the answer! Could you also take a look at the second half of my question too? My first half was a little duplicating of the questions I linked, and I was mainly interested in the other part of my question because of that. –  Jun 19 '19 at 19:04
  • I don't know how helpful I can be. Are you having trouble understanding how the test framework works? Are you wondering _what_ to test? For that, regression tests are a good choice: What test _would have prevented this bug we just fixed from getting pushed, if we'd been so forsightful_. – ShapeOfMatter Jun 19 '19 at 19:10
  • It was more "What regression tests are useful to a component that presumably works as intended?" Like, if I'm to create tests that are useful, what's useful being totally blind about what works, what doesn't, and how it even works at all? I exist in a vacuum apart from the software team and the people who gave me this task exist in a vacuum apart from both of us. –  Jun 19 '19 at 19:22
  • I'm still sure that you need to insist on getting in the same room as the the people writing the software, ideally for a couple of days, and probably more than once. I can relate to imposed absurd working situations, but at some point one either needs to risk one's job or accept that you're just a bench-warmer, I guess – ShapeOfMatter Jun 20 '19 at 00:54
  • Regarding Regression Testing: It's not conceptually different from any other testing: What would I want the program to do (or not do) before I claimed to a coworker that it was working? If there's records of new features or recent bugs, those are good places to start. Or just look through the documentation you have and pick something that looks testable. Java has typed functions, which is good. A clear type signature can tell you a lot about what to expect from a function (and can be thought of as a kind of unit test in itself). Checking NULL/empty-string/max-int behavior may also be good. – ShapeOfMatter Jun 20 '19 at 00:59
2

I'm going to assume at some point you can get the code to at least compile. If you can't even get that far you are on a fool's errand.

Lacking proper requirements, specifications, or screenshots is not a blocker for writing tests. As long as you can read source code, you can write tests.

If you are given permission to refactor the code base to isolate things like database connections behind their own interface, it becomes possible to write some black box unit tests — basically write tests to throw some input at a method and assert its behavior or output. Get tests covering each line of code in one method and then have one of the senior members of the team review your tests.

If you are not given permission to refactor the code base in order to write unit tests then full integration tests or UI automation tests are your only option. Even then, black box testing is your best strategy — throw some input at the UI and see how it reacts. Make your asserts. Have a senior member of the team review your tests.

At some point you will have enough automated tests that you can begin refactoring the code base with confidence to introduce unit tests. The UI tests ensure the main business use cases work, and then you can retrofit an architecture conducive to unit or component level testing.

Another benefit of UI tests is that you can build a reputation with your team that you understand the code base, which in turn makes them more open to you introducing change, because the proof is in the pudding. And you will have made pudding by way of writing passing tests.

In short:

  • Write Black Box tests as unit or automated UI tests
  • Have senior members review your tests

You would be surprised how quickly you can learn the big picture view of a 700,000 line application

Greg Burghardt
  • 34,276
  • 8
  • 63
  • 114
0

Based on the description of the issue and your comments, I think the best you can do is to start with the Java API and try to build a single unit test around an isolated method.

Without access to the code, I can only give you limited guidance but I would look for something that in it that a) has no dependencies b) makes no state changes. For example. let's say there's a method that takes a value and checks that it falls in a given range. If you can't find something with no dependencies, try something that retrieves a value from a dependency and attempt to mock it out.

Once you find something small like this, you can start building tests. If the method tests of a positive value, then pass it a negative and make sure it catches it etc. The problem here is that you may not know for sure what the correct behavior is. You may need to ask for help or dig through documentation for that.

You are unlikely to get very far with this. The reality is that writing code so that it can be unit-tested is an art to itself. Code that was not written with this in mind can be difficult-to-impossible to unit test.

Another option which may be more easily implemented is binary compatibility testing. That is, you capture the inputs and outputs of a system and then to test, you feed those same inputs and compare the outputs. This will not tell you whether the code is right or wrong to start with but it can help detect regression errors where things changed unintentionally when making some other modification. You will need to be able to run the entire application in order to make that happen, however.

JimmyJames
  • 24,682
  • 2
  • 50
  • 92