19

I'm wondering whether measuring conditional code coverage by current tools for Java are not obsolete since Java 8 came up. With Java 8's Optional and Stream we can often avoid code branches/loops, which makes it easy to get very high conditional coverage without testing all possible execution paths. Let's compare old Java code with Java 8 code :

Before Java 8:

public String getName(User user) {
    if (user != null) {
        if (user.getName() != null) {
            return user.getName();
        }
    }
    return "unknown";
}

There are 3 possible execution paths in the above method. In order to get 100% of conditional coverage we need to create 3 unit tests.

Java 8:

public String getName(User user) {
    return Optional.ofNullable(user)
                   .map(User::getName)
                   .orElse("unknown");
}

In this case, branches are hidden and we only need 1 test to get 100% coverage and it doesn't matter which case we will test. Though there are still the same 3 logical branches which should be covered I believe. I think that it makes conditional coverage statistics completely untrusted these days.

Does it make sense to measure conditional coverage for Java 8 code? Are there any other tools spotting undertested code?

Karol Lewandowski
  • 349
  • 2
  • 8
  • 5
    Coverage metrics have *never* been a good way to determine whether your code is well-tested, merely a way to determine what *hasn't* been tested. A good developer will think through the various cases in her mind, and devise tests for all of them -- or at least all that she thinks are important. – kdgregory Dec 01 '16 at 20:00
  • 3
    Of course high conditional coverage doesn't mean that we have good tests, but I think that it's HUGE advantage to know which execution paths are uncovered and this is what the question is mostly about. Without conditional coverage it's much harder to spot untested scenarios. Regarding paths: [user:null], [user:notnull, user.name:null], [user:notnull, user.name:notnull]. What I'm missing? – Karol Lewandowski Dec 01 '16 at 20:13
  • 6
    What's the contract of `getName`? It seems to be that if `user` is null, it should return "unknown". If `user` is not null and `user.getName()` is null, it should return "unknown". If `user` is not null and `user.getName()` is not null, it should return that. So you would unit-test those three cases because that's what the contract of `getName` is about. You seem to be doing it backward. You don't want to see the branches and write the tests according to those, you want to write your tests according to your contract, and ensure the contract is fullfilled. That's when you have good coverage. – Vincent Savard Dec 01 '16 at 20:29
  • Coverage is worth what's it's worth. You have the same issue with `int add(Integer i, Integer j) { return i + j }`. You can easily have 100% coverage by testing `add(new Integer(1), new Integer(2)) == 3`, but coverage still won't tell you that `add(new Integer(1), null)` fails. Actually, you just exposed the problem with coverage: just because you covered it with tests doesn't mean you tested it correctly, and relying on that lulls you into a false sense of confidence. – Vincent Savard Dec 01 '16 at 20:32
  • 1
    Again, I'm not saying that coverage proves my code is perfectly tested, but it was EXTREMELY valuable tool showing me what I haven't tested for sure. I think that testing contracts is inseparable from testing execution paths (your example is exceptional as it involves implicit language mechanism). If you haven't tested path, then you haven't fully tested contract or contract is not fully defined. – Karol Lewandowski Dec 01 '16 at 21:42
  • Before Java 8 I had little friend telling me what I missed in my tests and when Java 8 came my friend retired. Currently nothing supports me to find undertested contracts and I need to do it on my own which is not easy task. I always preferred to delegate work to tools/people that do it better than me and to have less things to care about. – Karol Lewandowski Dec 01 '16 at 21:44
  • It's not clear whether you are saying that the tools you used before for sure don't work or whether you are asking if they will still work. Most of these tools look at the bytecode and I don't think it was necessary to introduce new bytecode to support these features. Can you clarify? – JimmyJames Dec 01 '16 at 22:00
  • Wait, is is that there's only one line with multiple branches and these tools focus on lines, right? I get it. In theory, the branches are still traceable in the bytecode. The tool would have to be updated to support showing that say only 2/3 branches in that line were covered. Is that it? – JimmyJames Dec 01 '16 at 22:02
  • I'm saying that there are no tools (to my knowledge) providing me with information what I missed in my tests. Existing tools base on "technical" branches like if/case/loops and don't care about lambdas cooperating with new Java 8 objects what reduces most of old null-checks. I know that it's possible to make them provide the same information again (by modifying bytecode to put "checkpoints" in lambdas), but if it hasn't been done yet, I doubt anyone plans to do it (and I can't understand why). – Karol Lewandowski Dec 01 '16 at 22:20
  • I am voting to close this question in its current format, because it's a request for recommended tools/libraries to measure logical branches. – Andy Dec 02 '16 at 14:09
  • 2
    I'm going to repeat my earlier point: that's **always** the case, unless you limit yourself only to basic language features and never call any function that hasn't been instrumented. **Which means no third-party libraries, and no use of the SDK.** – kdgregory Dec 02 '16 at 14:10

1 Answers1

4

Are there any tools that measure logical branches that can be created in Java 8?

I'm not aware of any. I tried running the code you have through JaCoCo (aka EclEmma) just to be sure, but it shows 0 branches in the Optional version. I don't know of any method of configuring it to say otherwise. If you configured it to also include JDK files, it would theoretically show branches in Optional, but I think it would be silly to start verifying JDK code. You just have to assume it's correct.

I think the core issue, though, is realizing that the additional branches you had prior to Java 8 were, in a sense, artificially created branches. That they no longer exist in Java 8 just means you now have the right tool for the job (in this case, Optional). In the pre-Java 8 code you had to write extra unit tests so that you could have confidence that each branch of code behaves in an acceptable way - and this becomes a bit more important in sections of code that aren't trivial like the User/getName example.

In the Java 8 code, you're instead placing your confidence in the JDK that the code works properly. As is, you should treat that Optional line just as code coverage tools treat it: 3 lines with 0 branches. That there are other lines and branches in the code below is something you just haven't paid attention to before, but has existed every time you've used something like an ArrayList or HashMap.

Shaz
  • 2,614
  • 1
  • 12
  • 14
  • 2
    "That they no longer exist in Java 8..." - I can't agree with it, Java 8 is backward compatible and `if` and `null` are still parts of the language ;-) It's still possible to write code in old way and to pass `null` user or user with `null` name. Your tests should just prove that contract is met regardless of how method is implemented. The point is that there is no tool to tell you if you fully tested contract. – Karol Lewandowski Dec 01 '16 at 22:32
  • 1
    @KarolLewandowski I think what Shaz is saying is that if you trust how `Optional` (and related methods) work, you no longer have to test them. Not in the same way you tested an `if-else`: every `if` was a potential minefield. `Optional` and similar functional idioms are already coded and guaranteed not to trip you over, so essentially there's a "branch" that vanished. – Andres F. Dec 01 '16 at 23:22
  • 1
    @AndresF. I don't think Karol is suggesting that we test `Optional`. Like he said, logically we should still test that `getName()` handles various possible inputs in the way we intend, regardless of its implementation. It's harder to determine this without code coverage tooling helping in the way it would pre-JDK8. – Mike Partridge Dec 02 '16 at 13:33
  • 1
    @MikePartridge Yes, but the point is that this isn't done via branch coverage. Branch coverage is needed when writing `if-else` because each of those constructs is completely ad-hoc. In contrast, `Optional`, `orElse`, `map`, etc, are all already tested. The branches, in effect, "vanish" when you use more powerful idioms. – Andres F. Dec 02 '16 at 15:13