0

In my company, there is a KPI about code coverage. We measure results by code coverage percentage.

So we combine code coverage result of unit test and end-to-end test together.
Does it make sense to do this?

BobDalgleish
  • 4,644
  • 5
  • 18
  • 23
Sarawut Positwinyu
  • 511
  • 6
  • 16
  • 3
    That depends on what you do with the code coverage information. If it just to satisfy a useless KPI, then anything goes. – Bart van Ingen Schenau Oct 12 '17 at 10:56
  • As @BartvanIngenSchenau stated, code coverage is a senseless KPI. The much more valuable information is *requirement coverage*. Unfortunately we have no tool to measure that. The only way to keep it high is TDD... – Timothy Truckle Oct 12 '17 at 11:53
  • @TimothyTruckle: That seems a bit fatalist. Something is only senseless if it has no value at all, and I certainly don't believe that TDD is the only way to keep "requirement coverage" high. – Robert Harvey Oct 12 '17 at 15:42
  • @TimothyTruckle code coverage and requirement coverage are both useful and show different things imho. certainly for safety related stuff we use both – jk. Jan 30 '20 at 18:15
  • @jk. yes, code coverage has a value for estimating the *thrustworthyness* of the safety net the uniottests provide, but it does not make sense as a KPI. The reason is that if you define a minimum code coverage as a KPI you end up with bad unittest that do not verify desired behavior (as defined by the requirements) but just raise the code coverage. Developers later have a hard time to find out which UTs can safely be ignored and which are actual protecting existing desired behavior. – Timothy Truckle Jan 31 '20 at 08:10

2 Answers2

0

I think it does but with a few caveats:

Obviously you need to combine the metrics properly, not just add the coverage percentages. Usually there are tools to do this e.g. https://www.eclemma.org/jacoco/trunk/doc/merge-mojo.html

As with any metric you need to be careful how it is used. You probably don't want to look at just the merged value, the separate unit coverage and integration coverage is also potentially useful.

Unless you are required to meet a safety standard then aiming for 100% is probably pointless, the absolute number of the coverage may also be pointless but it may be useful to e.g. look at relative changes between builds.

jk.
  • 10,216
  • 1
  • 33
  • 43
-3

No, because you would get a result more than 100% which would be non nonsensical.

The correct way to combine them would be to check the percentage of code covered by Both e2e and unit tests. But again, this isn't a very useful number compared to a break down. eg:

  • unit test coverage 60%
  • integration test coverage 30%
  • functional test coverage 10%
Ewan
  • 70,664
  • 5
  • 76
  • 161
  • 3
    I combined coverage results many times but never had it over 100%. It was always "merge" kind when repeated coverage doesn't add up. Wonder what kind of tool or process one has to use to get senseless combined result over 100% – gnat Oct 12 '17 at 10:47
  • i wonder whether your 'merge' was AND or OR – Ewan Oct 12 '17 at 10:52
  • 1
    I assume that one would combine these two metrics in a way that didn't create coverage over 100% (by adding the weighted averages, for example). – Robert Harvey Oct 12 '17 at 15:41
  • Jacoco has a tool specifically to do this properly, I'm sure other coverage tools also do similar things. obviously you can't just naively add the resulting percentages. – jk. Jan 30 '20 at 18:12