6

Code Exploration (CE) is quite a new term and I wonder if there already any successful examples of implementing this techniques in terms of Continuous Integration principles?

In short, Code Exploration can be described as the process of testing your code using different parameters to achieve high code coverage. The one-and-only tool for .NET world is PEX, but I know there are commercial instruments for code exploration in java world.

This topic is very powerful in conjunction with Design By Contract, because it provides you with the means of some formalism in reasoning about code correctness. For example PEX would show that next code contains error (source code uses Code Contracts and I assume default assembly configuration):

public int Factorial(int number)
{
    //postcondition of positive result, precondition is omitted 
    Contract.Ensures( Contract.Result<int>() > 0 ); 

    return number <= 0 ? 1 : number * Factorial( number - 1 );
}

Because by default overflow is not checked in arithmetic operations the input 17 will give the result of -288522240.

PEX was discussed here several times here:

But I don't want to touch internal implementation details, I'm interested in practical aspects of this problem.

Main questions are:

  1. How exactly do you integrate Code Exploration with Continuous Integration?

  2. Do you reject build if PEX gives errors or you notify team about inconsistencies and contracts violations?

  3. Scott Hanselman has point, that PEX should run on separate machine (or multiple machines) continuously testing your code for errors (so called Continuous Exploration). You can listen to the talk Hanselminutes Podcast 93 - Pex. Are there any practical implementations of dedicated machine (would it be virtual or physical), that digs into your codebase in search of errors?

  4. How do you treat test coverage of such tools? I think it would be unreasonable to merge code coverage results for unit tests written manually with the results by a code exploration tool.

  5. Are there any examples of Parameterized Unit Tests in real projects, that would increase you productivity and unit tests quality?

Community
  • 1
Ilya Ivanov
  • 121
  • 5

1 Answers1

2

Even if you have a boundary checking tool generate your unit tests, they're still just unit tests.

  1. Do you reject build if PEX gives errors or you notify team about inconsistencies and contracts violations?

Do you have gated check-ins turned on for your other unit tests?


This really isn't some new, distinct concept that needs us to make new rules to handle it. It's just being better about making the unit tests we should've been writing if we weren't too lazy or too naive about how corner cases impact code quality.

Telastyn
  • 108,850
  • 29
  • 239
  • 365
  • I did not mean to gate check-in, but to gate builds on you CI pipeline. If you work only in TDD - then you probably won't need dedicated CE, but what about big legacy code-bases? What if you already dealing with system with less than 30% code coverage and you want to refactor it? – Ilya Ivanov Aug 22 '12 at 13:19
  • @InnerSelf - Sorry, I don't see the distinction. You don't need TDD (which to me implies test-first coding) to actually have good unit tests. Big legacy codebases are orthogonal to these concepts. – Telastyn Aug 22 '12 at 13:32
  • I think, big untested legacy codebases aren't quite orthogonal to this topic. I'm also don't separate CE and casual unit testing. Just wonder if there where good benefits from CE systems, like PEX. – Ilya Ivanov Aug 22 '12 at 13:40
  • @InnerSelf big legacy codebases and big **untested** legacy codebases are different things. If your codebase doesn't have unit tests, then start making unit tests. No amount of automated test generation is going to fix that. – Telastyn Aug 22 '12 at 14:06