32

Guava has unit test cases automatically generated:

Guava has staggering numbers of unit tests: as of July 2012, the guava-tests package includes over 286,000 individual test cases. Most of these are automatically generated, not written by hand, but Guava's test coverage is extremely thorough, especially for com.google.common.collect.

How they were generated? What techniques and technologies were used to design and generate them?

dzieciou
  • 650
  • 6
  • 19
  • I remember seeing a talk by some google dude that touched this subject. No clue about the name though, the talk was at some java convention I think – Zavior Nov 01 '12 at 15:08
  • 3
    package [com.google.common.collect.testing](http://code.google.com/p/guava-libraries/source/browse/guava-testlib/src/com/google/common/collect/testing/ "source code") has a lot of classes with "Generator" in their names - making it look like a framework for tests generation. There are also sub-packages with classes documented as "skeletons" or "base classes" for tests... – gnat Nov 01 '12 at 16:01
  • 1
    @gnat Yes, I was sure I've seen it somewhere. [com.google.common.collect.testing.features](http://code.google.com/p/guava-libraries/source/browse/guava-testlib/src/com/google/common/collect/testing/#testing%2Ffeatures) for instance shows tfeatures/constrains a collection class should satisfy, and a test case is a combination of them. This way they can **parametrize** testing – dzieciou Nov 01 '12 at 16:44
  • You may have more luck on [Software Quality Assurance and Testing.SX](http://sqa.stackexchange.com) – Martin Schröder Nov 02 '12 at 18:57
  • The question got much attention from the community but no reasonable answer so far, so I've followed Martin's suggestion and put it here: http://sqa.stackexchange.com/questions/5214/how-have-guava-unit-tests-been-generated-automatically. – dzieciou Nov 15 '12 at 08:30

2 Answers2

8

A large part of this mass of tests is for the Guava collection implementations. They've written generic tests that exhaustively test the collection interfaces, and this generates a suite per implementation. See, for example, classes called CollectionAddAllTester, ListIndexOfTester.

This is all backed by a library called testlib, which ships as part of Guava. This is quite generic. It supports writing generic tests for any interface (not just collections). You can specify Features of possible implementations and test those (e.g. if your set is unmodifiable you expect a different outcome from set.add()), and when you run the tests you specify which features your implementation supports.

It's based on JUnit 3, not 4. Normally, you have a class extending TestCase full of methods named testSomething(), and JUnit runs them reflectively. The testlib library hooks into the running of these tests so that the lifecycle looks like this:

  • For each implementation you want to test
  • For each (applicable) test method
  • Create the TestCase instance
  • Initialise the TestSubjectGenerator – this is the testlib interface that you extend where you actually create the test subject
  • Reflectively run the test method. During this method, getSubjectGenerator() gives access to the test subject

The key bit is the extra initialisation step that allows them to inject a specific test subject into the generic test case.

I wrote a post on how to write testlib generating suites for your own interfaces.

(Also posted to the same question on the sqa site.)

Community
  • 1
Joe Kearney
  • 196
  • 1
  • 4
6

There are unit test generators. For example, in the .NET world, something like Microsoft Pex could do this.

For example, Microsoft Pex tries based on code analysis all possible values as arguments for a method. Some arguments are expected to let the method throw an exception. Such things can automatically tests created for. Static values like an empty string that is returned in certain cases can also be automatically be tested.

Sebazzz
  • 253
  • 2
  • 7
  • 3
    That's a random testing useful only for negative path testing (exceptions, invalid input, crash-es, timeout). I believe they generated tests also for happy path, and this requires more design, not just launching static analysis tool. – dzieciou Nov 01 '12 at 13:18
  • And I know there are ways and tools for generating test for happy path (e.g., see this [answer](http://sqa.stackexchange.com/a/5029/1933)), but I'm interested how this was done in particular case of Guava – dzieciou Nov 01 '12 at 13:34