Should usage examples in XML documentation be explicitly tested?

Question

If usage examples are provided in XML documentation, should those examples be explicitly tested?

/// <summary>
/// Gets a collection of Foo objects. 
/// </summary>
/// <param name="bar">The model database connection.</param>
/// <example>Example code for using this method.
/// <codelang = "vb">
/// Dim bar As New Bar()
/// Dim foos As ReadOnlyCollection Of Foo) = helperClass.GetFooFromBar(bar)
/// </code>
/// </example>
public ReadOnlyCollection<Foo> GetFooFromBar(Bar bar)

Granted, the above contrived example is probably not at much risk of regressing, but it would be very embarrassing if API users pasted the example code and it did not work.

Presuming that the consensus opinion is that these code examples should be tested, the follow-up question is How can these types of tests be automated to eliminate developer error as much as possible

I wonder if it's possible to add a compiler task or something to add an `` doc command. — Zymus, Jul 25 '17 at 17:42
Or you could generate the snippet from real unit tests. Can C# generate dynamic documentation? — Laiv, Jul 25 '17 at 20:16

score 2 · Answer 1 · answered Jul 25 '17 at 21:28

Personally, yes, I believe, they should be tested. A lot of people say that one of the bad things about documentation comments is that they are not checked by the compiler. Well, here you have at least a portion of your documentation comment which can be checked by a testing framework, so do it!

There are some interesting directions in which this idea can be developed: the Python community has a library called doctest, which can automatically extract usage examples (which in turn are (or look like) just copy&pasted REPL sessions) from documentation and turn them into unit tests.

And a more extreme example is the book Agile Web Development with Ruby on Rails, which is an entire book (not just a small documentation comment), where every code example is fully executable. And it is executable in two modes: you can either run the book against a version of Rails, and the framework will ensure that all output is the same as the one printed in the book; in this mode, the book acts as a regression suite (and informal spec) for Rails. And in the other mode, you can have the framework automatically update the results printed in the book to match the ones from Rails. So, you can either use the code examples to ensure that Rails doesn't change in backwards-incompatible ways, or you can use it to ensure that the examples in the book are up-to-date with backwards-incompatible changes in Rails. IOW: the framework can automatically catch all differences for you and you can consciously decide for each one whether to change Rails to match the book (in which case you already have a failing test case to fix) or change the book to match Rails (and the framework can do the latter for you).

So, if you execute your code examples in your documentation, you will catch cases where the documentation no longer matches the code, and you can decide whether that was an intentional change (and update the documentation accordingly, even automatically given the right tools), a bug in the documentation, or a bug in the code (in which case the example already acts as a failing test case).

Should usage examples in XML documentation be explicitly tested?

1 Answers1