3

Working on Plone projects our team tries to achieve full test coverage at least for important products. The kind of tests we write are unit tests, functional tests and integration tests. (Also stress-tests, but those aren't in the scope of this question). The goal is two fold at least: to facilitate upgrades and to catch bugs (sometimes it even happens in the process of writing tests).

However, Plone/Zope is a complex system, and with years of experience I've noticed, that test strategy should be re-thought. First of all, unit tests, which often require to use a lot of mocking, are not that (cost)efficient. They are mostly easy and beneficial when some core, logic-heavy functionality is being written, like pure Python modules, which have negligible couplings with Plone/Zope, databases, etc. I rarely seen unit-tests catching any bugs at all, except while writing them.

So, when doing the usual thing (writing views/portlets/viewlets), from my experience, it's much more efficient to write functional and integration tests. The rationale of it is that in case Plone/Zope changes (in an upgrade) we can catch the mishaps in our code. Views usually do not have a lot of "algorithmic" logic, they glue together several data sources (like catalog queries), maybe with some form handling and preprocessing for templates. It is quite often views call one or more tools to do some routine job (like getting navigation tree or looking up site root). Mocking it all seems unwise. For example, sometimes Plone/Zope changes some default to another type and all those mocked tests happily continue to work while code fails in the real instance.

Functional/Integration tests may be at times fragile (HTML can change), but they are cheaper to produce too. They provide basic coverage, and trigger alarms when underlying system changes. Spotting the source of mishap is usually not an issue. (update: Wrong: spotting where integration test fails can be a big issue, but our code's unit tests are usually of no help.)

The question is, am I overlooking something of importance confining unittesting to functions and classes, which do not require mocking the environment heavily and are instead "purely" logic-heavy? I can't find any justification for writing unit-test "first", or even at all, for every piece of code in Plone/Zope (I do not mean the core of the system, just our own additions for our clients).

To make the question less opinion based, are there other reasons, I have not mentioned or tackled above, which necessiate to consider writing more unit tests (and less integration tests) in a situation when code heavily depends on a complex (and somewhat hairy) framework, and code serves more as a glue for framework's subsystems?

Roman Susi
  • 1,763
  • 2
  • 12
  • 20
  • 1
    possible duplicate of [When is unit testing inappropriate or unnecessary?](http://programmers.stackexchange.com/questions/147055/when-is-unit-testing-inappropriate-or-unnecessary) – gnat Sep 29 '13 at 19:43
  • 1
    @gnat Thanks for pointing related questions. I do not think my question is a duplicate, because my question describes much more concrete situation and different angle to the problem, as I am asking about shifting from unit to func/integration tests. It can't be deduced from the question you cite. – Roman Susi Sep 29 '13 at 19:59

1 Answers1

2

I rarely seen unit-tests catching any bugs at all, except while writing them.

Well that's one good reason to keep writing unit tests - unit tests are there not just for regression testing but to help guide the design of your system. If unit tests can help you notice bugs in the code under test which you can then fix on the fly, that's real value right there

It is quite often views call one or more tools to do some routine job (like getting navigation tree or looking up site root).

And there's two more reasons - behavioural verification and regression testing - knowing with 100% certainty that View X did make a call out to tool Y and did pass the result to the widget processor therefore the reason why widget Z is not rendering, must be the widget processor (or whatever) and you didn't even have to use the debugger to fix it!

Spotting the source of mishap is usually not an issue.

A very debatable and admittedly subjective (with my tongue firmly in my cheek!) "benefit" or at least "Food for thought" I hope, starts with : can you define "usually"?

In the "unusual" cases where you don't instantly spot the source of the problem, because it's 2 a.m and your eyes are bleeding from the glare of the monitor...how long do you really spend chasing down problems? Have you measured how much time it might be taking you in these cases vs writing tests up front that will pinpoint the problem for you - if you analyse this and find a net gain in time by TDD up front, then I say that's your bottom line.

Of course if analysis shows that in your case it really does not save you time, then there's a proven, good justification for not doing it - you can't lose!

Stephen Byrne
  • 695
  • 1
  • 5
  • 10
  • 1) I am not going to drop writing unit tests. My point is, unit tests specifically for views is wasting time because complex subsystems view ties together need to be mocked. Unit test which verifies something is actually called has little value, because one need to change the view to achieve the result! Unit test is just writing code in other words in that case. – Roman Susi Oct 01 '13 at 17:03
  • 1
    3) I mean, when integration test fails, it is usually not because of the code under test but some regression in the system. The system is much much larger in volume than the views and whatnot we write (the system is where most mishaps happen). And our unittests unfortunately of no help to pinpoint problems. Integration tests work much better. In one blow they catch errors in the usual flow of usage. (low-level, "leaf" functions still have unit tests - I find it very useful to write unit tests for those before/during/right after development) – Roman Susi Oct 01 '13 at 17:10
  • Thanks! I have corrected myself regarding spotting the source of problem. – Roman Susi Oct 01 '13 at 17:19
  • 1
    Cool beans - it's really good that you're questioning the value of these things, because only then can you really step back and come to an honest answer :) – Stephen Byrne Oct 01 '13 at 18:29