16

I follow TDD religiously. My projects typically have 85% or better test coverage, with meaningful test cases.

I do a lot of work with HBase, and the main client interface, HTable, is a real pain to mock. It takes me 3 or 4 times longer to write my unit tests than it does to write tests that use a live endpoint.

I know that, philosophically, tests that use mocks should take priority over tests that use a live endpoint. But mocking HTable is a serious pain, and I'm not really sure it offers much of an advantage over testing against a live HBase instance.

Everyone on my team runs a single-node HBase instance on their workstation, and we have single-node HBase instances running on our Jenkins boxes, so it's not an issue of availability. The live endpoint tests obviously take longer to run than the tests that use mocks, but we don't really care about that.

Right now, I write live endpoint tests AND mock-based tests for all my classes. I'd love to ditch the mocks, but I don't want quality to decline as a result.

What do you all think?

k3b
  • 7,488
  • 1
  • 18
  • 31
sangfroid
  • 3,219
  • 4
  • 17
  • 12
  • 8
    Live endpoint isn't really a unit test, is it? It's an integration test. But ultimately it's probably a question of pragmatism; you can either spend the time writing mocks, or spend the time writing features or fixing bugs. – Robert Harvey Mar 18 '14 at 20:22
  • 4
    I've heard stories of people taking down 3rd party services by running a unit test against their own code... that was hooked up to a live endpoint. Rate limiting isn't something that unit testing typically does or care about. –  Mar 18 '14 at 20:56
  • 14
    You're not a bad person. You're a good person doing a bad thing. – Kyralessa Mar 18 '14 at 20:58
  • 15
    *I follow TDD religiously* Maybe that's the problem? I don't think any of this methodology is meant to be taken *that* seriously. ;) – FrustratedWithFormsDesigner Mar 18 '14 at 21:42
  • @FrustratedWithFormsDesigner: I agree that it could be better to avoid any kind of excess. – Revious Mar 18 '14 at 21:55
  • 9
    Following TDD religiously would mean that you discard the 15% uncovered code. – mouviciel Mar 19 '14 at 08:05
  • Yes. Yes you are. – Andy Mar 24 '14 at 22:44

6 Answers6

23

  • My first recommendation would be to not mock types you don't own. You mentioned HTable being a real pain to mock - maybe you should wrap it instead in an Adapter that exposes the 20% of HTable's features you need, and mock the wrapper where needed.

  • That being said, let's assume we're talking about types you all own. If your mock-based tests are focused on happy path scenarios where everything goes smoothly, you won't lose anything ditching them because your integration tests are probably already testing the exact same paths.

    However, isolated tests become interesting when you start thinking about how your system under test should react to every little thing that could happen as defined in its collaborator's contract, regardless of the actual concrete object it's talking to. That's part of what some call basic correctness. There could be many of those little cases and many more combinations of them. This is where integration tests start getting lousy while isolated tests will remain fast and manageable.

    To be more concrete, what happens if one of your HTable adapter's methods returns an empty list ? What if it returns null ? What if it throws a connection exception ? It should be defined in the Adapter's contract if any of those things could happen, and any of its consumers should be prepared to deal with these situations, hence the need for tests for them.

To sum up : you won't see any quality decline by removing your mock-based tests if they tested the exact same things as your integration tests. However, trying to imagine additional isolated tests (and contract tests) can help you think out your interfaces/contracts extensively and increase quality by tackling defects that would have been hard to think about and/or slow to test with integration tests.

guillaume31
  • 8,358
  • 22
  • 33
  • +1 I find it a lot easier to construct edge cases with mock tests than populating the database with those cases. – sea-rob Mar 19 '14 at 17:43
  • I agree with most of your answer. However, I'm not sure I agree about the Adapter part. HTable is a pain to mock because it's fairly bare-metal. For example, if you want to do a batch get operation, you must create a bunch of Get objects, put them in a list, then call HTable.batch(). From a mocking perspective, this is a serious pain, because you have to create a custom Matcher that examines the list of get objects you pass to HTable.batch() and then returns the correct results for that list of get() objects. A SERIOUS pain. – sangfroid Mar 25 '14 at 22:37
  • I suppose I could create a nice, friendly wrapper class for HTable that took care of all that housekeeping, but at that point ... I feel like I'm kinda building a framework around HTable, and should that really be my job? Usually "Let's build a framework!" is a sign I'm going in the wrong direction. I could spend days writing classes to make HBase more friendly, and I don't know if that's a great use of my time. Plus, then I'm paasing around an interface or wrapper instead of just the plain old HTable object, and that would definitely make my code more complex. – sangfroid Mar 25 '14 at 22:40
  • But I agree on your main point, that one shouldn't write mocks for classes they don't own. And definitely agree there's no point in writing a mock test that tests the same thing as an integration test. It would seem mocks are best for testing interfaces/contracts. Thanks for the advice -- this has helped me a lot! – sangfroid Mar 25 '14 at 22:42
  • I have little idea what HTable really does and how you use it, so don't take my wrapper example to the letter. I had mentioned a wrapper/adapter because I thought the thing to wrap was relatively small. You don't have to introduce a one-to-one replica to HTable, which would of course be a pain, let alone a whole framework - but you need a *seam*, an interface between the realm of your application and that of HTable. It should rephrase some of HTable's functionality into your application's own terms. The Repository pattern is a perfect incarnation of such a seam when it comes to data access. – guillaume31 Mar 26 '14 at 08:30
11

philosophically, tests that use mocks should take priority over tests that use a live endpoint

I think at the very least, that's a point of current ongoing controversy amongst TDD proponents.

My personal view goes beyond that to say that a mock-based test is mostly a way of representing a form of interface contract; ideally it breaks (i.e fails) if and only if you change the interface. And as such, in a reasonably strongly typed languages like Java, and when using an explicitly-defined interface, it is almost entirely superfluous: the compiler will already have told you if you have changed the interface.

The main exception is when you are using a very generic interface, perhaps based on annotations or reflection, that the compiler isn't able to usefully police automatically. Even then you should check to see if there is a way of doing validation programatically (e.q. a SQL syntax checking library) rather than by hand using mocks.

It is that latter case you are doing when you test with a 'live' local database; the htable implementation kicks in and applies much more comprehensive validation of the interfacve contract than you would ever think to write out by hand.

Unfortunately, a much more common use of mock-based testing is is the test that:

  • passes for whatever the code was at the time the test was written
  • provides no guarantees about any properties of the code other than that it exists and kind of runs
  • fails any time you change that code

Such tests should of course be deleted on sight.

soru
  • 3,625
  • 23
  • 15
  • 1
    I can't back this up enough. I'd rather have 1pc coverage with great tests than 100pc coverage of filler. – Ian Mar 18 '14 at 23:20
  • 3
    Mock-based tests indeed describe the contract used by 2 objects to talk together, but they go far beyond what the type system of a language like Java can do. It's not just about method signatures, they can also specify valid ranges of values for arguments or returned results, which exceptions are acceptable, in what order and how many times methods can be called, etc. The compiler alone won't warn you if there are changes in those. In that sense I don't think they are superfluous at all. See http://www.infoq.com/presentations/integration-tests-scam for more info about mock-based tests. – guillaume31 Mar 19 '14 at 15:52
  • 1
    ...agree i.e. testing the logic around the interface call – sea-rob Mar 19 '14 at 17:41
  • 1
    Can certainly add unchecked exceptions, undeclared preconditions and implicit state, to the list of things that make an interface less statically-typed, and so justify mock-based testing instead of simple compilation. The problem, however, is that when those aspects _do_ change, their specification is implicit and distributed amongst the tests of all clients. Which are likely to not get updated, and so sit there silently hiding a bug behind a green tick. – soru Mar 19 '14 at 21:11
  • "their specification is implicit" : not if you write contract tests for your interfaces (http://blog.thecodewhisperer.com/2011/07/07/contract-tests-an-example/) and stick to them when setting up mocks. – guillaume31 Mar 20 '14 at 14:36
  • I agree it's not a systematic red tick when they change though - it requires a little discipline to keep contract tests and mock-based tests in sync. Some tools like Bogus in Ruby (https://www.relishapp.com/bogus/bogus/docs/contract-tests) can help you do that. – guillaume31 Mar 20 '14 at 14:43
  • Contract tests as described at that link aren't mocks; they are fakes. Which is closer to what the OP is doing by using a local instance of HBase instead of the real thing. A fake-generation tool as good as mockito would be a useful addition to everyone's test toolkit. – soru Mar 20 '14 at 14:54
  • @soru "The problem, however, is that when those aspects do change, their specification is implicit and distributed amongst the tests of all clients. Which are likely to not get updated, and so sit there silently hiding a bug behind a green tick". I know what you're getting at but tests can never prove the absence of defects. The benefit of this testing then is that when the defect finally does surface you have an easy place to setup the specific case that causes failure, prove the failure, fix it and show its now fixed. You also have a record of how the system should behave. – Andy Mar 24 '14 at 22:51
5

How much longer does an endpoint-based test take to run than a mock-based test? If it's significantly longer, then yes, it's worth the investment of your test-writing time to make the unit tests quicker - because you'll have to run them many, many times. If it's not significantly longer, even though the endpoint-based tests are not "pure" unit tests, as long as they're doing a good job of testing the unit, there's no reason to be religious about it.

Carl Manaster
  • 4,173
  • 18
  • 31
4

I agree completely with the response of guillaume31, never mock types that you don't own!.

Normally a pain in the test (mocking a complex interface) reflect a problem in your design. Perhaps you need some abstraction between your model and your data access code, form example using an hexagonal architecture and a repository pattern its the most usual way to solve this kind of problems.

If you want to do a integration test for checking things do a integration test, if you want to do a unit test because you are testing your logic do a unit test and isolate de persistence. But doing an integration test because you don't know how to isolate your logic from a external system (or because isolating its a pain) its a big smell, you are choosing integration over unit for a limitation in your design not for a real need to test integration.

Take a look to this talk form Ian cooper: http://vimeo.com/68375232 , he talks about hexagonal architecture and testing, he talks about when and what to mock, a really inspired talk that solves many questions like yours about real TDD.

AlfredoCasado
  • 2,159
  • 11
  • 11
1

TL;DR - The way I see it, it depends on how much effort you end up spending on tests, and whether it would have been better to spend more of it on your actual system instead.

Long version:

Some good answers here, but my take on it is different: testing is an economic activity that needs to pay back for itself, and if the time you spend isn't returned in development and system reliability (or anything else you look to get out of tests) then you may be making a bad investment; you're in the business of building systems, not writing tests. Therefore, reducing the effort to write and maintain tests is crucial.

For example, some main values I gain from tests are:

  • Reliability (and therefore development speed): refactor code/integrate a new framework/swap a component/port to a different platform, be confident that stuff still works
  • Design feedback: classic TDD/BDD "use your code" feedback on your low/mid-level interfaces

Testing against a live endpoint should still provide these.

Some drawbacks for testing against a live endpoint:

  • Environment setup - configuring and standardizing the test running environment is more work, and subtly different environment setups could result in subtly different behavior
  • Statelessness - working against a live endpoint can end up promoting writing tests that rely on a mutating endpoint state, which is fragile and hard to reason against (i.e. when something is failing, is it failing because of weird state?)
  • Test running environment is fragile - if a test fails, is it the test, the code, or the live endpoint?
  • Run speed - a live endpoint is usually slower, and sometimes is harder to parallelize
  • Creating edge cases for testing - usually trivial with a mock, sometimes a pain with a live endpoint (e.g tricky ones to set up are transport/HTTP errors)

If I were in this situation, and the drawbacks didn't seem to be a problem whereas mocking the endpoint slowed down my test writing considerably, I'd test against a live endpoint in a heartbeat, as long as I'd be sure to check again after a while to see that the drawbacks don't turn out to be a problem in practice.

orip
  • 309
  • 1
  • 5
1

From a testing perspective there are some requirements that are an absolute must:

  • Testing (unit or otherwise) must never have a way to touch production data
  • The results from one test must never affect the results of another test
  • You must always start from a known position

That's a big challenge when connecting to any source that maintains state outside of your tests. It's not "pure" TDD, but the Ruby on Rails crew solved this issue in a way that might be able to be adapted for your purposes. The rails test framework worked in this way:

  • Test configuration was automatically selected when running unit tests
  • The database was created and initialized at the start of running unit tests
  • The database was deleted after unit tests were run
  • If using SqlLite, the test configuration used a RAM database

All this work was built into the test harness, and it works reasonably well. There's a lot more to it, but the basics are enough for this conversation.

On the different teams that I've worked with over time, we would make choices that would promote code getting tested even if it wasn't the most correct path. Ideally, we would wrap all calls to a data store with code we controlled. In theory, if any of these old projects got new funding we could go back and move them from being database bound to Hadoop bound by focusing our attention on just a handful of classes.

The important aspects are not to mess with production data, and make sure you are truly testing what you think you are testing. It's really important to be able to reset the external service to a known baseline on demand--even from your code.

Berin Loritsch
  • 45,784
  • 7
  • 87
  • 160