1

at company we're developing quite a big project and we're arguing at the testing strategy. The question is: should all of the tests be executed in isolation of external services like database or APIs (facebook etc) or just part of them? Of course what I have in mind is using mockups. We discussed following strategies:

  1. Write all of the tests in isolation of external services - mock externals everywhere
  2. Write part of the tests in isolation and create single bigger functional test for every feature that tests it using externals (without mocking anything) - using fixtures
  3. Run all tests communicating with externals (of course that's never appliable to unit tests so they're out of scope in that case)

I know it can start quite a discussion but I think that's what it is all about, I'd like to find pros and cons that I didn't think of already.

Mikz
  • 113
  • 2
  • 6
    This is like asking "Should we have unit tests or integration tests?". The answer must be "you should have both, because they fulfill different functions." Mocking is appropriate for unit tests, but not for integration/functional tests. – Kilian Foth Aug 13 '14 at 11:42
  • Hi, thank you for your response. So you say that functional tests should always be executed without mocking? But how to perform them when you have no idea what you'll get in response from external service (external response can f. ex. be RSS that will or won't contain images and you need them to be there to test your functionality)? – Mikz Aug 14 '14 at 08:19
  • related: [Staying OO and Testable while working with a database](http://programmers.stackexchange.com/questions/42792/staying-oo-and-testable-while-working-with-a-database) – gnat Sep 17 '14 at 11:17

1 Answers1

3

If done right, your second strategy can have the best of both worlds:

Write part of the tests in isolation

Isolating most tests from external services is a good idea because you have no control over that service. The worst-case scenario is to have unit tests fail intermittently, because eventually you'll start to ignore the failures, at which point the tests are useless. Creating a mock that provides consistent responses to your tests makes it much easier to verify how your code reacts to a given response from the external service.

create single bigger functional test for every feature that tests it using externals

This can provide important insight into the behavior of the system closer to actual operating conditions, allowing you to recognize and handle different issues than you'll encounter when external services are mocked, such as network issues, or input or output data formatting errors.

  1. sometimes you don't know how external source might react in some conditions, it's hard to mock it (for example Solr with complex query)

  2. sometimes you can forget about software-specific case (for example switching from MySQL to Postgres, while using ORM, might return different results and cause some other problems)

If you can dependably reproduce some response from the external service, and you want to verify that your code reacts to it in a certain way, then it makes sense to test it. The important thing is that you don't allow intermittent failures in your tests that e.g. run with every build. I've seen very good developers switch off the tests because a build needed to be made and they didn't have the time to fix an obscure, complex test. That's a bad situation to be in, because the tests are no longer serving their purpose, and confidence in them is reduced.

Mike Partridge
  • 6,587
  • 1
  • 25
  • 39
  • I work with @Mikz, so thanks for your input. My question is - how about complex services and their behavior, that can impact testing "in real life"? I mean: 1) sometimes you don't know how external source might react in some conditions, it's hard to mock it (for example Solr with complex query), and 2) sometimes you can forget about software-specific case (for example switching from MySQL to Postgres, while using ORM, might return different results and cause some other problems) – ex3v Aug 14 '14 at 06:43
  • 1
    @ex3v - I've updated the answer. – Mike Partridge Aug 14 '14 at 13:46