Is there a point to unit tests that stub and mock everything public?

Question

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything. I'm literally looking at my code and creating examples based on the flow of logic through my public methods. And every time the implementation changes, I have to go and change those tests, again, not really feeling that I'm accomplishing anything useful (be it mid- or long-term). I also do integration tests (including non-happy-paths) and I don't really mind the increased testing times. With those, I feel like I'm actually testing for regressions, because they have caught multiple, while all that unit tests do is show me that the implementation of my public method changed, which I already know.

Unit testing is a vast topic, and I feel like I'm the one not understanding something here. What's the decisive advantage of unit testing vs integration testing (excluding the time overhead)?

Answer 1

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything. I'm literally looking at my code and creating examples based on the flow of logic through my public methods.

This sounds like the method you are testing needs several other class instances (which you have to mock), and calls several methods on its own.

This type of code is indeed difficult to unit-test, for the reasons you outline.

What I have found helpful is to split up such classes into:

1. Classes with the actual "business logic". These use few or no calls to other classes and are easy to test (value(s) in - value out).
2. Classes that interface with external systems (files, database, etc.). These wrap the external system and provide a convenient interface for your needs.
3. Classes that "tie everything together"

Then the classes from 1. are easy to unit-test, because they just accept values and return a result. In more complex cases, these classes may need to perform calls on their own, but they will only call classes from 2. (and not directly call e.g. a database function), and the classes from 2. are easy to mock (because they only expose the parts of the wrapped system that you need).

The classes from 2. and 3. cannot usually be meaningfully unit-tested (because they don't do anything useful on their own, they are just "glue" code). OTOH, these classes tend to be relatively simple (and few), so they should be adequately covered by integration tests.

---

An example

One class

Say you have a class which retrieves a price from a database, applies some discounts and then updates the database.

If you have this all in one class, you'll need to call DB functions, which are hard to mock. In pseudocode:

1 select price from database
2 perform price calculation, possibly fetching parameters from database
3 update price in database

All three steps will need DB access, so a lot of (complex) mocking, which is likely to break if the code or the DB structure changes.

Split up

You split into three classes: PriceCalculation, PriceRepository, App.

PriceCalculation only does the actual calculation, and gets provided the values it needs. App ties everything together:

App:
fetch price data from PriceRepository
call PriceCalculation with input values
call PriceRepository to update prices

That way:

• PriceCalculation encapsulates the "business logic". It's easy to test because it does not call anything on its own.
• PriceRepository can be pseudo-unit-tested by setting up a mock database and testing the read and update calls. It has little logic, hence few codepaths, so you do not need too many of these tests.
• App cannot be meaningfully unit-tested, because it is glue-code. However, it too is very simple, so integration testing should be enough. If later App gets too complex, you break out more "business-logic" classes.

Finally, it may turn out PriceCalculation must do its own database calls. For example because only PriceCalculation knows which data its needs, so it cannot be fetched in advance by App. Then you can pass it an instance of PriceRepository (or some other repository class), custom-tailored to PriceCalculation's needs. This class will then need to be mocked, but this will be simple, because PriceRepository's interface is simple, e.g. PriceRepository.getPrice(articleNo, contractType). Most importantly, PriceRepository's interface isolates PriceCalculation from the database, so changes to the DB schema or data organisation are unlikely to change its interface, and hence to break the mocks.

---

Note: I recently noticed that this concept is fairly similar to Alistair Cockburn's Hexagonal architecture. So I guess I've just been reinventing the wheel...or maybe great minds think alike?

Answer 2

What's the decisive advantage of unit testing vs integration testing?

That's a false dichotomy.

Unit testing and integration testing serve two similar, but different purposes. The purpose of unit testing is to make sure your methods work. In practical terms, the unit tests make sure that the code fulfills the contract outlined by the unit tests. This is evident in the way that unit tests are designed: they specifically state what the code is supposed to do, and assert that the code does that.

Integration tests are different. Integration tests exercise the interaction between software components. You can have software components that pass all of their tests and still fail integration tests because they don't interact properly.

However, if there is a decisive advantage to unit tests, it is this: unit tests are much easier to set up, and require far less time and effort than integration tests. When used properly, unit tests encourage the development of "testable" code, which means the final result is going to be more reliable, easier to understand, and easier to maintain. Testable code has certain characteristics, like a coherent API, repeatable behavior, and it returns results that are easy to assert.

Integration tests are more difficult and more expensive, because you often need elaborate mocking, complex setup, and difficult assertions. At the highest level of system integration, imagine trying to simulate human interaction in a UI. Entire software systems are devoted to that sort of automation. And it is automation that we're after; human testing is not repeatable, and doesn't scale like automated testing does.

Finally, integration testing makes no guarantees about code coverage. How many combinations of code loops, conditions and branches are you testing with your integration tests? Do you really know? There are tools that you can use with unit tests and methods under test that will tell you how much code coverage you have, and what the cyclomatic complexity of your code is. But they only really work well at the method level, where unit tests live.

---

If your tests are changing every time you refactor, that's a different problem. Unit tests are supposed to be about documenting what your software does, proving that it does that, and then proving that it does that again when you refactor the underlying implementation. If your API changes, or you need your methods to change in accordance with a change in the system design, that's what is supposed to happen. If it's happening a lot, consider writing your tests first, before you write code. This will force you to think about the overall architecture, and allow you to write code with the API already established.

If you're spending a lot of time writing unit tests for trivial code like

public string SomeProperty { get; set; }

then you should reexamine your approach. Unit testing is supposed to test behavior, and there is no behavior in the line of code above. You have, however created a dependency in your code somewhere, since that property is almost certainly going to be referred to elsewhere in your code. Instead of doing that, consider writing methods that accept the needed property as a parameter:

public string SomeMethod(string someProperty);

Now your method doesn't have any dependencies on something outside of itself, and it is now more testable, since it is completely self-contained. Granted, you won't always be able to do this, but it does move your code in the direction of being more testable, and this time you're writing a unit test for actual behavior.

Answer 3

The unit tests with mocks are to make sure the implementation of the class is correct. You mock the public interfaces of the dependencies of the code that you are testing. This way you have control over everything external to the class and are sure that a failing test is due to something internal to the class and not in one of the other objects.

You are also testing the behavior of the class under test not the implementation. If you refactor the code (creating new internal methods, etc) the unit tests should not fail. But if you are changing what the public method does then absolutely the tests should fail because you have changed the behavior.

It also sounds like you are writing the tests after you have written the code, try instead writing the tests firsts. Try outlining the behavior that the class should have and then write the minimum amount of code to make the tests pass.

Both unit testing and integration testing are useful for ensuring the quality of your code. The unit tests examine each component in isolation. And the integration tests make sure that all the components interact properly. I want to have both types in my test suite.

Unit tests have helped me in my development as I can focus on one piece of the application at a time. Mocking the components that I haven't made yet. They also a great for regression, as they document any bugs in the logic that I have found (even in the unit tests).

UPDATE

Creating a test that only makes sure that methods are called has value in that you are making sure that the methods actually do get called. Particularly if you are writing your tests first, you have a check list of methods that need to happen. Since this code is pretty much procedural, you don't have much to check other than that the methods get called. You are protecting the code for change in the future. When you need to call one method before the other. Or that a method always gets called even if the initial method throws an exception.

The test for this method may never change or may change only when you are changing the methods. Why is this a bad thing? It helps reinforce using the tests. If you have to fix a test after changing the code, you will get in the habit of changing the tests with the code.

Answer 4

I was experiencing a similar question - until I discovered the power of component tests. In short, they are the same as unit tests except that you don't mock by default but use real objects (ideally via dependency injection).

That way, you can quickly create robust tests with a good code coverage. No need for updating your mocks all the time. It might be a bit less precise than unit tests with 100% mocks, but the time and money you save compensate for that. The only thing you really need to use mocks or fixtures for are storage backends or external services.

Actually, excessive mocking is an anti-pattern: TDD Anti-Patterns and Mocks are evil.

Answer 5

Though the op has already marked an answer, I am just adding my 2 cents here.

What's the decisive advantage of unit testing vs integration testing (excluding the time overhead)?

And also in response to

When doing unit tests the "proper" way, i.e. stubbing every public call and return preset values or mocks, I feel like I'm not actually testing anything.

There is a helpful but not exactly what OP has asked:

Unit Tests work but there are still bugs?

from my little experience on testing suites, I understand that Unit Tests are always to test the most basic method level functionality of a class. In my opinion every method either public, private or internal deserves to have a dedicated unit tests. Even in my recent experience I had a public method that was calling other small private method. So, there were two approaches:

1. don't create a unit test(s) for private method.
2. create a Unit Test(s) for a private method.

If You think logically, the point of having a private method is: main public method is getting too large or messy. In order to solve this You refactor wisely and create out small chunks of code that deserve to be separate private methods which in turn make Your main public method less bulky. You refactor by keeping in mind that this private method might be reused later on. There can be cases in which there are no other public method depending upon that private method, but who knows about future.

---

Considering the case when private method is reused by many other public methods.

So, If I had chosen approach 1: I would have duplicated Unit tests and they would have been complicated, as You had number of Unit Tests for each branch of public method as well as private method.

If I had chosen approach 2: the code written for unit tests would be relatively less, and it would be much easier to test.

---

Considering the case when private method is not reused There is no point of writing a seperate unit test for that method.

As for as Integration Tests are concerned, they tend to be exhaustive and more of high level. They will tell You that given a input, all of Your classes should come to this final conclusion.To understand more about usefulness of integration testing please see the mentioned link.

Answer 6

I have moved away from unit testing, and almost exclusively prefer integration testing.

Actually, I dislike the terms "unit" and "integration". I would prefer to say that I focus on testing the public interface, and only the public interface.

Sometimes the public interface is a fairly low-level API. In which case, you write tests that directly test that API. These are likely to be fairly low-level tests and could fairly be called "unit tests".

But more often, the public interface is at a higher level, and your units are an implementation detail of that public interface. There are several problems with focusing on the units and exhaustive unit testing instead of higher-level tests (which I have learned by painful experience):

1. Being part of the implementation, units are subject to change. Often, you will not discover this until you try and integrate the units together, and you realise that you have to re-write some or all of the units. This pain is unavoidable and part of the software development cycle, but if you've dutifully and thoroughly written unit tests for all of said units, the pain is doubled. In contrast, if you've written your tests at the level of public interface ("integration" level), these tests will often remain the same through a major refactor of your units. They will give you confidence that your overall system is still doing what it was meant to even as you refactor the units.
2. As a variant of the first point, if you focus on the unit level and thorough unit testing, it's easy to get carried away and find that you've written code in your units that wasn't really necessary when you started to integrate them (again doubling the waste if you've written tests for the redundant code). If you apply test-driven practices at the level of public interface, you tend to avoid falling into the trap of writing unnecessary code in your units and you'll only write what is necessary.
3. Because units interact with other units when they are integrated, you end up needing to mock those other units to isolate your unit-under-test to get a "pure" unit test. You've clearly experienced the drawbacks of mocks - they are by definition always less accurate than the units they are mocking, which leaves sleeping bugs in your units that don't show up until you start to integrate them. Attempts to fix this over time produce mocks that are often nearly as complicated as the unit they are mocking (but still never as accurate).

It is true that integration tests often run slower than unit tests. But IMO it doesn't matter how fast the tests run if they are inaccurate or testing code that is redundant. I prefer the accurately test code that is actually necessary to achieve what I want to achieve (the "public interface").

It is also true that you sometimes can't get 100% coverage of your internal units with integration tests alone. But in my experience, this is usually an indication that you have functionality in your units that you don't really need.

At the end of the day, we care about what our software does - not how it does it. We should test what we care about, and avoid testing what we ultimately do not care about as that is a waste of time and effort. Units are generally more about the "how" than the "what".