How narrow should my unit tests be?

Question

Here's an example:

I have a chat module in my app, and there's a ChatService class that is responsible for networking, and there's a ChatNotificationService helper class that is responsible for sending out and receiving broadcasts of messages, and there's also a ChatNotificiationDelegate protocol that has methods on it that get called when a notification arrives.

In order to test message sending, in theory I could just call ChatService.sendMessage() and check if the ChatNotificationDelegate.didSentMessage() got called.

However, sending a message calls several different methods before it gets to the delegate. It sends the message to the server, the server then sends a response that triggers sending of the notification, and the notification service then receives the notification and calls the delegate method.

This whole chain can be covered by one unit test, or I could test each piece of functionality separately.

What I'm asking is which approach should I take on this, test every bit we know exactly what went wrong, or to make my tests as general as possible (just the endpoints) to allow more freedom of refactoring the code.

Answer 1

The question you should actually be asking yourself is this: is my code testable? If you find yourself writing elaborate tests or using elaborate mocks in your tests, it is the fault of the code under test, not the tests themselves.

Unit tests test a unit of code. So it comes down to "what is a unit?" A unit, in general terms, is one small bit of clearly-defined functionality that is readily testable. Usually, it is a single method. If you write code that is testable, your proper test "width" should naturally arise from that.

In any case, what you are describing in your question is not a unit test; it is an integration test. The relative merits of unit tests vs integration tests are discussed at length elsewhere, but suffice it to say that you write unit tests to make sure your code works, but you write integration tests to make sure your code works together.

Answer 2

One approach I sometimes use is to write tests against your interface ie.

SendMessageTest(IChatService service)
{
    service.SendMessage(message);
    Assert.IsTrue(message.Sent);
}

Then use data driven testing to call the test with multiple configurations of service. First one with all mocks, this is your unit test, only testing the service itself, then an In process service, so using real messages but all instantiated in process for debugging. Then multiple integration versions using a client version of the service which connects to test, uat and live environments.

each test case should be labeled up as "Integration" or whatever so you can choose which to run under what scenario.

So for CI builds you might only run the unit tests, for debugging you can run the in process ones, after deploying to test you can run the test integration ones and if you have a weird problem in live you can run the live integration ones.

The benefit of this is you only have to write one set of tests but are able to run them in a variety of ways.

PS. you can/should also write further tests for specific logic etc, but I find this 'top level' testing of the most practical use. After all most of the time you know it 'should work' you are trying to find out why it doesn't under circumstance X

Answer 3

You are comparing 2 kinds of tests: unit tests and integration tests. Unit tests should be fast. Myself, I like unit tests to run a fraction of a second. This matters when you run a test repeatedly while writing/changing code. To make unit test fast, your test can't call any remote resource or access a database. Most definitely, you don't want to start a chat server in your unit test.

Unit tests should also help you with refactoring. It's best done when your unit test only goes to the nearest architectural boundary. You test that you reach the boundary and hand correct message over the boundary. You may also check that your code correctly handles whatever may be returned from across the boundary. This way your system is loosely coupled and changeable - and so are your tests.

In addition to unit tests, you also test that your system interacts correctly across the boundaries and with other systems. People call these tests interaction or integration tests. They may take longer to run because they typically perform remote access and may even have to launch a test instance of a server. Because you already decoupled your layers in unit tests, you don't need to test every possible permutation of inputs and outputs in your integration tests. Each integration test runs longer, but you have fewer of them.

Putting it all together, I suggest that you create "small" unit tests for your ChatService, ChatNotificationDelegate and so on, going only as far as the next service. You can mock the other service to verify that your test subject passes the right parameters across the boundary and reacts correctly to returns. You would aim to achieve a high coverage of unit tests. In addition, you will have a small number of integration tests that each work across one boundary. Or maybe even a single end to end test of sending the message, as you suggested.

So my answer to

test every bit we know exactly what went wrong, or to make my tests as general as possible (just the endpoints) to allow more freedom of refactoring the code

is both: many small unit tests to know exactly what went wrong and one big integration test to be sure all pieces fit together. And I'd like to mention that a single end to end test doesn't "allow freedom of refactoring". The reverse is often true. Such a black box test can tell you that something is wrong, but you will have no idea what exactly broke. If system can break mysteriously, without an easy way to identify or at least pinpoint the root cause, you are more likely to fear making significant changes, such as refactoring.