Valid use-cases for white-box testing?

Question

White-box testing focuses on testing implementation details of a component. Black-box testing sees a component as a black box accessible only via its API.

Both Lean and Agile software development focus on the value delivered to the customer (Lean calls everything else "waste").

Taking this seriously, white-box testing has no place in development as it doesn't represent any customer value:

Values are specified by requirements, requirements define contracts and contracts are implemented (for example) by interfaces:

interface MyUseCase {

    int deliverySomeValue(String input);
}

class MyUseCaseImpl implements MyUseCase {

    public int deliverySomeValue(String input) {
        validate(input);
        String valueAsString = transform(input);
        return calculate(valueAsString);
    }

    void validate(String input) throws InvalidInputException { ... }

    String transform(String input) { ... }

    int calculate(String value) { ... }
} 

If there is a test with all possible variants of input, testing whether the output is correct, it is actually not important whether any of methods validate, transform, calculate doesn't work, as far as the result is right.

In fact, if there is a method which doesn't work and the result is still correct, the method is probably just unused or redundant - this should be addressed by refactoring, not testing.

A more difficult (to test) example could be:

interface MyRepository {

    void save(String input);
}

The method save doesn't return anything (void), so the side effect cannot be tested just via this contract. Either I would expect a second method read inside this or another interface. I can still use the read method to create a black-box (integration) test.

Problematic could be this example:

interface EmailService {

    void send(String to, String subject, String body);
}

The implementation is probably using some e-mail client and the naive approach to test it would be to create a mock of the client and verify, that a sending method was called. But what would be the point of such a test? Such a test brings me no guarantee that the e-mail was really sent. So, testing implementation doesn't help in this case as well. Probably only way how to test this would be to create a test mailbox and pull the sent e-mail with an e-mail client.

Are my assumptions correct? When does testing implementation make sense?

Answer 1

Your question shows a misconception about what White-Box testing means: you wrote

White-box testing focuses on testing implementation details of a component.

That is not correct, at least not literally. White box testing is not about testing implementation details of a component directly, but it makes use of the knowledge about internal implementation to create "better" test cases. Let me explain what this means it in terms of your example. You wrote

If there is a test with all possible variants of input, testing whether the output is correct, it is actually not important whether any of methods validate, transform, calculate doesn't work, as far as the result is right.

One cannot test "all possible variants of a string variable input" exhaustively - the input space is way too huge, even it is just one variable, any attempt to try this would take until the end of the universe.

So one has to pick a few "interesting test cases". But how can you do this in an efficient manner?

The "blackbox way" of creating tests is just to look at the spec, design your test cases around this and hope that is 100% sufficient. The "whitebox way" of doing this, however, is to look, into the code of, for example, the validate method and check which different cases are really validated in there. For which kind of input should it throw an exception? For which not? Which kind of input leads to code coverage or branch coverage? Are there cases which were forgotten in the spec, or for which it is not immediately clear if they fit exactly to the spec?

Hence, looking into the code can you give a lot of additional insights of how to pick the important test cases. Interestingly, these test cases stay often useful even when the implementation is changed later on by refactoring. So the tests don't "test implementation details". They still focus on the correct behaviour, even if they were created initially by using knowledge about implementation details.

Finally, let me comment on this part of your question:

Taking this seriously, white-box testing has no place in development as it doesn't represent any customer value.

There is customer value in functional as well as in non-functional requirements. One of the most important non-functional requirement is correctness of a program. Picking good test cases obviously helps to achieve this goal, and since whitebox testing does exactly this - assisting in finding good test cases, the value for the customer should be pretty clear.

Answer 2

Methods with side effects are naturally resistant to testing, because by definition, they're side effects because they're not directly resulting from calls to or from input/output. If these side effects involve calls to third-party systems like databases or e-mail clients, then it can be even more difficult to test. If you wish to test these methods, you should consider them as a test in two parts.

1. There is the logic being performed within your method prior to the actual call, say, in preparation of your e-mail to send
2. There is the actual call, or in this case the sending of an e-mail

The second part tests your configuration and the actual job of sending an e-mail or persisting to a database. This second part cannot be tested or mocked. If this is what you're interested in testing, then the only thing to do is simply to call it. If you're lucky enough that the side effect can be directly verified, then you can automate verification and you can test this easily. If not, say, through sending an e-mail, then not much else can be done for this second part.

If you want to test the first part, it is possible to test these methods by using dependency injection (DI) or by passing an interface whose only job is to stupidly perform the actual call to send the e-mail or to save to the database.

For example, an implementation of EmailService might be say:

class MyEmailService {
    EmailSender emailSender = ... // dependency injected here

    void send(String to, String subject, String body) {
        Message message = ... // message preparation logic here
        emailSender.send(message);
    }
}

The actual sending of the e-mail is performed in EmailSender which is an interface representing the direct call to perform the send with no logic. This allows you to inject your own mock version of EmailSender which you can use to verify the results of the call during your testing. Alternatively you could directly pass an instance of EmailSender to the send method, but the dependency injection is somewhat preferable for readability sake.

So in conclusion, it can make sense to test such methods, but it's important to make a distinction between testing the logic up to the point of the actual e-mail sending and the actual sending of the e-mail. If you have no logic prior to sending the e-mail then there is no "first part" to test prior to the actual sending and therefore nothing to test in that regard. In that case, you shouldn't bother with testing this method unless you were purely interested in testing the second part. And if needbe, you can test the second part independently of the first. You should evaluate whether or not you want to test the first or the second and perform the appropriate test for that method.

Answer 3

It is easy to get caught up in principles and definitions. When it comes down to it, the purpose of testing is to create good software.

So the question is, can white-box testing improve the quality of software? I think the answer is obviously yes.

Say you are testing a method, and you know that method internally uses a third-party component which you know have a bug which surfaces given some very specific input. So does it provide value to test the method with input which might trigger this bug? I think it is obvious it does - it prevent against a potential bug. Now if someone later rewrites the implementation to not use that third party component, the test is not affected (and should still pass), so it does not make the test code more fragile.

If you could test every possible input and scenario against the specification, then there wouldn't be any need for white-box testing. But this is only possible in a few limited cases. Usually you test only a few scenarios out of infinitely many possibilities.

Answer 4

The problem is that if you turn anything into dogma then your answers turn into rubbish. If the dogma calls not following the dogma “waste” that’s a dozen red flags waving at the same time. That’s a seller of snake oil.

Let’s say I build an airplane and use white box testing for its control software. Like the special case of crossing the equator. No value for the customer, right? Leaving out that particular test would have killed people when some US fighter plane was found to turn on its back when crossing the equator.