101

You have a class X and you write some unit tests that verify behaviour X1. There's also class A which takes X as a dependency.

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

But what with A? Unit tests for A will not fail when X's behaviour is modified (due to the mocking of X). How to detect that A's outcome will be different when run with the "real" (modified) X?

I'm expecting answers along the line of: "That's not the purpose of unit testing", but what value does unit testing have then? Does it really only tell you that when all tests pass, you haven't introduced a breaking change? And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences? Shouldn't we focus more on integration testing?

Philipp
  • 23,166
  • 6
  • 61
  • 67
bvgheluwe
  • 1,177
  • 2
  • 8
  • 12
  • 9
    Possible duplicate of [(Why) is it important that a unit test not test dependencies?](https://softwareengineering.stackexchange.com/questions/65477/why-is-it-important-that-a-unit-test-not-test-dependencies) – gnat Mar 27 '18 at 11:54
  • 37
    In addition to all the answers suggested, I must say I take issue with the following statement: 'Does it really **only** tell you that when all tests pass, you haven't introduced a breaking change?' If you really think removing the fear of refactoring is of little value, you are on the fast track towards writing unmaintainable code – crizzis Mar 27 '18 at 15:13
  • 5
    Unit testing tells you whether your unit of code behaves as you expect it to. No more or less. Mocks and test doubles provide an artificial, controlled environment for you to exercise your unit of code (in isolation) to see if it meets your expectations. No more or less. – Robert Harvey Mar 27 '18 at 17:37
  • 3
    I believe your premise is incorrect. When you mention `X1` you are saying that `X` implements interface `X1`. If you change the interface `X1` to `X2` the mock you used in the other tests should not compile anymore, hence you are forced to fix those tests too. Changes in the class behaviour should not matter. In fact, your class `A` should not depend on implementation details (which is what you'd be changing in that case). So the unit tests for `A` are still correct, and they tell you that `A` works given an ideal implementation of the interface. – Bakuriu Mar 27 '18 at 17:49
  • @Bakuriu This was something that crossed my mind too. Be interesting to see if the OP can post an example to illustrate. – Robbie Dee Mar 27 '18 at 19:23
  • 5
    I don't know about you, but when I have to work on a codebase with no tests, I'm scared to death I'm going to break something. And why? Because it happens so often that something breaks when it wasn't intended for. And bless our tester's hearts, they can't test everything. Or even close. But a unit test will happily chug away through boring routine after boring routine. – corsiKa Mar 27 '18 at 21:26
  • potentially related: [Answer about effectiveness of various classes of tests](https://softwareengineering.stackexchange.com/a/301540/134647) – Nick Alexeev Mar 28 '18 at 04:31
  • Did you included cases for A when mocked X behaves abnormally as well ? – S.D. Mar 28 '18 at 04:56
  • Just because you use genuine currency with magstipes, watermarks, etc, doesn't mean you're spending wisely. Unit tests are best for structure; not transient application code but rather parts like frameworks that underpin more specific application operations. Tests facilitate optimizing and expanding such middleware with minimal superstructure endangerment. While it's true that greater test abstraction lowers utility (will my app work?), it also boosts applicability (will this new storage routine play nice with our app suite?). – dandavis Mar 28 '18 at 06:33
  • Unit tests tell us that (hopefully) our code over at A did not break when we changed B – EpicKip Mar 28 '18 at 13:08
  • I've been on a project that wanted full test coverage and no regressions to accept a change, but only did fully integrated UI-to-storage tests. The tests were very good at telling me my change broke something. The tests were very, very bad at telling me what broke things. Integrated tests will say "well your output was three pixels off from the output when somebody wrote the tests" but it won't tell you why adding a new button to panel A would make the contents of its sibling panel above shift, because the breaking effects of the change aren't localized beyond a very coarse-grained interaction – millimoose Mar 28 '18 at 13:57
  • Relatedly, doing isolated unit tests can force your design to be more loosely coupled so such spooky action at a distance is less likely to occur. Even though arguably this is neither guaranteed, nor the only way to achieve this; but integrated tests won't affect your design as much as isolated tests that require every object-object boundary to be mockable. – millimoose Mar 28 '18 at 13:59
  • See also: http://blog.thecodewhisperer.com/permalink/integrated-tests-are-a-scam. Note the distinction between "integrated" and "integration" tests; the latter are perfectly fine and necessary when you're actually testing the interaction of more than one unit or layer. He's talking about writing end-to-end tests because you can't be bothered to work at a finer level of detail. (Note that there's dissenting opinions on this: I've seen someone argue this adds busywork by making you deal with broken internal tests even though the observable effect of a system remain the same.) – millimoose Mar 28 '18 at 14:02
  • @crizzis This is the real answer! It's the main goal of unit tests to make sure you don't accidentally change the behaviour of an implementation and that it matches the interface it's supposed to provide. You should turn this into an answer. – Frank Hopkins Mar 28 '18 at 17:36
  • 1
    I think if you mock X then others will mock you – user541686 Mar 31 '18 at 04:39
  • what if X just renamed index in returned array from foobar to foobarRandomNumber, how can I count with that? if you get my point, this is basically my issue, I renamed a returned column from secondName x1 to surname x2, a classic task, but my test will never know, since its mocked. I just have such a strange feeling as if many people in this question never actually tried something like that, before commenting – FantomX1 Oct 06 '19 at 22:27

11 Answers11

124

When you write unit tests for A, you mock X

Do you? I don't, unless I absolutely have to. I have to if:

  1. X is slow, or
  2. X has side effects

If neither of these apply, then my unit tests of A will test X too. Doing anything else would be taking isolating tests to an illogical extreme.

If you have parts of your code using mocks of other parts of your code, then I'd agree: what is the point of such unit tests? So don't do this. Let those tests use the real dependencies as they form far more valuable tests that way.

And if some folk get upset with you calling these tests, "unit tests", then just call them "automated tests" and get on with writing good automated tests.

David Arno
  • 38,972
  • 9
  • 88
  • 121
  • by `If you have parts of your code using mocks of other parts of your code` you mean mocking several dependencies within the same dependency graph? Following this analogy, if each node of the graph is a dependency, are not unit test meant to test nodes one by one? (implementation details aside) – Laiv Mar 27 '18 at 12:34
  • 95
    @Laiv, No, unit tests are supposed to act as a unit, ie run in isolation, *from other tests*. Nodes and graphs can take a hike. If I can run an isolated, side-effect free end-to-end test in a short time, that's a unit test. In if you don't like that definition, call it an automated test and stop writing crap tests to fit silly semantics. – David Arno Mar 27 '18 at 12:36
  • 9
    @DavidArno There is alas a **very** broad definition of isolated. Some would like a "unit" to include the mid-tier and the database. They can believe whatever they like but the chances are that on a development of any size, the wheels will come off in fairly short order as the build manager will throw it out. Generally, if they're isolated to the assembly (or equivalent), that is fine. N.B. if you code to interfaces then it is far easier to add mocking and DI later. – Robbie Dee Mar 27 '18 at 12:47
  • Your comment above is echoing the "Fakes" section of the article, [*The Philosophy of Testing*](https://www.codesimplicity.com/post/the-philosophy-of-testing/), which is a *very* worthwhile read. – Wildcard Mar 27 '18 at 22:42
  • 14
    You're advocating for a different kind of test rather than answering the question. Which is a valid point, but this is a pretty underhanded way of doing it. – Phil Frost Mar 27 '18 at 23:53
  • 17
    @PhilFrost, to quote myself, "*And if some folk get upset with you calling these tests, "unit tests", then just call them "automated tests" and get on with writing good automated tests.*" Write useful tests, not silly tests that simply meet some random definition of a word. Or alternatively, accept that maybe you have your definition of "unit test" wrong and that you are over using mocks because you have it wrong. Either way, you'll end up with better tests. – David Arno Mar 28 '18 at 07:21
  • 31
    I'm with @DavidArno on this one; my testing strategy changed after watching this talk by Ian Cooper: https://vimeo.com/68375232. To boil it down to a single sentence: **Don't test classes**. Test _behaviours_. Your tests should not have knowledge of the internal classes / methods used to implement the desired behaviour; they should only know the public surface of your API / library and they should test that. If tests have too much knowledge then you're testing implementation details and your tests become brittle, coupled to your implementation, and actually just an anchor around your neck. – Richiban Mar 28 '18 at 12:33
  • 5
    @DavidArno "*some random definition of a word*" - it really [couldn't be clearer](https://en.wikipedia.org/wiki/Unit_testing). Hell, it is even on the first line. Please **do** call them automated tests or whatever your *mot du jour* is and stop deliberately conflating concepts. The tag is unit-testing - not integration testing and not BDD. – Robbie Dee Mar 28 '18 at 17:17
  • 4
    This doesn't answer the question - if a test for X doesn't mock A, then it's also a test for A. But there is always an Y that you don't want to test with A, for which the question applies. Unless you only have full integration tests which seems weird and unmaintainable with a decent coverage for more complex projects. – Frank Hopkins Mar 28 '18 at 17:32
  • 1
    @Darkwing Yep, it's happy path testing however one spins it. Something that'll be picked up by manual tests most of the time anyway. – Robbie Dee Mar 28 '18 at 17:54
  • @Richiban I agree with you and David - unit tests should be testing a specific behaviour assuming a perfect world (ie, that all dependencies do exactly what you expect them to - and you achieve that by unit testing them as well). Once you start assuming a non-perfect world, you are testing something else - which is fine, integration tests, end to end tests etc are all usable. – Moo Mar 28 '18 at 19:37
  • @Richiban There is a newer version of the talk [here](https://www.youtube.com/watch?v=EZ05e7EMOLM). I mostly agree although many of the flaws he identifies which went before boil down to bad process. – Robbie Dee Mar 28 '18 at 20:36
  • 2
    Testing that two classes work appropriately together is an integration test. This answer is not about unit testing and so I don't think it really answers the question very well. – Nick Coad Mar 29 '18 at 03:31
  • 8
    @NickCoad, "*Testing that two classes work appropriately together is an integration test*". See the point is, I say you are wrong. Mocking out one of those classes in the mistaken belief that it's then a unit test leads to crappy tests that are of no use to anyone. So let's stop calling those crappy tests, "unit tests" and let's reserve the term for good quality, useful tests. – David Arno Mar 29 '18 at 07:48
  • 2
    I definite 'unit' as a handful of classes that get tested together as a unit :P. A test that tests a class with all of its dependencies mocked out isn't testing anything real. – 17 of 26 Mar 29 '18 at 15:21
  • 1
    Testing business behaviour (instead of classes, as mentioned above) is perfectly possible with by-the-book unit-tests. Only the "unit" needs to be defined in a sensible way. Hint: "smallest piece of code that compiles in isolation" is not a sensible way of defining unit. Also, It's true that the answer is good, but doesn't answer the question: the same problem can be imagined for a dependency that must be mocked. – BartoszKP Mar 31 '18 at 19:29
79

You need both. Unit tests to verify behavior of each of your units, and a few integration tests to make sure they're connected correctly. The problem with relying only on integration testing is combinatorial explosion resulting from interactions between all your units.

Let's say you have class A, that requires 10 unit tests to fully cover all paths. Then you have another class B, that also requires 10 unit tests to cover all the paths the code can take through it. Now let's say in your application, you need to feed the output of A into B. Now your code can take 100 different paths from the input of A to the output of B.

With unit tests, you only need 20 unit tests + 1 integration test to completely cover all the cases.

With integration tests, you will need 100 tests to cover all code paths.

Here's a very good video about the downsides of relying on integration tests only J B Rainsberger Integrated Tests Are A Scam HD

Eternal21
  • 1,584
  • 9
  • 11
  • 1
    I'm sure it is no coincidence that the question mark over the efficacy of integration tests has gone hand in hand with unit tests covering every more layers. – Robbie Dee Mar 27 '18 at 13:00
  • 16
    Right, but nowhere does your 20 unit test's require mocking. If you have your 10 tests of A that cover all of A and your 10 tests that cover all of B and also retest 25% of A as a bonus this seems between "fine" and a good thing. Mocking A in Bs tests seems actively stupid (unless there really is a reason why A is a problem, e.g. it's the database or it brings in a long web of other things) – Richard Tingle Mar 27 '18 at 21:22
  • 10
    I disagree with the idea that a single integration test is sufficient if you want full coverage. The reactions of B to what A outputs will vary based on the output; if changing a parameter in A changes its output, then B may not handle it correctly. – Matthieu M. Mar 28 '18 at 09:09
  • @MatthieuM. If A is unit tested for all possible inputs, and B is unit tested for all possible inputs, then a single integration test, to make sure that the two interact properly is sufficient. That's the beauty of unit tests. Sure, you may want to add another one or two, to ensure proper handling when object B is null (if for example you are injecting it in A via constructor). – Eternal21 Mar 28 '18 at 11:39
  • 3
    @Eternal21: My point was that sometimes, the problem is not in the individual behavior, but in an unexpected interaction. That is, when the *glue* between A and B behaves unexpectedly in some scenario. So both A and B act according to specifications, and the happy case works, but on some inputs there's a bug in the glue code... – Matthieu M. Mar 28 '18 at 11:53
  • The state of the rest of the system is an implied input to A unless you mock up all possible states of its dependencies, unless most of your system is stateless to a fault. (In which case, great, but there’s other scenarios for testing other than “we’re doing it all right” greenfield development.) – millimoose Mar 28 '18 at 17:09
  • @Eternal21 the beauty of unit tests you talking about will be gone, when you start refactoring or for example want to extract some functionality from B, because your unit tests will fail. – Fabio Mar 28 '18 at 18:22
  • 1
    @MatthieuM. I’d argue this is beyond the scope of Unit Testing. The glue code can be unit tested itself while the interactions between A and B via the glue code is an Integration Test. When specific edge cases or bugs are found they can be added to the glue code unit tests and ultimately verified in Integration testing. – Andrew T Finnell Apr 01 '18 at 20:25
74

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1. Time goes by, people do use your system, needs change, X evolves: you modify X to show behaviour X2. Obviously, unit tests for X will fail and you will need to adapt them.

Woah, wait a moment. The implications of the tests for X failing are too important to gloss over like that.

If changing the implementation of X from X1 to X2 breaks the unit tests for X, that indicates that you've made a backwards incompatible change to the contract X.

X2 isn't an X, in the Liskov sense, so you should be thinking about other ways of meeting the needs of your stake holders (like introducing a new specification Y, that is implemented by X2).

For deeper insights, see Pieter Hinjens: The End of Software Versions, or Rich Hickey Simple Made Easy.

From the perspective of A, there is a precondition that the collaborator respects the contract X. And your observation is effectively that the isolated test for A doesn't give you any assurance that A recognizes collaborators that violate the X contract.

Review Integrated Tests are a Scam; in high level, you are expected to have as many isolated tests as you need to ensure that X2 implements the contract X correctly, and as many isolated tests as you need to ensure that A does the right thing given interesting responses from an X, and some smaller number of integrated tests to ensure that X2 and A agree on what X means.

You will sometimes see this distinction expressed as solitary tests vs sociable tests; see Jay Fields Working Effectively with Unit Tests.

Shouldn't we focus more on integration testing?

Again, see integrated tests are a scam - Rainsberger describes in detail a positive feedback loop that is common (in his experiences) to projects that are relying upon integrated (note spelling) tests. In summary, without the isolated/solitary tests applying pressure to the design, the quality degrades, leading to more mistakes and more integrated tests....

You will also need (some) integration tests. In addition to the complexity introduced by multiple modules, executing these tests tends to have more drag than the isolated tests; it's more efficient to be iterating on very fast checks when work is in progress, saving the additional checks for when you think you are "done".

VoiceOfUnreason
  • 32,131
  • 2
  • 42
  • 79
  • 8
    This should be the accepted answer. The question outlines a situation where the behaviour of a class has been modified in an incompatible way, but still outwardly looks identical. The problem here lies in the design of the application, not the unit tests. The way this circumstance would be caught in testing is using an integration test between those two classes. – Nick Coad Mar 29 '18 at 03:30
  • 1
    "without the isolated/solitary tests applying pressure to the design, the quality degrades". I think this is an important point. Alongside with behaviors checks, unit-tests have the side effect to force you to have a more modular design. – MickaëlG Mar 30 '18 at 12:07
  • I suppose this is all true, but how does this help me if an external dependency introduces a backwards incompatible change to the contract X? Perhaps an I/O-performing class in a library breaks compatibility, and we're mocking X because we don't want unit tests in CI to depend on heavy I/O. I think the OP is asking for to test for this, and I don't understand how this answers the question. How to test for this? – gerrit Aug 23 '19 at 15:21
17

Let me start by saying that the core premise of the question is flawed.

You are never testing (or mocking) implementations, you are testing (and mocking) interfaces.

If I have a real class X that implements the interface X1, I can write a mock XM that also complies with X1. Then my class A must use something that implements X1, which can either be class X or mock XM.

Now, suppose we change X to implement a new interface X2. Well, obviously my code no longer compiles. A requires something that implements X1, and that no longer exists. The issue has been identified and can be fixed.

Suppose instead of replacing X1, we just modify it. Now class A is all set. However, the mock XM no longer implements the interface X1. The issue has been identified and can be fixed.

The whole basis for unit testing and mocking is that you write code that uses interfaces. A consumer of an interface doesn't care how the code is implemented, only that the same contract is adhered to (inputs/outputs).

This breaks down when your methods have side effects, but I think that can safely be excluded as "cannot be unit tested or mocked".

Vlad274
  • 295
  • 2
  • 7
  • 11
    This answer makes lots of assumptions that need not hold. Firstly, it assumes, roughly, that we are in C# or Java (or, more precisely, that we're in a compiled language, that the language has interfaces, and that X implements an interface; none of these need be true). Secondly, it assumes that any change to the behaviour or "contract" of X requires a change to the *interface* (as understood by the compiler) that X implements. This is plainly *not* true, even if we *are* in Java or C#; you can change a method implementation without changing its signature. – Mark Amery Mar 28 '18 at 12:33
  • 7
    @MarkAmery It's true that the "interface" terminology is more specific to C# or Java, but I think the point stands assuming a defined "contract" of behavior (and if that isn't codified then automatically detecting this is impossible). You're also completely correct that an implementation can be changed without a change to the contract. But a change to implementation without a change in interface (or contract) shouldn't impact any consumer. If the behavior of A depends on how the interface (or contract) is implemented, then it's impossible to (meaningfully) unit test. – Vlad274 Mar 28 '18 at 12:47
  • 1
    *"You're also completely correct that an implementation can be changed without a change to the contract"* - while also true, this isn't the point I was trying to make. Rather, I'm asserting a distinction between the *contract* (a programmer's understanding of what an object is supposed to do, perhaps specified in documentation) and the *interface* (a list of method signatures, understood by the compiler) and saying that the contract can be changed without changing the interface. All functions with the same signature are interchangeable from the type system's perspective, but not in reality! – Mark Amery Mar 28 '18 at 12:55
  • 1
    @Vlad274 imagine a date class. If I change the implementation from Gregorian to Julian, the consumer is sure going to notice, even though the compiler won't. – gbjbaanb Mar 28 '18 at 16:42
  • But _can_ you really write a mock XM that complies with X? To really comply with X, MX must be a _complete implementation of X_, which is exactly the problem you were solving writing X1 in the first place. If an algorithmic change is made to A that means it uses X differently but to the same end result, A should still pass all of its unit tests. To take a more concrete example, the contract isn't "This method will call snprintf exactly this number of times with exactly these parameters in this particular order" but "This method will return this string under these conditions". – AJMansfield Mar 28 '18 at 20:47
  • 1
    @Vlad274 _"You are never testing (or mocking) implementations, you are testing (and mocking) interfaces."_ This is wrong, you don't "test" interfaces, there's nothing to test in an interface because interfaces have no implementation in code. You test an implementation and you pass it any dependencies by mocking the interfaces of the dependencies. – Nick Coad Mar 29 '18 at 03:26
  • 5
    @MarkAmery: I don't think Vlad is using the word "interface" in the same sense as you're using it; the way I read the answer, it's not talking about interfaces in the narrow C#/Java sense (i.e. a set of method signatures) but in the general sense of the word, as used e.g. in the terms "application programming interface" or even "user interface". [...] – Ilmari Karonen Mar 29 '18 at 14:33
  • 2
    ... That said, replacing the word "interface" in the answer with, say, "contract" would probably help avoid some confusion from people more used to the C#/Java sense of the word, despite the arguable loss of precision (as an interface, in the general sense, is more properly a set of related contracts). – Ilmari Karonen Mar 29 '18 at 14:35
  • 6
    @IlmariKaronen If Vlad is using "interface" to mean "contract" rather than in the narrow C#/Java sense, then the statement *"Now, suppose we change X to implement a new interface X2. Well, obviously my code no longer compiles."* is just plain false, since you can change a contract without changing any method signatures. But honestly, I think the problem here is that Vlad isn't using either meaning consistently, but is *conflating* them - which is what leads down the path of claiming that any change to contract X1 will necessarily cause a compilation error without *noticing* that that's false. – Mark Amery Mar 29 '18 at 14:42
  • 1
    "You are never testing (or mocking) implementations, you are testing (and mocking) interfaces" -> I have small utility classes that don't require interfaces, which still need testing, what then? Convert them to one method long interface? KISS. – PmanAce Mar 29 '18 at 16:58
  • Ive read all your answers and somehow accept,I seem to agree with Mark Amery but I am still a little bit lost, just one simple example, just imagine a column in a database changes its name, I mock the returned column names from the database. Till the tested methods gets into the direct database call, its get its data passed in 3 different classes lets say. I might push the mocking further unit testing across 3 classes, needing much more mocks, but eventually I wont be able to find out once the furthest point mocked side changed, integration test might not get into that execution branch neither – FantomX1 Oct 06 '19 at 22:18
9

Taking your questions in turn:

what value does unit testing have then

They're cheap to write and run and you get early feedback. If you break X, you'll find out more or less immediately if you have good tests. Don't even consider writing integration tests unless you've unit tested all your layers (yes, even on the database).

Does it really only tell you that when all tests pass, you haven't introduced a breaking change

Having tests that pass could actually tell you very little. You may not have written enough tests. You may not have tested enough scenarios. Code coverage can help here but it isn't a silver bullet. You may have tests that always pass. Hence red being the often overlooked first step of red, green, refactor.

And when some class's behaviour changes (willingly or unwillingly), how can you detect (preferably in an automated way) all the consequences

More testing - although tools are getting better and better. BUT you should be defining class behaviour in an interface (see below). N.B. there will always be a place for manual testing atop the testing pyramid.

Shouldn't we focus more on integration testing?

Ever more integration tests are not the answer either, they're expensive to write, run and maintain. Depending on your build setup, your build manager may exclude them anyway making them reliant on a developer remembering (never a good thing!).

I've seen developers spend hours trying to fix broken integration tests they'd have found in five minutes if they had good unit tests. Failing this, try just running the software - that is all your end users will care about. No point having million unit tests that pass if the whole house of cards falls down when the user runs the entire suite.

If you want to make sure class A consumes class X in the same way, you should be using an interface rather than a concretion. Then a breaking change is more likely to be picked up at compile time.

Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
9

That's correct.

Unit tests are there to test the isolated functionality of a unit, a first-glance check that it works as intended and doesn't contain stupid errors.

Unit tests are not there to test that the entire application works.

What a lot of people forget is that unit tests are just the quickest and dirtiest means of validating your code. Once you know your little routines work, you then have to run Integration tests as well. Unit testing by itself is only marginally better than no testing.

The reason we have unit tests at all is that they're supposed to be cheap. Quick to create and run and maintain. Once you start turning them into min integration tests, you're into a world of pain. You might as well go full Integration test and ignore unit testing altogether if you're going to do that.

now, some people think that a unit isn't just a function in a class, but the whole class itself (myself included). However, all this does is increase the size of the unit so you might need less integration testing, but you still need it. It is still impossible to verify your program does what it is supposed to do without a full integration test suite.

and then, you'll still need to run the full integration testing on a live (or semi-live) system to check that it works with the conditions the customer uses.

gbjbaanb
  • 48,354
  • 6
  • 102
  • 172
2

Unit tests do not prove the correctness of anything. This is true for all tests. Usually unit tests are combined with contract-based design (design by contract is another way to say it) and possibly automated correctness proofs, if correctness needs to be verified on a regular basis.

If you have real contracts, consisting of class invariants, preconditions, and post conditions, it is possible to prove correctness hierarchically, by basing the correctness of higher level components on the contracts of lower level components. This the fundamental concept behind design by contract.

Frank Hileman
  • 3,922
  • 16
  • 18
  • *Unit tests do not prove the correctness of anything*. Not sure I understand this, unit tests check their own results surely? Or did you mean a behaviour cannot be proved correct since this may encompass multiple layers? – Robbie Dee Mar 28 '18 at 17:56
  • 7
    @RobbieDee I guess, he meant that when you test for `fac(5) == 120`, you have **not** proven that `fac()` does indeed return the factorial of its argument. You have only proven that `fac()` returns the factorial of five when you pass in `5`. And even that is not certain, as `fac()` could conceivably return `42` instead on the first mondays after a total eclipse in Timbuktu... The problem here is, that you cannot prove compliance by checking individual test inputs, you would need to check **all** possible inputs, and also prove that you have not forgotten any (like reading the system clock). – cmaster - reinstate monica Mar 29 '18 at 10:08
  • 1
    @RobbieDee Tests (including unit tests) are a poor substitute, often the best available, for the real goal, a machine checked proof. Consider the entire state space of the units under test, including the state space of any components or mocks therein. Unless you have a very restricted state space, tests cannot cover that state space. Complete coverage would be a proof, but this is only available for tiny state spaces, for example, testing a single object containing a single mutable byte or 16 bit integer. Automated proofs are vastly more valuable. – Frank Hileman Mar 29 '18 at 15:56
  • 1
    @cmaster You summarized the difference between a test and a proof very well. Thanks! – Frank Hileman Mar 29 '18 at 15:57
2

I find heavily mocked tests rarely useful. Most of the time, I end up reimplementing behaviour which the original class already has, which totally defeats the purpose of mocking.

M.e. a better strategy is to have good separation of concerns (e.g. you can test Part A of your app without bringing in parts B through Z). Such a good architecture really helps to write good test.

Also, I am more than willing to accept side effects as long as I can roll them back, e.g. if my method modifies data in the db, let it! As long as I can roll the db back to the previous state, what's the harm? Also, there is the benefit that my test can check if the data looks as expected. In-Memory DBs or specific test-versions of dbs really help here (e.g. RavenDBs in-memory test version).

Finally, I do like to do mocking on service boundaries, e.g. do not make that http call to service b, but let's intercept it and introduce an appropiate

Christian Sauer
  • 1,269
  • 1
  • 9
  • 16
1

I wish people in both camps would understand that class testing and behaviour testing are not orthogonal.

Class testing and unit testing are used interchangeably and they perhaps shouldn't be. Some unit tests just happen to be implemented in classes. That is all. Unit testing has happened for decades in languages without classes.

As for testing behaviours, it is perfectly possible to do this within class testing using the GWT construct.

Furthermore, whether your automated tests proceed along class or behaviour lines rather depends on your priorities. Some will need to rapidly prototype and get something out the door while others will have coverage constraints due to in house styles. Many reasons. They're both perfectly valid approaches. You pays your money, you takes your choice.

So, what to do when code breaks. If it has been coded to an interface, then just the concretion needs to change (along with any tests).

However, introducing a new behaviour needn't compromise the system at all. Linux et al are full of deprecated features. And things like constructors (and methods) can happily be overloaded without forcing all calling code to change.

Where class testing wins is where you need to make a change to a class that hasn't yet been plumbed in yet (due to time constraints, complexity or whatever). It is just so much easier to get started with a class if it has comprehensive tests.

Belgian Dog
  • 85
  • 2
  • Where you have *concretion* I would have written *implementation*. Is this a calque from another language (I would guess French) or is there an important distinction between *concretion* and *implementation*? – Peter Taylor Mar 29 '18 at 11:44
0

Unless the interface for X changed, you do not need to change the unit test for A because nothing related to A has changed. It sounds like you really wrote a unit test of X and A together, but called it a unit test of A:

When you write unit tests for A, you mock X. In other words, while unit testing A, you set (postulate) the behaviour of X's mock to be X1.

Ideally, the mock of X should simulate all possible behaviors of X, not just the behavior you expect from X. So no matter what you actually implement in X, A should already be capable of handling it. So no change to X, other than changing the interface itself, will have any effect on the unit test for A.

For example: Suppose A is a sorting algorithm, and A provides data to be sorted. The mock of X should provide a null return value, an empty list of data, a single element, multiple elements already sorted, multiple elements not already sorted, multiple elements sorted backwards, lists with the same element repeated, null values intermingled, a ridiculously large number of elements, and it should also throw an exception.

So perhaps X initially returned sorted data on Mondays and empty lists on Tuesdays. But now when X returns unsorted data on Mondays and throws exceptions on Tuesdays, A doesn't care -- those scenarios were already covered in A's unit test.

Moby Disk
  • 230
  • 2
  • 8
  • what if X just renamed index in returned array from foobar to foobarRandomNumber, how can I count with that? if you get my point, this is basically my issue, I renamed a returned column from secondName to surname, a classic task, but my test will never know, since its mocked. I just have such a strange feeling as if many people in this question never actually tried something like that, before commenting – FantomX1 Oct 06 '19 at 22:25
  • The compiler should have detected this change and given you a compiler error. If you are using something like Javascript, then I recommend either switching to Typescript or using a compiler like Babel which can detect this stuff. – Moby Disk Oct 13 '19 at 15:07
  • What if I am using arrays in PHP, or in Java or in Javascript, if you change an index of array or remove it, non of those languages compilers will tell you about it, the index might be nested at the 36th - thinked out number nested level of the array, therefore I think compiler is not the solution for this. – FantomX1 Oct 13 '19 at 17:28
-2

You have to look at different tests.

Unit tests themselves will only test X. They are there to prevent you changing the behavior of X but not secure the whole system. They make sure you can refactor you class without introducing a change in behavior. And if you break X, you broke it...

A should indeed mock X for its unit tests and the test with the Mock should keep passing even after you change it.

But there is more than one level of testing! There are also integration tests. These tests are there to validate the interaction between the classes. These tests usually have a higher price since they don't use mocks for everything. For example an integration test might actually write a record into a database, and a unit test should have no external dependencies.

Also if X needs to have a new behavior, it would be better to supply a new method that would supply the desired outcome

Heiko Hatzfeld
  • 97