530

In the comment to this great post, Roy Osherove mentioned the OAPT project that is designed to run each assert in a single test.

The following is written on the project's home page:

Proper unit tests should fail for exactly one reason, that’s why you should be using one assert per unit test.

And, also, Roy wrote in comments:

My guideline is usually that you test one logical CONCEPT per test. you can have multiple asserts on the same object. they will usually be the same concept being tested.

I think that, there are some cases where multiple assertions are needed (e.g. Guard Assertion), but in general I try to avoid this. What is your opinion? Please provide a real world example where multiple asserts are really needed.

J. Mini
  • 997
  • 8
  • 20
Restuta
  • 437
  • 3
  • 6
  • 14
  • 3
    How do you do mocking without having multiple assertions? Each expectation on the mock is an assertion in itself, including any order of calls you impose. – Christopher Creutzig Mar 15 '13 at 19:05
  • 28
    I've seen the one-assert-per-method philosophy abused in the past. An old co-worker used a funky inheritance mechanism to make this possible. It led to a lot of sub-classes (one per branch) and lots of tests that did the same set-up/tear-down process only to check the different results. It was slow, hard to read and a severe maintenance problem. I never convinced him to switch back to a more classic approach. [Gerard Meszaros book](http://www.amazon.com/xUnit-Test-Patterns-Refactoring-Code/dp/0131495054) talks about this topic in detail. – Travis Parks Mar 15 '13 at 19:39
  • 5
    I think as a general rule of thumb you should try to minimize the number of asserts per test. However, as long as the test sufficiently narrows the problem to a specific place in the code, then it's a useful test. – ConditionRacer Mar 29 '13 at 16:12
  • 2
    I've seen cases where multiple asserts were used instead of `RowTest` (MbUnit) / `TestCase` (NUnit) to test a variety of edge-case behaviors. Use the proper tools for the job! (Unfortunately, MSTest doesn't seem to have a row-test capability yet.) – GalacticCowboy Mar 29 '13 at 17:35
  • @GalacticCowboy You can get similar functionality of `RowTest` and `TestCase` using [test data sources](http://msdn.microsoft.com/en-us/library/ms243192.aspx). I'm using a simple CSV file with great success. – julealgon Mar 12 '14 at 15:57
  • See also http://stackoverflow.com/questions/2430429/are-multiple-asserts-bad-in-a-unit-test-even-if-chaining# – CAD bloke Jul 09 '14 at 21:52
  • FWIW, XCT tests in XCode allow multiple asserts within a single test, however they can run each assert regardless of previous failures, so a test with five asserts will get results for the other four even if the first fails – GoatInTheMachine Dec 12 '18 at 16:51

18 Answers18

448

Tests should fail for one reason only, but that doesn't always mean that there should be only one Assert statement. IMHO it is more important to hold to the "Arrange, Act, Assert" pattern.

The key is that you have only one action, and then you inspect the results of that action using asserts. But it is "Arrange, Act, Assert, End of test". If you are tempted to continue testing by performing another action and more asserts afterwards, make that a separate test instead.

I am happy to see multiple assert statements that form parts of testing the same action. e.g.

[Test]
public void ValueIsInRange()
{
  int value = GetValueToTest();

  Assert.That(value, Is.GreaterThan(10), "value is too small");
  Assert.That(value, Is.LessThan(100), "value is too large");
} 

or

[Test]
public void ListContainsOneValue()
{
  var list = GetListOf(1);

  Assert.That(list, Is.Not.Null, "List is null");
  Assert.That(list.Count, Is.EqualTo(1), "Should have one item in list");
  Assert.That(list[0], Is.Not.Null, "Item is null");
} 

You could combine these into one assert, but that's a different thing from insisting that you should or must. There is no improvement from combining them.

e.g. The first one could be

Assert.IsTrue((10 < value) && (value < 100), "Value out of range"); 

But this is not better - the error message out of it is less specific, and it has no other advantages. I'm sure you can think of other examples where combining two or three (or more) asserts into one big boolean condition makes it harder to read, harder to alter and harder to work out why it failed. Why do this just for the sake of a rule?

NB: The code that I am writing here is C# with NUnit, but the principles will hold with other languages and frameworks. The syntax may be very similar too.

James Skemp
  • 115
  • 6
Anthony
  • 221
  • 2
  • 6
  • 12
  • 71
    The key is that you have only one action, and then you inspect the results of that action using asserts. – 王奕然 Aug 09 '15 at 12:00
  • 1
    I think it is also interesting, to have more as one Assert, if Arrange is time expensive. – Rekshino May 12 '17 at 07:18
  • 2
    @Rekshino, if arrange is time expensive, we can share the arrange code, for instance, by putting the arrange code in the test initialization routine. – Shaun Luttin Feb 24 '18 at 00:43
  • @ShaunLuttin After test has run(after action) not always you can reuse the arrangement.. – Rekshino Feb 24 '18 at 15:10
  • 3
    So if I compare to Jaco answer, the "only one assert" become "only one group of asserts" which make more sense to me. – Walfrat Apr 19 '18 at 07:42
  • 2
    This is a good answer, but I disagree that a single assert isn't better. The bad assert example isn't better, but but that doesn't mean a single assert wouldn't be better if done right. Many libraries allow custom asserts/matchers so something could be created if not already present. For example `Assert.IsBetween(10, 100, value)` that prints `Expected 8 to be between 10 and 100` *is* better than two separate asserts in my opinion. You can certainly argue it isn't necessary, but it's usually worth considering whether it's easy to reduce to a single assert before making a whole set of them. – Vala Nov 15 '18 at 08:42
  • @Thor84no Hmm, I don't disagree with your comment--but I don't fully agree. I think it largely depends on context. But I appreciate the comment very much. I think it's something to consider when making tests that could potentially fail and tying the failing test back to requirements, e.g. `Expected 8 to be between 10 and 100`. Unless, the bounds of the range are two separate requirements. See? Context. :wink: Anyway, thanks again for the thought-provoking comment. – fourpastmidnight Jun 21 '19 at 16:21
  • 1
    @Thor84no The more complicated the condition in an assert, the better the chance to get it wrong. Having an assertion when the code is correct is easy to fix, but having an assert that doesn't find the bugs, that's quite fatal. – gnasher729 May 10 '22 at 07:09
  • @gnasher729 I don't see how that's _in any way_ incompatible with my statement that "it's usually worth considering whether it's easy to reduce to a single assert before making a whole set of them". Emphasis on _easy_. – Vala May 11 '22 at 10:58
305

I don't think it's necessarily a bad thing, but I do think we should strive towards only having single asserts in our tests. This means you write a lot more tests and our tests would end up testing only one thing at a time.

Having said that, I would say maybe half of my tests actually only have one assert. I think it only becomes a code (test?) smell when you have about five or more asserts in your test.

How do you solve multiple asserts?

Jaco Pretorius
  • 4,057
  • 2
  • 27
  • 38
  • 12
    Like this answer - i.e. its OK, but its not, in the general instance, *good* (-: – Murph Sep 28 '10 at 11:45
  • 2
    I do it a bit. For example, if I'm testing comparability, and that `ItemA > ItemB` I'll also assert that `ItemB < ItemA` in the same test. – CaffGeek Oct 26 '12 at 13:53
  • 2
    Passing an argument into a single argument method can result in several properties being changed of that object. Hence a simple operation can require multiple asserts. Take `Stream.Write` – jgauffin Oct 30 '12 at 14:00
  • @jgauffin wouldn't those be seperate tests though? eg. SomeMethod_Alters_FooProperty(), SomeMethod_Alters_BarProperty() etc. The unit tests would be identical apart from the assertion. – MattDavey Oct 30 '12 at 15:03
  • 5
    Ehh? Why would you do that? the method execution is exactly the same? – jgauffin Oct 30 '12 at 15:08
  • 328
    A single assert per unit test is a great way to test the reader's ability to scroll up and down. – Tom Oct 30 '12 at 15:45
  • bad thing strive towards only having single asserts code (test?) smell – Thomas Eding Nov 06 '12 at 23:17
  • @Tom, I have as a requirement of my test runners that they link to the code - e.g. If i double click on a failed test it should take me straight to the code file. Console runners obviously are more difficult! – Byron Ross Apr 08 '14 at 23:25
  • @jgauffin It depends whether you are using unit test to mechanically execute the code or to define the expected behavior. If you take the behavior approach (which I prefer) you end up with 2 tests. – Byron Ross Apr 08 '14 at 23:27
  • a case where you need multiple assertions for me is when i need to test math/geometry functions. i add assertions for the basic cases, corner cases, and then a couple random ones. so if i need to test some point conversion between views, for example, i will add the points to the corners of the view, a point inside the view, and a point outside the view. – pvinis Jul 17 '15 at 11:30
  • 8
    Any reasoning behind this? As is, this current answer just states what it should be, but not why. – Steven Jeuris Dec 22 '15 at 13:36
  • 81
    Strongly disagree. The answer doesn't list any advantage of having single assert, and the obvious disadvantage is that you need to copy-paste tests only because an answer on the internet says so. If you need to test multiple fields of a result or multiple results of a single operation, you absolutely should assert all of them in independent asserts, because it gives far more useful information than testing them in a big blob-assert. In a good test, you test a single operation, not a single result of an operation. – Peter Jun 02 '16 at 12:18
  • 6
    woah, it's a code smell to have 5 or more asserts? definitely not and depends on the situation. – dtc Jan 29 '19 at 17:09
  • Link returns a 404 – ToastyMallows Oct 24 '19 at 12:52
  • @ToastyMallows The link works fine for me? http://www.owenpellegrin.com/blog/testing/how-do-you-solve-multiple-asserts/ – Jaco Pretorius Oct 25 '19 at 13:21
  • @JacoPretorius ah my HTTPS Everywhere extension was forcing the link to HTTPS, which is a 404. HTTP works just fine. – ToastyMallows Oct 28 '19 at 14:08
  • 4
    You did not explain why `we should strive towards only having single asserts`. – OSGI Java Oct 06 '20 at 23:31
  • 5
    Disagree with this. If the code leading to an assertion is elaborate, it makes little sense to repeat that for the sake of perceived single-assertion purity. Perfectly acceptable to have multiple assertions in a test if they're all related. – Lee Benson Nov 09 '20 at 08:56
  • @tom Maybe the class itself is too big then? – Artur INTECH Mar 19 '21 at 16:03
  • @Artur Maybe it's not? Maybe someone is just doing extensive testing? – gnasher729 May 10 '22 at 07:07
  • Definitely not the case. I'd say 1 set of asserts per test case. But for tests that require a lot of setup I'm basically putting the entire test into a separate method, then calling it for a million specific tests. May as well just have a theory test with different sets of data making up the test cases. And in the case of a failing test I'd be investigating what made it fail. I don't need visibility of all asserts separately because once there's a failing test I'll be investigating what to do to fix it anyway. – TheEvilMetal Jun 15 '23 at 07:39
97

I have never thought that more than one assert was a bad thing.

I do it all the time:

public void ToPredicateTest()
{
    ResultField rf = new ResultField(ResultFieldType.Measurement, "name", 100);
    Predicate<ResultField> p = (new ConditionBuilder()).LessThanConst(400)
                                                       .Or()
                                                       .OpenParenthesis()
                                                       .GreaterThanConst(500)
                                                       .And()
                                                       .LessThanConst(1000)
                                                       .And().Not()
                                                       .EqualsConst(666)
                                                       .CloseParenthesis()
                                                       .ToPredicate();
    Assert.IsTrue(p(ResultField.FillResult(rf, 399)));
    Assert.IsTrue(p(ResultField.FillResult(rf, 567)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 400)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 666)));
    Assert.IsFalse(p(ResultField.FillResult(rf, 1001)));

    Predicate<ResultField> p2 = (new ConditionBuilder()).EqualsConst(true).ToPredicate();

    Assert.IsTrue(p2(new ResultField(ResultFieldType.Confirmation, "Is True", true)));
    Assert.IsFalse(p2(new ResultField(ResultFieldType.Confirmation, "Is False", false)));
}

Here I use multiple asserts to make sure complex conditions can be turned into the expected predicate.

I am only testing one unit (the ToPredicate method), but I am covering everything I can think of in the test.

Alexander
  • 378
  • 1
  • 5
  • 14
Matt Ellen
  • 3,368
  • 4
  • 30
  • 37
  • 59
    Multiple asserts are bad because of error detection. If you have failed your first Assert.IsTrue, other asserts will not be executed, and you won't get any information from them. On other hand if you had 5 tests instead of 1 with 5 asserts you could get something usefull – Sly Sep 28 '10 at 10:52
  • 8
    Do you consider it still bad if all the asserts test the same kind of functionality? Like above, the example tests the conditionals and if any of this fails, you should fix it. Does it matter to you that you may miss the last 2 asserts if an earlier one fails? – cringe Sep 28 '10 at 10:55
  • 131
    I fix my problems one at a time. So the fact that the test could fail more than once doesn't bother me. If I split them up I would have the same errors come up, but all at once. I find it easier to fix things a step at a time. I admit, that in this instance, the last two assert could probably be refactored into their own test. – Matt Ellen Sep 28 '10 at 11:00
  • 22
    Your case is very representative, that is why NUnit has additional attribute TestCase - http://nunit.org/?p=testCase&r=2.5 – Restuta Sep 28 '10 at 11:12
  • @Restuta: I had no idea you could do that! I'm using MbUnit, and have found a similar attribute: [RowAttribute](http://www.gallio.org/api/html/T_MbUnit_Framework_RowAttribute.htm), so I can compress the above test, after the refactoring the last two asserts. – Matt Ellen Sep 28 '10 at 12:03
  • 1
    @Sly: if the testing tool bails out when at the first failed assert, then you're right. If the tool continues until an unrecoverable condition (unexpected exception), multiple asserts would be valid. – JBRWilkinson Sep 28 '10 at 23:21
  • 1
    Can you give an example of a tool that continues? – Restuta Feb 06 '11 at 08:47
  • 14
    google C++ testing framework has ASSERT() and EXPECT(). The ASSERT() stops on failure while EXPECT() continue. That's very handy when you want to validate more than one thing in the test. – ratkok Jun 25 '11 at 04:28
  • 3
    @Sly: Yes, I don't get information from the later asserts. Should I care? When I fix the first problem, most probably the other asserts will pass as well. If not, I'll fix them. In the - very rare - cases I need more information at the beginning I can write an additional test. – maaartinus Aug 15 '11 at 12:02
  • 2
    you should care about other tests, because it's useful to know how many assertions you broke by your change. And to see that full picture for the first time, not step-by-step fixing assertions one-by-one. – Restuta Mar 22 '12 at 01:42
  • 2
    @Sly Tho this problem is being mitigated by some frameworks (Google Tests is the only major one I know of) which are now implementing the concept of `expects`. A test can have multiple `expects` and if they all fail, they will all be shown as failing. – Kazark Mar 29 '13 at 17:44
  • 3
    @Restuta: It's useful to know if the number of failures that you created is 0 or not. The number is of very little importance. – gnasher729 Jun 02 '16 at 09:27
  • @gnasher729 it would be useful if you specified why, I could disagree in more constructive manner. In the meantime I'll try to explain why number is of _big_ importance. When you make a change and break 1 test it may suggest that it's a minor break and easy to fix. And when you see you broke 5, it may suggest that this is more things to fix and tell what what exactly broke to give your more context, so you can take more informed decision. So not only number of tests matters, but test names and failure messages do as well. – Restuta Jun 02 '16 at 21:09
  • 3
    @Restuta: If I break one thing then it's broken and needs fixing. If I break five things then it's broken and needs fixing. No difference. Five tests with one assert each, or one test with five asserts, I learn about the breakage and that's the important thing. – gnasher729 Jun 11 '16 at 15:40
  • I think the point is to have the tests be useful to other team members too. Possibly someone who just started. If they do a change that breaks something elsewhere (encapsulation notwithstanding), it could be useful to show all the errors. – Juha Untinen Jun 16 '16 at 06:03
  • 4
    i think that Judging the severity of the failure by how many tests have broken is like measuring productivity by lines of code. I can write 40 tests that test the same functionality with different values. If this single functionality fails that wouldn't mean that its 40 times bad! I can just test it with 1 assert of one value! but well adding some more assert values would just make the test more secure with minor overhead. – Anestis Kivranoglou Oct 31 '17 at 11:09
22

When I'm using unit testing to validate high-level behavior, I absolutely put multiple assertions into a single test. Here's a test I'm actually using for some emergency notification code. The code that runs before the test puts the system into a state where if the main processor gets run, an alarm gets sent.

@Test
public void testAlarmSent() {
    assertAllUnitsAvailable();
    assertNewAlarmMessages(0);

    pulseMainProcessor();

    assertAllUnitsAlerting();
    assertAllNotificationsSent();
    assertAllNotificationsUnclosed();
    assertNewAlarmMessages(1);
}

It represents the conditions that need to exist at every step in the process in order for me to be confident that the code is behaving the way I expect. If a single assertion fails, I do not care that the remaining ones won't even get run; because the state of the system is no longer valid, those subsequent assertions wouldn't tell me anything valuable.* If assertAllUnitsAlerting() failed, then I wouldn't know what to make of assertAllNotificationSent()'s success OR failure until I determined what was causing the prior error and corrected it.

(* -- Okay, they might conceivably be useful in debugging the problem. But the most important information, that the test failed, has already been received.)

BlairHippo
  • 8,663
  • 5
  • 41
  • 46
  • When you do that you you should better using testing frameworks with dependant tests, it is better (e.g. testng supports this feature) – Kemoda Oct 26 '12 at 13:44
  • 9
    I write tests like this too so that you can be confident of what the code is doing and state changes, I don't think this is a unit test, but an integration test. – mdma Sep 11 '13 at 09:30
  • What are your opinions about refactoring it into a assertAlarmStatus( int numberOfAlarmMessages);? – Borjab Mar 07 '16 at 16:37
  • 2
    Your asserts would make for very nice test names. – Bjorn Oct 21 '16 at 18:38
  • It's better to let the test run, even on invalid input. It gives you more information that way (and especially if it still passes when you didn't expect it to). – CurtainDog Apr 19 '18 at 04:47
15

I think there are plenty of cases where writing multiple asserts is valid within the rule that a test should only fail for one reason.

For example, imagine a function that parses a date string:

function testParseValidDateYMD() {
    var date = Date.parse("2016-01-02");

    Assert.That(date.Year).Equals(2016);
    Assert.That(date.Month).Equals(1);
    Assert.That(date.Day).Equals(0);
}

If the test fails it is because of one reason, the parsing is incorrect. If you would argue that this test can fail for three different reasons, you would IMHO be too fine grained in your definition of "one reason".

Pete
  • 8,916
  • 3
  • 41
  • 53
  • You can have one date object as seen in your code and 3 separate tests using it. The you can have 3 tests: 1. expect year to be equal 2. expect month to be equal 3. expect day to be equal The advantage is that there would be only one reason for failing these individual tests: 1. year parser is broke 2. month parser is broke 3. day parser is broke The bottom line is that this is not a good example to support your claim. – OSGI Java Oct 06 '20 at 23:39
  • @OSGIJava that implies that you have three tests, *each* of them calling the same method before making its one assertion. Likewise, if I have a single method that extracts data from a worksheet and I want to check that it comes up with exactly the 177 values I expect it to, instead of running the extraction once and then checking the 177 values, I'd have to have rerun my test setup and call the extraction method *177 times*, and assert only one of those facts after each, just so I can say I'm following someone's recommended paradigm for unit testing. – Green Grasso Holm Jul 02 '21 at 18:13
  • Alternatively, I can call the extraction method just once, in the test setup code, but then I'm breaking a different paradigm: the one under which each test method is structured as arrange, act, assert: I'd only be asserting, with all the arranging and acting happening in the setup code. Paradigms at loggerheads. – Green Grasso Holm Jul 02 '21 at 18:16
  • @GreenGrassoHolm Would you please provide a link to the paradigm you mentioned? Alternatively, you could write your own custom `Assert` which would compare all 3 date units and print out the parts which fail the test. So it would be a single `Assert` printing out all the failures. – OSGI Java Jul 03 '21 at 19:27
  • @OSGIJava This is the mentioned paradigm "You can have one date object as seen in your code and 3 separate tests using it. " Which GreenGrassoHolm interpreted as "start with string, do date.parse _in each test_ to test one 'thing' in result". Hence "extract data from worksheet _in each test_ to test one row in result". Did you mean the Date.Parse to take place in the testing fixture, and the three tests to only consist of assertions? Or first a test that parsing succeeds, storing the Date for other tests? Either would break "Arrange, Act, Assert" like crazy. – Wolfzoon Aug 03 '21 at 09:12
8

Another reason why I think, that multiple asserts in one method is not a bad thing is described in following code:

class Service {
    Result process();
}

class Result {
    Inner inner;
}

class Inner {
    int number;
}

In my test I simply want to test that service.process() returns the correct number in Inner class instances.

Instead of testing...

@Test
public void test() {
    Result res = service.process();
    if ( res != null && res.getInner() != null ) Assert.assertEquals( ..., res.getInner() );
}

I'm doing

@Test
public void test() {
    Result res = service.process();
    Assert.notNull(res);
    Assert.notNull(res.getInner());
    Assert.assertEquals( ..., res.getInner() );
}
Martijn Pieters
  • 14,499
  • 10
  • 57
  • 58
Betlista
  • 349
  • 3
  • 10
  • 2
    And that's a good thing, you shouldn't have any conditional logic in your tests. It makes test more complex and less readable. And I guess Roy outlined it right in his blog post that multiple asserts on one object is okay most of the time. So those that you have are just guard asserts and it's okay to have them. – Restuta Oct 30 '12 at 20:44
  • 2
    Your `Assert.notNull`s are redundant, your test will fail with an NPE if they're null. – sara Jun 02 '16 at 10:01
  • 1
    Also, your first example (with the `if`) will pass if `res` is `null` – sara Jun 02 '16 at 10:02
  • 3
    @kai notNull's are redundant, I agree, but I feel that it is cleaner to have the assert (and if I'm not lazy also with proper message) instead of exception... – Betlista Jun 02 '16 at 13:50
  • 1
    Well a failed assertion also throws an exception, and in both cases you get a direct link to the exact row that threw it with an accompanying stack trace so personally I'd rather not clutter the test with preconditions that will get checked anyway. I'd prefer a oneliner à la `Assert.assertEquals(..., service.process().getInner());`, possible with extracted variables if the line gets "too long" – sara Jun 02 '16 at 13:53
5

The goal of the unit test is to give you as much information as possible about what is failing but also to help accurately pinpoint the most fundamental problems first. When you know logically that one assertion will fail given that another assertion fails or in other words there is a dependency relationship between the test then it makes sense to roll these as multiple asserts within a single test. This has the benefit of not littering the test results with obvious failures which could have been eliminated if we bailed out on the first assertion within a single test. In the case where this relationship does not exist the preference would naturally be then to separate these assertions into individual tests because otherwise finding these failures would require multiple iterations of test runs to work out all of the issues.

If you then also design the units/classes in such a way that overly complex tests would need to be written it makes less burden during testing and probably promotes a better design.

jpierson
  • 301
  • 3
  • 7
4

I don't know of any situation where it would be a good idea to have multiple assertions inside the [Test] method itself. The main reason people like to have multiple Assertions is they are trying to have a one [TestFixture] class for each class being tested. Instead, you can break up your tests into more [TestFixture] classes. This allows you to see multiple ways in which the code may not have reacted in the way you expected, instead of just the one where the first assertion failed. The way you achieve this is you have at least one directory per class being tested with lots of [TestFixture] classes inside. Each [TestFixture] class would be named after the specific state of an object you will be testing. The [SetUp] method will get the object into the state described by the class name. Then you have multiple [Test] methods each asserting different things you would expect to be true, given the object's current state. Each [Test] method is named after the thing it is asserting, except perhaps it might be named after the concept instead of just an English readout of the code. Then each [Test] method implementation only needs a single line of code where it is asserting something. Another advantage of this approach is it makes the tests very readable as it becomes quite clear what you are testing, and what you expect just by looking at the class and method names. This will also scale better as you begin to realize all the little edge cases you want to test and as you find bugs.

Usually this means that the final line of code inside the [SetUp] method should store a property value or return value in a private instance variable of the [TestFixture]. Then you might assert multiple different things about this instance variable from different [Test] methods. You also might make assertions about what different properties of the object under test are set to now that it is in the desired state.

Sometimes you need to make assertions along the way as you are getting the object under test into the desired state in order to make sure you did not mess up before getting the object into the desired state. In that case, those extra assertions should appear inside the [SetUp] method. If something goes wrong inside the [SetUp] method, it will be clear that something was wrong with the test before the object ever got into the desired state you intended to test.

One other problem you may run into is you may be testing an Exception that you expected to be thrown. This can tempt you into not following the above model. However, it can still be achieved by catching the exception inside the [SetUp] method and storing it into an instance variable. This will allow you to assert different things about the exception, each in it's own [Test] method. You can then also assert other things about the object under test to make sure there were no unintended side effects from the exception being thrown.

Example (this would be broken up into multiple files):

namespace Tests.AcctTests
{
    [TestFixture]
    public class no_events
    {
        private Acct _acct;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
        }

        [Test]
        public void balance_0() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }

    [TestFixture]
    public class try_withdraw_0
    {
        private Acct _acct;
        private List<string> _problems;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
            Assert.That(_acct.Balance, Is.EqualTo(0));
            _problems = _acct.Withdraw(0m);
        }

        [Test]
        public void has_problem() {
            Assert.That(_problems, Is.EquivalentTo(new string[] { "Withdraw amount must be greater than zero." }));
        }

        [Test]
        public void balance_not_changed() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }

    [TestFixture]
    public class try_withdraw_negative
    {
        private Acct _acct;
        private List<string> _problems;

        [SetUp]
        public void SetUp() {
            _acct = new Acct();
            Assert.That(_acct.Balance, Is.EqualTo(0));
            _problems = _acct.Withdraw(-0.01m);
        }

        [Test]
        public void has_problem() {
            Assert.That(_problems, Is.EquivalentTo(new string[] { "Withdraw amount must be greater than zero." }));
        }

        [Test]
        public void balance_not_changed() {
            Assert.That(_acct.Balance, Is.EqualTo(0m));
        }
    }
}
still_dreaming_1
  • 261
  • 1
  • 13
  • How do you handle TestCase input in this case? – Simon Gillbee Feb 12 '16 at 21:45
  • So, at my current organization, we have 20,000+ unit tests very similar to what you show. This is a nightmare. Much of the test setup code is copied/pasted, resulting in incorrect test setup and invalid tests which pass. For each `[Test]` method, that class is re-instantiated and the `[SetUp]` method is executed again. This kills the .NET Garbage Collector and causes the tests to run extremely slowly: 5+ minutes locally, 20+ minutes on the build server. 20K tests should run in about 2 - 3 minutes. I would **not** recommend this testing style at all, especially for a large-ish test suite. – fourpastmidnight Jul 09 '18 at 22:03
  • @fourpastmidnight most of what you said seems like valid criticism, but the point about copying and pasting setup code and therefore being wrong, that is not a problem of structure but of irresponsible programmers (which could be the result of irresponsible managers or a negative environment more so than bad programmers). If people just copy and paste code and hope it is correct and don't bother to understand the code, in any context ever for any reason, they need to either be trained not to do this or be let go if they can't be trained. That goes against every good principal of programming. – still_dreaming_1 Jul 10 '18 at 21:31
  • But in general, I agree, this is crazy overkill that will result in lots of bloat/baggage/duplication that will lead to all kinds of problems. I used to be crazy and recommend stuff like this. That's what I hope to be able to say every day about myself the day before because that means I never stop finding better ways of doing things. – still_dreaming_1 Jul 10 '18 at 21:33
  • @still_dreaming_1 wrt "irresponsible programmers": I agree that this behavior is a major problem. However, this sort of test structure really does invite this sort of behavior, for better or worse. Bad development practices aside, my main objection to this form is that it really kills test performance. There's nothing worse than a slow running test suite. A slow running test suite means people won't run tests locally and are even tempted to skip them on intermediate builds--again, a people problem, but it happens--which can all be avoided by ensuring you have fast running tests to start with. – fourpastmidnight Jul 11 '18 at 04:15
  • @still_dreaming_1 - I do this for classes where you want to represent various scenarios on the SUT. I would recommend switching the SetUp to a FixtureSetUp. – Bronumski Jun 20 '19 at 22:55
  • @fourpastmidnight - I disagree that this is bad in all cases. Everything depends on context. 1st these tests work well when testing complex behavior. I would not use on simple functionality. 2nd because the assertions should be side effect free switch Setup for FixtureSetup. 3rd I take issue with the copy pasting. I see copy pasting all the time when one test is copied to create another similar test. Tests should be treated like production code. If we see duplication, refactor it. With this method you can express the context of the test better than you can with a test method name. – Bronumski Jun 20 '19 at 22:58
  • @Bronumski I agree, it depends on context. But I think in most cases, what I wrote still applies. Again, in _most cases_, not in _all cases_. Context is king. I, too, take issue with copy/paste tests. But again, my assertion in my comment was that the _test structure itself_, with all of its ceremony in setup and the highly specific test assertions, leads to many more tests than are necessary (many tests testing one logical unit of behavior), which leads to the _very undesired_ copy/paste behavior. There's no getting around humane nature. – fourpastmidnight Jun 21 '19 at 16:09
  • In my opinion, @Anthony should have received the Accepted Answer, because he's spot on with the intent behind a unit test and it's structure. – fourpastmidnight Jun 21 '19 at 16:17
4

Yes, it is ok to have multiple assertions as long as a failing test gives you enough information to be able to diagnose the failure. This is going to depend on what you're testing and what the failure modes are.

Proper unit tests should fail for exactly one reason, that’s why you should be using one assert per unit test.

I've never found such formulations to be helpful (that a class should have one reason to change is an example of just such an unhelpful adage). Consider an assertion that two strings are equal, this is semantically equivalent to asserting that the length of the two strings are the same and each character at the corresponding index is equal.

We could generalize and say that any system of multiple assertions could be rewritten as a single assertion, and any single assertion could be decomposed into a set of smaller assertions.

So, just focus on the clarity of the code and the clarity of the test results, and let that guide the number of assertions you use rather than vice versa.

CurtainDog
  • 334
  • 1
  • 6
3

If your test fails, you won't know whether the following assertions will break, too. Often, that means you'll be missing valuable information to figure out the source of the problem. My solution is to use one assert but with several values:

String actual = "val1="+val1+"\nval2="+val2;
assertEquals(
    "val1=5\n" +
    "val2=hello"
    , actual
);

That allows me to see all failed assertions at once. I use several lines because most IDEs will display string differences in a compare dialog side-by-side.

Aaron Digulla
  • 2,865
  • 21
  • 22
2

If you have multiple asserts in a single test function, I expect them to be directly relevant to the test you are conducting. For example,

@Test
test_Is_Date_segments_correct {

   // It is okay if you have multiple asserts checking dd, mm, yyyy, hh, mm, ss, etc. 
   // But you would not have any assert statement checking if it is string or number,
   // that is a different test and may be with multiple or single assert statement.
}

Having a lot of tests (even when you feel that it is probably an overkill) is not a bad thing. You can argue that having the vital and most essential tests is more important. SO, when you are asserting, make sure your assert statements are correctly placed rather than worrying about multiple asserts too much. If you need more than one, use more than one.

ha9u63a7
  • 397
  • 2
  • 12
2

The answer is very simple - if you test a function that changes more than one attribute, of the same object, or even two different objects, and the correctness of the function depends on the results of all of those changes, then you want to assert that every one of those changes has been properly performed!

I get the idea of a logical concept, but the reverse conclusion would say that no function must ever change more than one object. But that's impossible to implement in all cases, in my experience.

Take the logical concept of a bank transaction - withdrawing an amount from one bank account in most cases HAS TO include adding that amount to another account. You NEVER want to separate those two things, they form an atomic unit. You might want to make two functions (withdraw/addMoney) and thus write two different unit tests - in addition. But those two actions have to take place within one transaction and you also want to make sure that the transaction works. In that case it's simply not enough to make sure the individual steps were successful. You have to check both bank accounts, in your test.

There might be more complex examples that you wouldn't test in a unit test, in the first place, but instead in an integration or acceptance test. But those boundaries are fluent, IMHO! It's not that easy to decide, it's a matter of circumstances and maybe personal preference. Withdrawing money from one and adding it to another account is still a very simple function and definitely a candidate for unit testing.

cslotty
  • 131
  • 3
2

Having multiple assertions in the same test is only a problem when the test fails. Then you might have to debug the test or analyse the exception to find out which assertion it is that fails. With one assertion in each test it's usually easier to pinpoint what's wrong.

I can't think of a scenario where multiple assertions are really needed, as you can always rewrite them as multiple conditions in the same assertion. It may however be preferrable if you for example have several steps to verify the intermediate data between steps rather than risking that the later steps crash because of bad input.

Guffa
  • 2,990
  • 20
  • 16
  • 3
    If you are combining multiple conditions into a single assert, then on failure all you know is that one failed. With multiple asserts you know specifically about some of them (the ones up to and including the failure). Consider checking a returned array contains a single value: check it is not null, then it has exactly one element and then then value of that element. (Depending on the platform) just checking the value immediately could give a null dereference (less helpful than the null assertion failing) and doesn't check the array length. – Richard Sep 28 '10 at 15:41
  • @Richard: Getting a result and then extracting something from that result would be a process in several steps, so I covered that in the second paragraph in the answer. – Guffa Sep 28 '10 at 16:42
  • 2
    Rule of thumb: if you have multiple assertions in a test, each one should have a different message. Then you don't have this problem. – Anthony Oct 26 '12 at 13:17
  • And using a quality test runner such as NCrunch will show you exactly on what line the test failed, both in the test code and in the code under test. – fourpastmidnight Jul 09 '18 at 21:52
1

This question is related to the classic problem of balancing between spaghetti and lasagna code issues.

Having multiple asserts could easily get in the spaghetti problem where you do not have an idea what the test is about, but having single assert per test could make your testing equally unreadable having multiple tests in a big lasagna making the finding which test does what impossible.

There are some exceptions, but in this case keeping the pendulum in the middle is the answer.

gsf
  • 264
  • 3
  • 8
1

As other answerers have mentioned, there are situations where using multiple assertions adds transparency and value to your test cases and there are situations where a single assertion is the best choice to keep tests fast, independent/isolated, repeatable and self-validating. Test cases must also thoroughly validate the results of an action which is executed, and if it requires multiple assertions to validate the correctness of the code, then so be it.

One scenario where multiple assertions are quite valuable is for verifying API tests. API testing is a great example because we typically perform one action which we are testing, a single API call, but the response from that API call will more than likely contain a complex object with many strings, numbers, booleans, arrays, and other objects. If a single action modifies more than one property in the system, then each property must be validated in order to determine that the code is correct and that the system is in the expected state.

Some test frameworks, such as NUnit for C# and Jasmine for JavaScript, use soft assertions and will execute every assertion in the test case, and then at the end, report on every detected failed assertion. Jasmine is able to pull this off because their test runner and assertion library are tightly integrated.

But many other testing frameworks use hard assertions. They throw errors or exceptions at the first encountered failing assertion. This reduces transparency by hiding other possible defects in the system.

One argument against multiple assertions is that an early failure masks other failures in the system, requiring developers and testers to apply a fix and rerun the test to further detect errors hidden by the previously failed assertions. In some cases this may be good. Some may prefer this more iterative methodology. Others may prefer the advantages conferred by having full transparency. Perhaps seeing all of the reported failed state in the test case may help developers and testers analyse the defect and apply the best possible fix.

I recently discovered Jasmine executes all assertions, while Mocha and Chai together do not. To avoid the costs of switching to Jasmine, I built a Node.js module called multi-assert, which wraps groups of assertions together in order to execute each one and report all the failures in an easy-to-read format, such as in the example below:

1) Test - async/promises
       should expect status to be yellowblue, yellowred, bluegreen, and 3 to equal 4:
     AssertionError: 

      MultipleAssertionError: expected 3 to equal 4
        at /Users/user123/proj/multi-assert/examples/example-async.spec.js:17:32
        at /Users/user123/proj/multi-assert/multi-assert-async.js:12:27
        at Array.map (<anonymous>)

      MultipleAssertionError: expected 'bluegreen' to equal 'yellowblue'
        at /Users/user123/proj/multi-assert/examples/example-async.spec.js:16:75
        at async /Users/user123/proj/multi-assert/multi-assert-async.js:12:21
        at async Promise.all (index 0)

      MultipleAssertionError: expected 'bluegreen' to equal 'yellowred'
        at /Users/user123/proj/multi-assert/examples/example-async.spec.js:19:75
        at async /Users/user123/proj/multi-assert/multi-assert-async.js:12:21
        at async Promise.all (index 3)

      at /Users/user123/proj/multi-assert/multi-assert-async.js:22:23``

So, what problem does this -- the ability to enable soft assertions -- solve? It eliminates one of the reasons to not use multiple assertions. It nullifies the idea that multiple assertions are bad because they hide information.

But does this mean you should use multiple assertions everywhere? As others have mentioned, multiple assertions may be a sign that a test case is actually performing multiple, unrelated actions within a single test case. Before using multiple assertions, take a good look at your test case and ask yourself if the test case can be split up into separate test cases.

After splitting apart one of these test cases, if you find part of the derived test cases contain duplicated code, then examine what part specifically is duplicated. If it is the setup portion, consider refactoring the setup portion into a before hook, and group those test cases into a single suite so the test action and assertions are presented in reports as clearly distinct tests. If one fails, it is then much more clear to developers and testers responsible for fixing the defect what has gone wrong.

In conclusion, there is nothing wrong with multiple assertions or single assertions. Instead, what is important is that test cases are written in such a way that they're fast, independent and isolated from one another, that they can be easily repeatable, are self validating, and that they are complete and thorough. How to achieve this depends on critically analyzing the specific nature of what is being tested.

jmort253
  • 9,317
  • 2
  • 35
  • 63
0

I'm going to adjust the assertion here.

The problem isn't so much with multiple assertions but with multiple actions.

We all know arrange, act, assert (which is google-able if that's not true). That's one ACT only.

So given a situation, when I invoke ONE action, it's likely that I will have a single assertion when there is only a piece of single evidence that the single action has taken place.

On the other hand, there may be multiple pieces of evidence that the action has taken place. I may have one assert to ensure that the answer is correct, and another that the call was logged, and maybe another that some expected, intentional side-effect took place.

I may have multiple places to "connect my probes", so to speak, and maybe one of them is the directly returned value and another is a mock.

If we keep it to a single action/stimulus of an object in a known state, then there is nothing wrong with examining all of the evidence that the behavior has taken place as intended.

What you don't want is to have a string of actions and interwoven assertions (or worse, a whole raft of barely-related assertions at the very end).

tottinge
  • 403
  • 2
  • 5
  • So you'd be ok with a test named validString that acted once to build/get the string but asserted with multiple asserts: not null, not blank, not all caps, no underscores... – candied_orange Dec 16 '21 at 19:09
  • Yes, IMHO, provided that you were testing the composition of that string, and not trying to do an awful lot of unrelated magic in one string. – tottinge Jul 20 '23 at 16:12
0

What makes it easiest for you to write many unit tests? That’s what you should do. Better to write many not-quite-perfect unit tests, then to do them all perfectly and giving up because it’s too much work.

gnasher729
  • 42,090
  • 4
  • 59
  • 119
0

My (very late) 2cents:

If your asserts are testing the state/result of your code and does not generate additional side effects in between any of the assertions, it's ok to have multiple asserts.

You can consider it like this:

assertObjectEquals(result, expected);

is the same as:

assertEqual(result.prop1, expected,prop1);
assertEqual(result.prop2, expected,prop2);
assertEqual(result.prop3, expected,prop3);

This can be expanded to the state of your code, which you can easily create a custom assertion function/class for to make it seem like a single assertion, but you don't really need to if you can just use simple asserts to do the same

Example of what not to do:

assertEqual(result.prop1, expected.prop1);
result.doSomething(); // it doesn't necessarily modify anything, but it could, since we're not privy to internal implementation.
assertEqual(result.prop2, expected.prop2);
Populus
  • 187
  • 8