Is it anti-pattern to mix unit test and web test?

Question

It is a language agnostic question, e.g. I have a unit test like

# Unit test
User user = User.create('john');
assertEquals("User name is john", "john", user.getName());

# Web test start from here
goTo("/user/john");
assertTitleEquals("john");

Do you think it is an anti-pattern? Although I can write a full selenium test suites to do the same thing, but using the above code save much of my time (e.g. no more click('Login'); waitUntil etc, and I don't need to worry if my UI is changed at all, sound like it is a quick and more stable approach to test?

Answer 1

Yes, this is an anti-pattern. "Unit tests should only have one reason to fail." is a common and well known best practice. This test can fail for two reasons.

And I would caution you against saving a little bit of time now (by condensing the tests) to pay for it later by spending a lot of time tracking down the coupling in your test or the ambiguity in your test failures.

Answer 2

That's not a unit test; it's an acceptance test. I think a few people are getting hung up on that fact that you called it a unit test without looking at what you're actually trying to achieve.

It looks as if what you're doing in the first set of steps is using perfectly decent code which happens to come from a unit testing framework to set up a context. That is, something like:

Given John is registered as a user

If you were doing BDD, or using a BDD framework, that would be your step. It wouldn't matter how the context was set up, because what we're really interested in is the outcome:

Then John's title should be "John".

This should be true regardless of how the context was created. Creating the context by hitting the database or using a service or any other mechanism is OK. If you ever feel the need to check that the user-creation pages work OK, make them into a separate scenario (or test).

Your test is therefore just fine, because what you're doing is a system-level test (or acceptance test if you like), not a unit test (even though you're using unit-testing code inside it).

The assertion that John's registration exists might be redundant, but is useful if you want to check that your context has been set up successfully (for instance, the database hasn't got out of sync with that User.create call) without having to wait for the next steps in the test to fail.

You might also want to put some abstractions around the steps, just to keep them maintainable. You could use the English Language BDD frameworks, or more maintainably, create a nice little DSL like this. It's also worth having a look at the Page Object pattern. This will keep your context-setup clean, and make it much easier to change it later on if you decide to do something different.

The equivalent to unit-testing's "only one reason to fail" in system-level tests is "only one aspect of behaviour", which is the same thing, only at a higher level of abstraction.

Don't forget that class-level TDD is also very valuable!

Answer 3

How much scaffolding your Selenium UI tests require or how long it takes to run them are genuine practical factors. You might rationally segment such tests into an "beyond unit test" suite. Just don't eliminate that testing step "because it isn't a unit test." If you're working toward software quality, you can't be "saved by the bell" of what makes a pure unit test or not.

A traditional response would be: Well, you need to add in other testing layers (module, integration, specification, acceptance, ...) and test those things as well--but that's outside the scope of pure unit testing. But that distinction is often used to punt on doing those other forms of testing, or leave it to someone else. "Not my job, man. I'm just responsible for unit tests. Someone else is responsible for integration and UI testing." That punting is an abdication of responsibility, and inimical to TDD's philosophy of integrating coding and testing. It's also a big reason unit tests are infamous for passing with flying colors, even when the code as a whole is still breaking or not doing the job it was designed to do.

There is a place for segmenting testing realms. Performance testing, walk-through-the-UI testing, intrusion testing, and exhaustive compliance testing against all possible supported libraries and platforms can each require spinning up considerable middleware and virtual instances, and might take dekaminutes or hours to complete. Run those test suites occasionally, rather than every code edit. But for goodness sake do not use their "higher level" nature to avoid doing them.

There is also often a notion that integration, UI, and other forms of testing require different tools. While you may need some additional tools (Selenium, in this case), but I cannot see the point in requiring an entirely different testing foundation of test runners, fixtures, mocking and support libraries, and so on. I'd argue that having different infrastructure for 4 or 5 desired layers of testing is anti-simplicity and ultimately itself an anti-pattern.

My experience is that rigid conformance to unit testing dogma--"only one possible reason to fail" and "don't test things together" e.g.--is restrictive and ultimately counter-productive. You certainly want tests to exercise your code in the clear, simple, direct ways. Pure unit tests can and do help with that. But why stop there?

I build "unit" tests with an expanded model of what constitutes a unit--not just individual software atoms or particles, but multiple routines and objects, acting together. Atoms composed into molecules, and then molecules composed into larger molecules.

Relaxing the notion of what constitutes a "unit" and piggybacking unit testing infrastructure for integration and UI tests may not win me awards for testing purity, but it helps larger requirements. It gets more tests written and executed--both in an absolute sense, and per hour of testing effort. And it gets a wider swath of my code tested, in ways that mimic real use.

Isn't that the ultimate goal? Having more robustly-tested, works-together, quality software products? If so, go ahead and use a pure, no-dependencies-whatever model of unit testing to start. Just don't stop there. That slightly higher-level or as-seen-by-a-user tests might fail in several ways, or that they have dependencies--that may be a reason to put those tests into different test specs/files, or to run them less frequently. But it's not a reason to skip them--and often, not even a reason to decouple them from the basic coding process.