should a single end-to-end test aim to validate a scenarios or a specific feature of an application

Question

I am currently working on a web product where we have a testing strategy that includes end-to-end testing.

Our tech lead is of the opinion that each e2e test should only validate a single feature. For example, we would have a test for creating an account, a separate test for logging in and an other separate test for updating personal info of the account. if I had to picture the idea, I guess it would ideally look like this :

test 1 : create account
test 2 : log into existing account
test 3 : update info of currently logged in account

To me this seems ineficient and hard to maintain because in reality each test is reliant on the previous one and they actually end up running more like this :

test 1 : create account
test 2 : create account -> log into created account
test 3 : create account -> log into created account -> update personal info of the account

When put like this it just seems like only test 3 should be ran because it juste goes through everything. And should on test, say "login", fail then test 2 would fail, as it should, but so will test 3. The "update info" feature is not being tested in either case, test 2 and test 3 both provide the exact same level of information.

I can see how ideally targetting a specific feature is better, since it is supposed to indicate precisely what is broken. But to me it is not achievable in an e2e testing context, so a scenario approach sounds more appropriate.

In your experience are e2e tests written as scenarios (like test 3) or are they targeting a specific feature to test ? If it is the later, did you manage to circumvent the issue I highlighted and if so how ?

Answer 1

The fundamental issue is thinking tests should depend on one another. You haven't specified a programming language or tool, so I'll speak in general terms about each test case.

Each test should be atomic and deterministic (or as deterministic as you can make an end-to-end test). A well-written test can be executed by itself. That means each test will have some steps repeated:

Create account

1. Open browser
2. Navigate to create account page
3. Fill in necessary info
4. Click submit button
5. Assert account was created (text appears on screen/record exists in database)

Log in

1. Stub out user profile or use predetermined account
2. Open browser
3. Navigate to login screen
4. Enter username and password
5. Click "Log In" button.
6. Assert some text appears on screen that you should only see after logging in

Update User Profile

1. Stub out user profile or use predetermined account
2. Open browser
3. Navigate to login screen
4. Enter username and password
5. Click "Log In" button.
6. Navigate to update profile screen
7. Fill in fields with test data
8. Submit user profile form
9. Assert profile was updated
   • Certain text appears on screen?
   • Data was updated in database?

You can see that steps 1 through 4 are repeated in the last two test cases. Furthermore, the "create account" test can be summarized in a step or two for the sake of brevity. This level of repetition is acceptable for a test. It is more important for a test to be atomic and repeatable than it is to be DRY (e.g. "Don't Repeat Yourself").

Answer 2

Log-in is a special beast, because it is an almost unavoidable step for testing any feature that requires an authenticated user.

Other than that, each test case should be targeted to a single feature as much as possible. And to the extent that your testing framework allows you to specify it, preconditions for a testcase, like the login step to establish an authenticated user, should be performed in a set-up part of the test. Some testing frameworks can then even indicate in the results if the test case itself failed or if there was an error in establishing the preconditions.

To link it to your example, there should not be a "create user account" step in sequences 2 and 3. Those tests can just as well be performed with a preexisting user account.