3

I want to know how rigid do we have to stick to the notion of end to end testing, please consider the following explanation:

Basically, when we talk about end to end test we send a request and expect a response, in an end to end test we verify this request and response with a reference payload that we have.

But, consider this scenerio: An api affects two tables, I'd like to verify that if a request fails, the number of rows that are in the second table remain constant. The request fails, and nothing but the status code and error message are returned. Currently there are no apis that return the row count of the second table. Would it be okay to use the service layer to verify the second condition for the second table in the end to end test? Would this violate the notion of an end to end test?

Philip Kendall
  • 22,899
  • 9
  • 58
  • 61
mleko
  • 47
  • 5
  • `Currently, there are no APIs that return the row count of the second table` but can the side effects be validated by other APIs? While there's nothing preventing you from mixing assertions of all sorts, you will find that to be a bit counterproductive because both kinds of code changes at a different pace and due to different reasons. – Laiv Sep 05 '22 at 14:56
  • @Laiv , are tests that compare data integrity as I have mentioned above common in practise? Are implementation results verified as such? Or are they usually done at all? – mleko Sep 05 '22 at 15:13
  • It's hard for anyone to say if something is common practice because things vary (a lot) from team to team or project to project. If you ask me I would say that I don't find it convenient. It's weird and couples code of different levels of abstractions/layers. If you have no way to E2E validate what happened then it's not supposed to be testable. To me, E2E tests are not meant to know that data is being persisted anywhere, that allows me to implement E2E in different technologies and automate'em in different ways, reuse them for loading tests, etc. But I'm just a stranger on the internet. – Laiv Sep 05 '22 at 15:39
  • @Laiv , but, not just e2e tests, I meant tests in general. Is data integrity verified at all with respect to some service layer operations? – mleko Sep 05 '22 at 16:26

2 Answers2

5

I think what you're realising here is that your definition of an end-to-end test

we send a request and expect a response

is incomplete. Not everything has requests and responses, yet it's still possible to do end to end tests on systems which don't have requests and responses. Without wanting to in any way elevate them to the status of "authority", this definition from Browserstack seems better to me:

End to end testing (E2E testing) refers to a software testing method that involves testing an application’s workflow from beginning to end. This method basically aims to replicate real user scenarios so that the system can be validated for integration and data integrity.

If nothing else, that lets us avoid any domain-specific terms like "request" and "response". It also illustrates that your idea of looking at the row count is probably wrong: a user doesn't care about that, because it's an internal implementation detail. Presumably at some point, something else reads that table and a user (maybe not the same user who made the request) does care - it's that which you should be testing, not the row count directly.

Philip Kendall
  • 22,899
  • 9
  • 58
  • 61
  • But, if I'd have to test if the api call did write in a single transaction, would that come into a separate type of test? – mleko Sep 04 '22 at 04:07
  • To quote from @candied_orange 's answer: "Who cares? These are your needs. The name you give the test is far less important than if it is useful to you." – Philip Kendall Sep 04 '22 at 07:37
  • no, i mean, I'll have to organize the project accordingly, so was wondering if it would be called a bit differently – mleko Sep 04 '22 at 09:59
  • 1
    "I'll have to organize the project accordingly" No, you don't "have" to. You (or your team) may have chosen to organise your project by arbitrary "unit test", "integration test", "end to end test", " test" labels, but that's a choice you made. You can always unmake that choice. – Philip Kendall Sep 04 '22 at 10:01
  • Are tests that compare data integrity as I have mentioned above common in practise? Are implementation results verified as such? – mleko Sep 05 '22 at 09:19
  • You shouldn't really care if something is "common" or not, you should care if it adds value to your system. – Philip Kendall Sep 05 '22 at 09:29
  • yes, that's true, but just for a frame of reference, since I can't really find examples of my particular use case. – mleko Sep 05 '22 at 09:45
  • or maybe rather than common, are they done at all? – mleko Sep 05 '22 at 10:36
5

Tests are code. They have requirements. You've clearly gathered the requirements for this test and they involve knowing the state of the Data Base. Fine. Create that test.

I want to know how rigid do we have to stick to the notion of end to end testing,

You don't. Tests come in many shapes and sizes.

Would this violate the notion of an end to end test?

Who cares? These are your needs. The name you give the test is far less important than if it is useful to you.

Now, there are consequences to sneaking behind an API. By using the service layer you're coupling the test to structure that the API never promised anyone would exist. This test locks down how the service layer works. It demands that it exist. This test will need to be rewritten if the service layer is rewritten. None of that would be true if the test had stuck to the API.

But, like you said, you can't create this test without doing that. Fine. Just be sure you're OK with the trouble this causes. Make sure other people understand that trouble. That way they know what it means when this test fails. Don't hide behind some particular definition of "End to end" testing (there are many). The people who wrote those definitions have never looked at your project.

candied_orange
  • 102,279
  • 24
  • 197
  • 315
  • so, basically it would make more sense to just check the status code ? I do have the requirement that the api call perform a single transactional write to the tables but, would that come into a separate type of test? – mleko Sep 04 '22 at 04:04
  • 1
    @mleko I'm trying to discourage thinking about "types of tests" and getting you to focus on your needs. You have an API to test. That API changes the state of the system. If the API provides no way to tell if the state of the system changed then as far as an API user cares there is no state. Therefor the test doesn't need to exist. But if you have things going on beyond the API you can justify reaching behind the API to the service layer. But doing that comes with a cost. So think hard about why any API user would care about the changed state and make sure you can't write an API only test. – candied_orange Sep 04 '22 at 14:06
  • Are tests that compare data integrity as I have mentioned above common in practise? Are implementation results verified as such? – mleko Sep 05 '22 at 09:19
  • or maybe rather than common, are they done at all? – mleko Sep 05 '22 at 10:35
  • @mleko they are certainly done. But understand that testing software is a young discipline. We’re making a lot of mistakes. My advice is to care much less about what’s been done and focus on the problems this is known to cause. Decide if doing the test this way is worth locking down what would otherwise be refactorable structure under behavioral tests. I can’t tell if you have reason to care about that. – candied_orange Sep 05 '22 at 17:19