3

I have an API I wrote that I want to test at the API level.

Given that I'm testing from an external point of view, how can I manage data sets for each tests? The simplest solution I could come up with is to create a test suite where each test depend on the state the previous one set.

For example to test that a comment is successfully added to a post I need to make sure that a certain post exists first and that post may have been created by a previous test.

Is this kind of storytelling common in API testing? Or is there a better way?

Shoe Diamente
  • 187
  • 1
  • 6
  • 1
    Does this answer your question? [How to structure tests where one test is another test's setup?](https://softwareengineering.stackexchange.com/questions/221766/how-to-structure-tests-where-one-test-is-another-tests-setup) – gnat May 29 '20 at 08:03
  • @gnat That seems to be more like "**Given that I want tests to be dependent on each other** should I put them all into a single method or split them up in different methods?". I'm instead asking if this is the right call, given that dependent tests for example don't allow to run a single tests and easily break when new tests are added at the beginning or middle of a "test sequence". – Shoe Diamente May 29 '20 at 08:07

2 Answers2

2

Yes if you want to actually test a live setup to see if its working you can't avoid having to set state for your tests.

So in your example, you would create a new Post and then add comments and have that dependency and ordering in your test suite.

However! this is why you should also have Delete Posts functionality, even if it's not something you expect users to do. It allows your test to clean up after itself so you don't end up will millions of test posts.

Ewan
  • 70,664
  • 5
  • 76
  • 161
  • I see, but I don't mind having millions of test posts. They may be useful later when I'm doing performance testing :P – Shoe Diamente May 29 '20 at 08:23
  • test comment #23432 – Ewan May 29 '20 at 08:29
  • test comment #6090 – Ewan May 29 '20 at 08:29
  • test comment #26756465 – Ewan May 29 '20 at 08:30
  • ... are you sure? – Ewan May 29 '20 at 08:30
  • Well yeah. I would test in an isolated environment. Not in production, so... v0v – Shoe Diamente May 29 '20 at 08:30
  • Moreover some entities may not have a reasonable "delete" functionality. Some track records need to remain at any cost. As an example a functional requirement may say that an e-invoice sent to the government can't ever be deleted and must be recorded forever. So deleting a sent invoice is something there should not even be code for. And this may be an important legal requirement. – Shoe Diamente May 29 '20 at 08:34
  • sure but your audit records aren't affecting other users of your application. You want to be able to run these kind of tests against your live environment. to smoke test deployments and just generally check everything is working – Ewan May 29 '20 at 08:36
  • Be careful about your test data not producing issues on users screen or admin screens (like wrong statistics), that it cannot ever overwrite existing data, and that it cannot send emails to reals emails (you can use emails in example.com and example.org because it's a reserved domain name by IANA so no one can use them). – Nyamiou The Galeanthrope May 29 '20 at 13:10
1

Shouldn't a test suite be set up with some test data anyway? Either as part of the build procedure for the application (e.g., test data scripts), or as part of the test's setup/teardown?

I tend to feel that all tests should be "standalone" and not depend on other tests. If you want to test that an Entry goes in, do that in one test, and if you want to test that a Comment goes to an Entry, pre-set an Entry prior to the actual test, so you know your test has an Entry on which to comment, but don't rely on a prior test to run.

The reason is because when you are running tests, you don't want one test failure to block the ability to test a bunch of subsequent stuff. You get in a CI/CD scenario, and you want that the Comment test should give a yes/no regardless of the Entry test, otherwise you stop and fix the Entry test and re-run the tests, only to find out that the Comment test is failing too, so on and so forth. Instead of having a comprehensive "true" list of all test results, you have to re-run your tests multiple times to see get the same "true" results.

As for millions of test posts, this should not be an issue because the test environment should be throwaway and rebuild from scratch each time you run the test suite.

jleach
  • 2,632
  • 9
  • 27
  • "The reason is because when you are running tests, you don't want one test failure to block the ability to test a bunch of subsequent stuff." -- Well, if any early on test fails than it is likely that the state of the application after those test is unrecoverable. So I'm actually fine with tests having an order in that regard. – Shoe Diamente May 29 '20 at 14:48
  • "Shouldn't a test suite be set up with some test data anyway?" -- But how? – Shoe Diamente May 29 '20 at 14:48
  • @ShoeDiamente - write some test data scripts or use a data generator of some sort to create the scripts. Reusable test data is pretty essential to integration testing though... might be worth studying a bit (if you built the api, you should have control of it pre-production, which is where this is typically handled) – jleach May 29 '20 at 15:08