23

Context: we operate in a highly regulated industry (medical), and aim to have automated test cases to cover all of our requirements - allowing us to still release quickly, but safely.

We have a requirement or acceptance criteria that reads something like:

x object should be read-only for users

Editing this object not a piece of functionality that is available in our web application (or via an API) - this state can only be created by the backend (kotlin) application itself, but it is important to do what we can to verify this, (and ideally in an automated way).

The problem: how do you test for the absence of some functionality?

Our current thinking is similar to this answer, specifically:

tests are just examples and not a proof

Therefore all our tests are examples of a sort, and it's acceptable to have a slightly wooly test, for example the absence of an edit button. It's likely that if we weren't in a regulated industry we'd not put as much thought into it, and accept that you have to trust the design to some extent for this type of requirement.

Some good thoughts from below (thanks for all the decent discussions):

  • Code reviews/verification: yep, we do this
  • API testing: testing the resource returned is read-only is something we do
  • Security testing: absolutely will do this
Charlie Calver
  • 259
  • 2
  • 5
  • 67
    This is not an absence of functionality. This is a functionality stating that a write attempt from a user should be rejected. – mouviciel Jun 22 '22 at 13:10
  • You can use machine learning, e.g. automata learning, to discover paths through your application that would allow changes. – Polygnome Jun 22 '22 at 18:02
  • Can you dump every attribute of x object, e.g. to JSON? – Eric Duminil Jun 22 '22 at 20:40
  • 2
    @mouviciel It's the absence of any functionality that allows a write. You may test that command X doesn't write and command Y doesn't write and command Z doesn't write but if command Q does allow a write (and you didn't think to test it) that's a fail. – user253751 Jun 23 '22 at 09:06
  • @user253751: If you define a functionality as the complete proof that something can never be done in any way, you're going to run into a variation of the halting problem. Your definition simply does not work. Mouviciel is correct that rejecting an update _is_ a functional behavior, and it is that behavior that you test. You don't write a single test that tries to verify the entire codebase and every possible input permutation, which is what you're effectively suggesting. – Flater Jun 23 '22 at 13:51
  • @Flater Not necessarily the halting problem. The halting problem is not a problem if you allow "too confusing" cases to be rejected. In this case, if it's undecidable whether the user can edit something they shouldn't be allowed to edit, that is a failure. – user253751 Jun 23 '22 at 14:17
  • @user253751: If you omit checking cases that are more than an _arbitrary_ amount of confusing/complex; then you no longer have a full proof, thereby negating the entire goal of the test. You can't both check less than everything and then be confident that you checked everything. – Flater Jun 23 '22 at 14:20
  • @Flater If you cannot prove your system meets the requirements then you failed. It doesn't matter if it actually does meet the requirements but in an undecidable way. You have to prove that it does. – user253751 Jun 23 '22 at 14:20
  • 3
    @user253751: You cannot prove a negative, and your definition is inverting a sensible requirement into a negative. You are confusing software development acceptance criteria testing with pen testing. In the latter, you do indeed look for ways in that were not designed. But that is not the case for software development itself; where you simply test that the explicitly developed behavior works as expected (i.e. performing an edit without having the proper authorization should refuse to execute the edit). – Flater Jun 23 '22 at 14:24
  • 4
    @Flater Of course you can prove a negative in formal logic. It's just really exhausting (no pun intended). You can enumerate all the things the system does do, and check that the thing that shouldn't happen isn't one of them. Since 100% of the system is modeled, if it does the thing, you won't succeed in proving it doesn't. – user253751 Jun 23 '22 at 15:17
  • 1
    @user253751 And we've come full circle to the halting problem again. – Flater Jun 23 '22 at 18:00
  • 4
    @Flater again, the "one-sided" halting problem is fine. Either the thing is true and you can prove it, it's false and you can prove it, or it may be true or false and you can't prove it. Only the first outcome is acceptable when designing most systems. If you can't prove it either way, you messed up, even if it happens to be true and unprovable. If your system is so complicated that statements like "users can't edit read-only data" are undecidable, then throw it away and design it again. – user253751 Jun 23 '22 at 18:03
  • 1
    @Flater: "You cannot prove a negative" is a negative. How did you prove it? Oh, that's right, you didn't, because it's a wrong statement, no matter how often people try to apply it. – Eric Duminil Jun 24 '22 at 08:54
  • You cannot prove anything with testing. Testing is aimed at finding bugs, and does not pretend to find all of them. If you need a proof, then you must use formal languages, like [B-Method](https://en.wikipedia.org/wiki/B-Method) for instance. But then you have only moved the verification problem to the specification, which may not reflect the actual user's needs. – mouviciel Jun 27 '22 at 15:53

8 Answers8

51

For a web application, this is actually easy.

Presumably there is functionality to edit these objects (e.g. for maintainers or admins), and this functionality must somehow map to a system state change when certain requests are received from the web app. All you have to do is verify that when the server side receives such a request in the context of a user session, the state change doesn't occur.

(If you also want to verify that the object doesn't look editable in the front-end, that is a pure front-end test like other GUI tests. But the more important thing, as always in web applications, is to ensure that users cannot effect a particular change, no matter whether they issue the request through the official web client or by circumventing it.)

Kilian Foth
  • 107,706
  • 45
  • 295
  • 310
  • 29
    Concretely, I'd write a test on a REST-API level like **Given** an object id=123 with state A, **When** I `POST /object/123 {"state": "B"}`, **Then** I get an error message **And** object 123 still has state A. – amon Jun 22 '22 at 13:00
  • 3
    yeah the difficulty is in getting all possible user permissions that _dont_ give the privilidge – Ewan Jun 22 '22 at 20:25
  • 2
    It seems to me that you are simply "presuming" away the premise of the question, here. You say that *"Presumably there is functionality to edit these objects (e.g. for maintainers or admins)"*, but there's no hint of that in the question, and the fact that the asker is even contemplating things like testing for the absence of an edit button is pretty good evidence that such functionality doesn't exist in the webapp at all, for users of any permission level. A team of devs who have put "much thought" into this would not have missed this (obvious) solution if it were really available. – Mark Amery Jun 23 '22 at 10:11
  • 5
    Soooo how do you ensure that there's not some other API that you might have forgotten about when writing your test? This is the same as proving there are no bugs in your application, you can make it less likely, but there's not going to be any proof. – Voo Jun 23 '22 at 12:56
  • 2
    Imagine you have an append only database. How do you test that it never changes any values? Your presumption would not be the case. – Tigerware Jun 23 '22 at 13:39
  • 2
    @Voo: You can't prove a negative, and you cannot reasonably write tests that encompass every and all entrypoints in the application. If you don't even have control over which entrypoints your application has and what they functionally do, you have significantly bigger fish to fry. A test should focus on a specifically implemented piece of logic; it should not try to verify open-ended abstract questions such as "might there be some code anywhere in this codebase which [...]?". You're going to run into a variation of the halting problem very quickly. – Flater Jun 23 '22 at 13:55
  • 3
    @Flater Oh I absolutely agree, which is why I think this should also at the very least be mentioned somewhere in an answer to such a problem. Sometimes the only right answer is "you can't, but you can do A,B and C to minimize the risk". – Voo Jun 23 '22 at 14:00
  • 1
    This is probably the best answer because it also gets at application security testing. If you only test via the UI, then you are ensuring that it is read-only for an ordinary user, but a malicious user, directly making REST calls against the server, may find a crack that isn't exposed at the UI level but is nontheless a security vulnerability. By testing at the API to the backend, you kill two birds with one stone. +1 – bob Jun 23 '22 at 15:21
  • @markAmery 's comment is correct - there isn't any functionality to edit this. To bob's comment (directly above), it isn't even available via an API. As he points out, it's more of a security requirement, and we'll definitely do internal and external testing of this. The problem really centers around directly proving this is 'read-only' to auditors that expect a more direct form of verificaftion – Charlie Calver Jun 28 '22 at 08:07
  • Auditors should not trust in your "automatic, direct and probably convenient" way to prove this. At least, if they are serious about this. It takes a source code review and, in some scenarios, it might involve checking O.S o DB logs (depending on the storage in use), because who knows what other systems could be writing these objects. In other words, you need to track the data's life cycle. – Laiv Jun 28 '22 at 09:28
17

A common approach for requirements of the form "this must never happen" is fuzz testing.

The test case can be formulated like this:

  1. Set up initial state.
  2. Have a subroutine that verifies that the state of objects is as it should be.
  3. Run a web application fuzzer for given time. Set it up so that it is logged in with user permissions and can click everything and send any forms.
  4. Run the subroutine from step 2 again to verify that things that should be protected didn't change.

The browser-level fuzz testers for web applications are often called monkey testers, due to the way how they randomly click all buttons available on the user interface.

If your application has an API between frontend and backend, it is useful to also run an API-level fuzzer on it. This verifies that any access checks are properly implemented on the backend, to avoid bypassing by modifying the frontend javascript using browser tools.

jpa
  • 1,368
  • 7
  • 11
8

Sometimes you can't actually test things. But you can verify them by inspection of the source code, build files, etc.

If you can show that the software provides no method for the user to edit something, then it must be read only.

Simon B
  • 9,167
  • 4
  • 26
  • 33
  • While I agree with the concept, doesn't this negate the desire to automated tests? You're effectively asking for a manual code inspection *any time* a change is made because you can't know that the change doesn't break the non-writability aspect – Peter M Jun 22 '22 at 21:00
  • 1
    @PeterM You don't bother doing the *formal, documented* code inspections until you have a software build that's ready to ship. Safey critical processes are only really interested in the one build that's deliverable. – Simon B Jun 22 '22 at 21:11
  • 1
    @SimonB of course the problem there is that your software build isn't ready to ship until after it has been fully inspected and no critical flaws are found. If your formal inspection is thorough but your automated pre-inspection testing is not, the inspection will likely find critical flaws that need to be fixed, and then the inspection will have to be performed again on the updated codebase (and repeat as necessary, until no more critical flaws are discovered). That can become quite a tedious undertaking, hence the desire for robust automated tests. – Jeremy Friesner Jun 23 '22 at 03:59
  • @PeterM: No, because it doesn't contradict the automated tests themselves. Those tests exist to confirm an explicitly implemented behavior (e.g. the rejection of an edit without the proper authorization). These tests should be not be doing things like trawling through the entire codebase to find any possible entrypoint in which it may be possible using particular input to achieve a particular outcome. That is _well_ outside of the scope of automated testing in a pipeline, and is sounding much more like pen testing. – Flater Jun 23 '22 at 13:58
  • @simonB "You don't bother doing the formal, documented code inspections until you have a software build that's ready to ship" - in our world, this is multiple times per day - thus the need for automation – Charlie Calver Jun 28 '22 at 08:13
5

Negative requirements are always tricky. I know many places where they are forbidden because of the difficulty in qualification.

Interestingly, the example you gave has already taken the step to turn it into a positive requirement. "x object should be read-only for users" is testable. You get object x, and you check that it is read-only.

You are reading into this, and turning it into "x object should not be writable for users (in any way)" which is truly a negative requirement. But the requirement as written has already been converted into a positive requirement by making a property "read-only" and using positive phrasings.

A common way to get close to the testing you seek is to break the software into layers. The general purpose layer is obliged to open files through an API provided by a core layer. In this API, you have a "is read-only" flag that can be tested for in a positive way. Then, in the core, you define what "read-only" actually means.

Testing the core may still require testing a negative, but now changes in the general layer can be tested with a simple positive test. Most changes are in this general layer, so most changes support fast releases.

And, once you get to the core, you might be able to test that negative by testing the complement. While there may be a myriad of ways to interact with a file in the general layer, there is likely to be only a handful of ways to interact with a file at the core. It then becomes reasonable to test the core behavior positively, and then argue that that correctly proves the negative phrased question.

All of this reminds me when I stood up on my soap box one day and asked for an explanation why "there will be no data races in this multi-threaded C++ application" wasn't a requirement...

Cort Ammon
  • 10,840
  • 3
  • 23
  • 32
  • 1
    Not sure if introducing an extra layer is a good idea here. It just increases the surface for bugs and requires more testing. Also, it sounds like it would make available an API for attempting to edit those read-only objects - which we could just as easily introduce without an extra layer. – Bergi Jun 23 '22 at 01:48
  • @Bergi I suppose it depends on how many places you might try to edit a read only file. If there's only two interfaces that a user has to touch a file, it may be easier to just handle it in a single layer. However, if there's a hundred rules which can make a file read only, and a hundred ways to edit the file, that's 100*100 patterns to worry about. Adding a layer reduces it to 100+100. – Cort Ammon Jun 24 '22 at 14:30
3

One option may be to test for the inverse, with functionality being considered a failure state. Let's break the requirement up, and look at its success and failure states:

Requirement: x is read-only for users.

Success: x is read-only.
Failure: x is not read-only.

Of these, it tends to be much easier to determine whether an object is writeable, so you may want to craft a test for failure. Determine how the user is able to interact with x, and test all possible interactions. If even one interaction successfully modifies x, then you know that x is not read-only, and thus that the overarching requirement has not been met.

Justin Time - Reinstate Monica
  • 149
  • 6
  • 1
    +1 This is what I would do. For any allowed interaction, write a test that performs that action and compares the current state of the object to the original state. If it's not identical, that particular test fails. – Not Entirely Serious Jun 22 '22 at 23:43
0

I am assuming the functionality does not exist and you therefore can't test it. That means for example data stored as a plain old object with no public member or setter. However you need to prove it does not exists.

From my experience with medical software you need to produce a document, probably a design review where you will describe several things :

  • How is the data stored (database, file, in memory object, etc.)
  • The means to edit this data is they were to exist (permissions, public setters, etc.)
  • A description of the absence of such mean, by code review for example.

Ideally this document should be revised every time there are modifications on the scope of your data to ensure there is still no mean to edit it (at least once for each release where such modifications occur).

JayZ
  • 827
  • 5
  • 11
-2

It’s tricky if your code is designed so that the compiler doesn’t even compile a statement that would modify x. Say const int x = 3; x = 5; You can’t compile and run a unit test that tests this.

I have at times collected such statements, commented them out with instructions that each of these statements should fail to compile, which you would have to check manually. So if someone changed x to be non-const, the assignment would compile which would be wrong.

For a developer it would be much preferable to have a const property than a read/write property where writing throws an unconditional exception.

gnasher729
  • 42,090
  • 4
  • 59
  • 119
-3

A lot of languages support reflection.

You can often use this to check for readonly or final modifiers.

ScottishTapWater
  • 187
  • 7
  • A lot of languages won't even compile if you try and write to a read-only property, C# for instance. So if you want to test it, then you need to use reflection. Whether you should bother or not depends on how important it is to you that no future dev says "oh, I wonder why that's readonly, oh well, it's not anymore! " – ScottishTapWater Jun 24 '22 at 12:23
  • If a java variable is marked as `final`, you don't need to test anything, with or without reflection, in order to check that the variable really is read-only (or write-once, let's say). The way I understood the requirements is that the object should be read-only to normal users, but might be edited by admin. I didn't downvote, BTW, and simply mentioned that I didn't understand your point. – Eric Duminil Jun 24 '22 at 12:31
  • I'd agree with you there that that's true. However, if for whatever reason someone has stipulated that there must be a test... You can do it with reflection – ScottishTapWater Jun 24 '22 at 12:35
  • 1
    @EricDuminil - That a bit better? – ScottishTapWater Jun 25 '22 at 22:21