10

Just for clarity, the stress test I have written steadily increases the load on the system until it reaches a breaking point. It theoretically runs indefinitely, but as the system resources are finite, it is expected to fail after some point of time. I have an expected load for the system, but this is tested separately in a load test. The purpose of this stress test is to find out how much load I can put on the system before I need to implement scaling.

I'm in the process of writing a stress test for a system, and I'm wondering whether it makes sense to have pass/fail criteria. By nature of the test, the load steadily increases until it hits a breaking point (i.e. it fails). I obviously don't know what this breaking point is beforehand, and therefore no expectation of the load the system can handle (in theory anyway).

Now I do have other performance tests to test the system under an expected load etc., which I can easily set pass/fail criteria for, and I could use these criteria as a basis for my stress test. In other words, I could set a minimum baseline for my stress test to reach, but I'm not sure if this is the right thing to do (is this 'duplicating' my other test?).

I'm hoping someone with more experience in performance testing can help me out here. What pass/fail criteria have others used when stress testing (if any)?

Alex
  • 211
  • 1
  • 5
  • 1
    If you don't have a pass/fail, why are you doing the test? – RemcoGerlich Nov 22 '16 at 14:46
  • @RemcoGerlich So I can know the limits of the system? This will help in capacity planning etc. – Alex Nov 22 '16 at 14:50
  • I think that capacity planning is where you decide the minimum load your system needs to be able to handle (so then you have a pass fail criterion). – RemcoGerlich Nov 22 '16 at 14:56
  • @RemcoGerlich Maybe I've got my terms mixed up, but basically I do have an expected load (which is tested separately), but I'm using this stress test to determine at what point (i.e. number of users) I will need to scale the infrastructure. It's a separate test because changes to the system may change the load the system can handle, which wouldn't be visible in a load test. – Alex Nov 22 '16 at 15:17
  • @Alex, nope you haven't got your terms muddled. You are precisely describing a stress test. The issue you have is that there's no pass/fail associated with stress testing, so it can't easily be run using "unit testing" tools. – David Arno Nov 22 '16 at 18:12

3 Answers3

10

In a stress test your job is not to define the stress the subject should be able to take. It's to measure the stress it takes before it fails.

You can use the performance criteria to define what a stress failure is. But the result of a stress test isn't pass/fail. It's "failed after 90 hours under 100% utilization with ventilation 50% compromised".

candied_orange
  • 102,279
  • 24
  • 197
  • 315
  • one question. Should the stress test cause a system crash? In other words. Is the crash what we consider the "failure"? – Laiv Nov 22 '16 at 22:02
  • 3
    @laiv The stress test should cause stress. And demonstrate how the subject responds to that stress. If it causes a system crash that should be documented. Stress tests are supposed to cause failures and show what it takes to cause them. A system crash is a failure, presuming the crashed system fails it's performance requirements. They usually do. – candied_orange Nov 22 '16 at 22:13
1

It depends on the requirements, if your requirements specifies that the expected result for the app performance it's X and actually you got Y, so it's a Fail.
If you don't have requirements defined, so you might stress your system and collect boundary data and then figure out and doc these limits.

Thiago F. Peçanha
  • 11
  • 2
0

You can easily update your primary stress test to also support a QA pass/fail verification, something like "able to reach/sustain X load without breaking". Ideally with X being configurable (for different release branches, for example).

The result would be fail if the system breaks before the load reaches X and pass if it doesn't break. You'd just have to stop increasing the load once it reaches the X value in a "sustain" scenario.

IMHO an automated test like this can be very useful in the context of CI/CD, especially on production branches.

Dan Cornilescu
  • 869
  • 7
  • 14