0

I've got a .NET Core application that takes HTTP requests. I'm a little bored, so I decided to stress-test it using West Wind Web Surge, and putting an example request as the body.

My question is, since the request is exactly the same every time, am I seeing artificially high performance because the application has already processed a similar request? Is it likely something is saved in-memory rather than recalculated every time, and I'm not aware of it? Is it worth the effort to build an application that will generate requests randomly to get a truer picture? My code is pretty standard, nothing fancy going on.

I know it's a generic question, but I don't really know where to look to find the answer.

foxtrotuniform6969
  • 799
  • 1
  • 7
  • 9
  • This really depends on the application in question! The application may have caching on many different levels. Say there is a database lookup involved - the lookup will most likely be much faster the second time due to caching inside the database engine. – JacquesB May 28 '18 at 17:09
  • @JacquesB There's not much caching going on. The database and/or message broker connections are created at startup, used throughout the application, then torn down at shutdown, and not instantiated every time they're used within the application. What about things such as reading the request? For example, I read a `byte[]` into a `MemoryStream` and then use a `BinaryReader` to read each `byte` and then load it into a model. Is there any inherent caching going on there, that I did not implement? If there is, I cannot find any. I was wondering if .NET does anything for me in the background – foxtrotuniform6969 May 28 '18 at 17:27
  • 1
    The only reliable way of find this out is by varying the requests a little bit, maybe utilizing a random generator, and comparing the results. – Doc Brown May 28 '18 at 18:23
  • @DocBrown I'm afraid I agree with you. – foxtrotuniform6969 May 28 '18 at 18:37

2 Answers2

3

Yes. (sort of)

You can have all kinds of caching between your client and the eventual response. If you are running your tests against an environment and network that you don't fully control, such as a live deployment where various networking and deployment teams may be involved, then its likely that there will be some caching policies in place.

Now, do you want to test the performance with or without caching? if you test it without then are your results useful? After all the live system will be using a cache.

If you test it with, then again, does the amount of caching match what you would see if live? You presumably don't expect 100% identical requests in live.

What you need to do is mimic the live system as best you can. This means generating pseudo or replaying live traffic against a system which is deployed to the same sort of hardware as your live system.

Ewan
  • 70,664
  • 5
  • 76
  • 161
1

Well-behaved load test is not hammering single URL, this form of test doesn't make a lot of sense (unless you're load testing .NET Core or IIS themselves), it represents real life user sitting behind real browser with all related stuff like:

  • Headers
  • Cookies
  • Behavior with regards to embedded resources (images, scripts, styles, fonts, etc.) - real browsers download them using 6 parallel threads and respecting Cache-Control headers
  • Cache itself, real browsers do download embedded resources, but do this only once, on subsequent requests the resources are being returned from cache and no actual request is being made
  • AJAX requests - although none of load testing tools can actually execute client-side JavaScript if your application uses AJAX technology and it generates network requests - you need to simulate this as well as browsers execute JavaScript in asynchronous manner and the majority of load testing tools cannot kick off extra threads

And last but not least, if your application assumes different use cases by different user roles you should consider creating different independent groups of users which will map to your application use cases (i.e. anonymous users just crawling, logged in users filling some forms, etc) and the workload will more or less represent real-life distribution.

So the main idea of good load test is that this test should be realistic, otherwise it does not make a lot of sense as you will not be able to tell for sure how many simultaneous users you application supports providing reasonably low response time, when it breaks, does it recover when load decreases, etc.

Dmitri T
  • 154
  • 1
  • In this particular case, the requests are not generated by users, but rather sensor data. Much of what you stated is not necessarily relevant in my case. However, I completely agree you've, when testing user interfaces. There are a LOT more variables to consider in that case. When testing APIs though, with very small messages and a predictable structure (as is my case), I think there are many differences. – foxtrotuniform6969 May 29 '18 at 12:09