12

Some parts of a game are easy to test in an automated way (logic, maths, input handling); but there's also a lot that's purely visual and not easily testable.

I would be surprised if the games industry left all of this to manual testing; there's enough money in it that I would guess that effort has been put into being able to regression test at least some visual aspect of games.

Is this true? If so, what are the possible ways that rendering of games can be tested? Capturing output and comparing images (can this be reliable?)? Intercepting data from the graphics card at a low level? Capturing vertex information (etc.) on its way to the graphics card? It seems like there are lots of possibilities; but I can't find any information on this :(

Note: This question was marked as a dupe of this one, but I'm not asking about specific tech/frameworks/tools on how to do this, but more broadly the ideas around this practice, and what the actual games industry do (if they do at all).

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
Danny Tuppeny
  • 896
  • 6
  • 13
  • I would say that probably they just use a well tested graphics library (either third party or developed in house). When they get to designing a game level, they do not check that every object is *exactly* at coordinates X, Y, Z, or if the perspective is right, but simpler things as ensuring that objects are there and not blocked by walls, etc. And if the graphics library is in house developed, the tests are automated and more basic. – SJuan76 Sep 28 '14 at 19:50
  • @SJuan76 I guess if you're building on top of an engine, the question applies to the testing of the engine – Danny Tuppeny Sep 28 '14 at 19:57
  • Isn't this question somewhat wider in scope than the one it was marked as a duplicate of? The OP did not mention C++ or OpenGL. The game industry is bigger than that. – toniedzwiedz Sep 29 '14 at 16:54
  • It's probably easier to formally prove that the code is correct given that the graphics APIs are bug-free than it is to test that the hardware is doing its job. Since AMD and NVidia's drivers are proprietary, I imagine whatever protocol is involved in communicating with the graphics card is 1) proprietary, 2) subject to change, and 3) varies from card to card. Besides, even if the data sent to the graphics card is correct, how do you know the graphics card doesn't have a hardware flaw? – Doval Sep 29 '14 at 17:58
  • http://meta.programmers.stackexchange.com/questions/6483/why-was-my-question-closed-or-down-voted/6490#6490 – gnat Sep 29 '14 at 19:03
  • [I didn't think the game industry did much automated testing](http://gamedev.stackexchange.com/a/3427/24969) (nb the answer is about TDD but I believe it applies to all testing - iterative *product design* (not just code design) makes testing largely redundant until the product is finalised, and then there's no point because it's being shipped) – Alex Sep 30 '14 at 09:41
  • 2
    I tried reopening this since the dupe target was more specifically about strategies in automated testing of OpenGL graphics. This isn't my speciality but I trust the feedback from others who know the game industry very well so I want to give it another chance. For reference to the similar question see here: http://programmers.stackexchange.com/questions/150688/how-would-you-unit-test-or-perform-the-most-effective-automated-testing-on-graph – maple_shaft Oct 01 '14 at 00:56
  • +1 I'm interested in this because it seems like answers here could be applicable to other non-game development areas as well. – J Trana Oct 01 '14 at 03:31
  • Shouldn't this go to gamedev.stackexchange.com? – Laurent Couvidou Oct 01 '14 at 03:58
  • @LaurentCouvidou Maybe; though it was more about practices and ideas than specific coding; so I thought it would be more appropriate here. – Danny Tuppeny Oct 01 '14 at 07:55
  • Don't underestimate the human ability to intuitively spot the tiniest visual glitches. This is reflected in both (a) the developers spotting issues well before getting to the QA testing stage and (b) testers noticing small glitches. Even if people can't explain what is happening, they can tell that something is off. – Flater Mar 01 '21 at 11:58

1 Answers1

1

Any company will do in different ways, even different games will be tested differently.

  • Maybe the lack of information about graphics is because rendering is indeed quite "repetitive" (I cannot think a better word, sorry) and in a typical complex scene there will be enough primitive combinations so most persons can usually see easily most of the glitches (with the exception of less used shaders, but a stress-shader special level/demo can be used). Remember that humans were hunters for thousands of years :) so probably we are engineered to see glitches.
  • Also the freedom of movement present in most games and the randomization of other elements on a typical game that makes to feel it as more "realistic" are usually a nightmare for applying pure automated tests: in fact a human tester will find more quickly and more errors than any automated test. You can code a logic for record timings, stats and positions/angles/other variables whenever a tester is playing... so even on no-automated tests you will have feedback like-automated-tests. It's usually useful to be capable of making a playback of any beta tester recorded session (and usually is a good feature for the final game as well).
  • About rendering stats, for example you can get: the draw-call count stats for each one of the shaders (as you will make the calls, it is enough to maintain some counters), the upload counts (the same, you will control when gpu buffers are updated), the timings (not only fps, also the lapses between each call), etc. If necessary (but I believe that is not recomended except when hunting for a specific error) you can get stats from the graphics card itself (for example you can get fill rates and others by using occlusion queries).
  • Finally, other reason to use beta testers is that different graphic cards will have probably different behaviors... you may need to have a farm of testing machines o_O. The beloved human beta testers will soft all that mess.

Hope that it helps!

Disclaimmer: I'm not a betatester, I'm a developer XDDD.

user1039663
  • 121
  • 2