-1

I had a conversation with a colleague about tests. They want to be able to access a db that is only accessed by tests. The db is created in memory and deleted after tests are done. My colleague wants to be able to access the db so that if an unexpected thing happens, they can check the db. I feel that tests should be able to tell you where you went wrong. This has led to two questions.

  1. Should we write our application in a way that a developer can access the db.
  2. Should developers be allowed to run tests across non-dev environments.

Your opinions will be appreciated.

Njuguna Mureithi
  • 101
  • 4

3 Answers3

4

I feel that tests should be able to tell you where you went wrong.

I get your point, and it's a principle I very much agree with. When you open the door to debugging the db, you lower the developer's incentive to put effort in making tests that adequately explain the issue when they fail.

However correct I think your argument is, you do need to allow real-world compromises. No test is perfect. If anything happens to the database, unrelated to and not caused by your code, then the test result might report an issue because it detected an anomaly, even if your tests are not at fault. Or maybe your tests forgot to cover an edge case.

When you force your developers to rely purely on existing tests, then your developers' work can only be as correct as the tests themselves are. If your tests contain any flaws whatsoever, developers won't just be unable to fix it, but they might even be sent off in the wrong direction, hunting for a bug they think exists because a flawed test told them so.

This is a tough choice. One the one hand, you want to strongly incentivize the developers to rely on automated tests as opposed to their own aptitude at debugging when looking at the data.
On the other hand, completely blocking access to the database implicitly assumes that your tests are perfect. It's impossible to definitively know that your tests are perfect. The best you can get is "not having seen a problem yet".

You can create an obstacle instead of a ban. E.g. only allow a subset of the developers access to the resources (e.g. the technical leads or seniors). That means that your devs will first have to talk to your experts before they get access to the resource, thus incentivizing them to not immediately want to run to the database for every minor issue they encounter.
When a problem arises that genuinely requires access to the database, the experts will be able to provide it. When a problem arises that doesn't require access to the database, they'll be able to block it.

  1. Should we write our application in a way that a developer can access the db.

There are compromises between no access and full access. But even more importantly, I don't think a dev needs access to the database, but rather the data that was in the database. That's an important distinction to make.

For example, you can dump the database content to a log file when a test error is encountered (make this a configurable option so it only happens when a dev explicitly asks for it).

Just to be clear, "dump the database content" is an overgeneralization. The point I'm trying to make is to log the relevant data. That way, your developers have access to the data (in log format) without needing access to the database itself.

  1. Should developers be allowed to run tests across non-dev environments.

No. There is a better alternative: allow developers to copy a database from a non-dev environment to a dev environment, and then they can test what they want.

This may be complicated because of GDPR/privacy rules, but that's a different discussion.

The only reason to look at a non-dev environment is when you're looking at an environment-specific issue. Require your developers to first prove that they're unable to reproduce the error in the dev environment (with the exact same data), before allowing them to go outside of the dev environment.

Flater
  • 44,596
  • 8
  • 88
  • 122
2

The tests should be self-contained, so using in-memory DBs is a great approach. But this should not prevent you from creating a useful test suite: if having access to the DB after a test failure aids debugging, then by all means make that access possible.

In most cases the diagnostics in the test suite itself should be suitable to debug a problem. This shouldn't just be a “test X failed”, but “test X expected data foo, but got data bar”. Which kind of information to show depends on your context. Preserving the DB after each test failure would likely be excessive. Instead, add an option to your test runner so that it can save the DB upon failure instead of tearing it down – if a developer wants to debug a problem they can try reproducing the failure with that switch activated.

Note that an in-memory DB is not always ideal, e.g. if your production DB is totally different. At some point you'll have to test against your production setup. Usually you'd have separate integration tests that cover this. Another option would be to make your unit test runner configurable so that it can connect to an external DB, but defaults to the in-memory DB. Note that this can have the disadvantage of making your tests more fragile.

amon
  • 132,749
  • 27
  • 279
  • 375
0

I feel that tests should be able to tell you where you went wrong.

How would that work?

Lets look at a pseudo-test:

// arrange
var name = "John";
var userService = SetupUserServiceWithMemoryDb();

// act
userService.AddUser(17, "John");

// assert
Assert.IsInDatabase("user", "John");

Now that test can tell me that it failed. It can tell me that "John" is not in the database. But it cannot tell me why. Without access to the database, I will now have to step through every piece of my program to find out why.

With database access, I can at least see where I have to start looking. Maybe the database table contains a lower case "john" because I forgot usernames get normalized to lowercase. Or maybe I put "john" in the Id field and his "name" is now 17. The point is, I can find out what went wrong without guessing. Because there is no way to write a test to tell you what went wrong. Letting me dig through each line of my code because you want to impose a standard seems pretty disruptive. Why would I be banned from accessing the RAM on my very own machine?

Should we write our application in a way that a developer can access the db.

Why not? At least read-access is definitely helping with finding errors faster. I don't know why write-access would be necessary, but again... why not. Assuming it costs next to nothing to dump the database to disk after a test failed, go for it.

I'm all in favor of having automated tests to a specified standard, but you need to do that explicitly. Don't restrict access to valuable information because you think it might indirectly lead to people following a certain standard. If you want that standard, use whatever way your company favors to set it and enforce it.

Should developers be allowed to run tests across non-dev environments.

Ah... no? I might be missing the context here. The whole in-memory dev environment is precisely to not have to allow that. I am assuming you have an environment that is exactly like production in dev. So the same database system for example. Not for every test, but for integration tests before something goes live.

Now, bugfixes will certainly be allowed, somehow the data has to be repaired once bugs crept in. But Testing? No way. That's what the testing environment is for.

nvoigt
  • 7,271
  • 3
  • 22
  • 26