6

I'm managing a team in .net. They are writing unit tests, use them locally on a regular basis, and love it. However, they keep pushing to have the unit tests kept as a separate project and only want to run them on a communal build server once a week or so.

In contrast, our JS teams are running tests on the build server on every pull request.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

It's boggling my mind and I'm wondering if there is something I'm missing.

What are likely reasons for this kind of pushback? They are happy to write the tests and they tell me they have close to 100 tests already written on branches they refuse to merge into their main branches until "we come to a decision." ( I.e. I tell them it's ok to keep tests in an isolated project far away from the main code.)

The team was quick to adopt writing unit tests and they find them useful and continue to write more. I am now trying to get them to run the tests in the CI and that is where I'm hitting resistance.

Doc Brown
  • 199,015
  • 33
  • 367
  • 565
Bob
  • 846
  • 7
  • 16
  • 2
    How long do the tests actually take to run? – Telastyn Mar 16 '18 at 15:19
  • 16
    _"or what happens when the tests prevent an urgent build"_ But isn't that build somehow useless if there are failing tests? – πάντα ῥεῖ Mar 16 '18 at 15:27
  • 3
    Possible duplicate of [What's the point of running unit tests on a CI server?](https://softwareengineering.stackexchange.com/questions/308515/whats-the-point-of-running-unit-tests-on-a-ci-server) <--- TL;DR stop wasting developers time and let machine automatically run the tests – gnat Mar 16 '18 at 15:27
  • 1
    @telastyn currently they take 3 seconds to run. – Bob Mar 16 '18 at 15:29
  • @gnat I'm currently trying to get the CI to run the tests but that is what they are resisting. – Bob Mar 16 '18 at 15:30
  • It sounds like you already know what the reason for their resistance is - they fear the builds will be slow and they're worried about the tests preventing an urgent build. Do you not believe the reasons they're giving you? – combinatorics Mar 16 '18 at 15:30
  • ”But isn't that build somehow useless if there are failing tests”-- exactly my concern and why I think it's possible I'm missing something. -- maybe they think the tests will actually always be passing cause they won't commit unless it passes on their machine? – Bob Mar 16 '18 at 15:31
  • @combinatronics yes I doubt that is the actual reason, cause 1. Tests finish in seconds, and 2. the the whole point of the tests is to prevent a build if there are bugs. – Bob Mar 16 '18 at 15:34
  • 1
    Would it be possible to get the team to agree to try running the tests on a CI server for a few weeks/months, with the option to disable the tests if it turns out to delay a critical build by more than, say, half an hour? Then the team can get some experience with running the tests in an automated manner and evaluate afterwards if they want to keep it. – Bart van Ingen Schenau Mar 16 '18 at 15:40
  • 3
    sorry but per my reading you simply misunderstood what they want, they don't really care how frequently tests are run, they only want them to run in a non-blocking manner, and that is perfectly doable with CI and you didn't show any reasons for why it shouldn't be that way (as opposed to your devs who seem to tell sensible reasons why it should). Whether it is done by extracting unit tests to separate project or by keeping within and only running these with a separate (non-default) CI config doesn't matter much, it's only a minor technical detail – gnat Mar 16 '18 at 15:44
  • @bart a good idea. If you can expound on that as an answer I'll comment if it worked. – Bob Mar 16 '18 at 15:46
  • @gnat I'm not sure what you mean. How is running the tests in a CI non blocking? What's the point of running the tests if it allows code to be merged /deployed with failing tests? – Bob Mar 16 '18 at 15:50
  • Also, how are they going about writing tests? Maybe the problem is that they are writing tests that are too closely coupled to the SUT code, so they can't change implementation details easily? – Filip Milovanović Mar 16 '18 at 15:50
  • technically non-blocking run is totally doable, you'll learn that when you try CI (assuming you picked a decent one). As for what's the point of allowing merge / deploy with failing tests, that's the wrong question - instead you need to ask what could go wrong if these are allowed (and given that as of now, when these are allowed, things seem to be okay anyway you maybe will find that things won't really go wrong) – gnat Mar 16 '18 at 15:58
  • 1
    @gnat things are not okay anyway. Our bugs are currently embarrassing and customers are complaining. We spend too much time on regression bugs. But thank you for giving me some insight into what they might be thinking. – Bob Mar 16 '18 at 16:16
  • 1
    well you didn't mention that in the question, nor did you mention whether and how you discussed these issues with developers. Anyway, running tests on CI at every commit in a non-blocking manner could help in resolving these issues too if you do it right, so you're not necessarily have to push the blocking approach through and waste your time on fighting resistance. It can even led your developers to gradually learn to appreciate and eventually adopt the blocking approach, without needless fighting – gnat Mar 16 '18 at 16:23
  • 2
    writing unit tests and not running them is like buying condoms and never wearing them –  Mar 17 '18 at 12:20
  • `or what happens when the tests prevent an urgent build.` Basically, they want to be capable to press the panic button and to build and release something untested. I would invite them to update their resumes and start looking for a new job. – Laiv Mar 17 '18 at 12:35
  • 1
    They want control of the tests. They are right to want control of the tests. If the moment I wrote a test I lost control of it and someone who didn't understand it used it against me to stop me from doing my job I'd never write a good test. Don't use tests to stop coders. Use tests to show them what expectations used to be and let them take responsibility for expressing the new expectations with new tests. Doing this shouldn't require permission. But it should be reviewed so everyone understands the new expectations. – candied_orange Mar 17 '18 at 13:13
  • Seems to me a : "oh no, If I merge into master and someone's else code make my tests to fail, I will have to waste time reviewing why. No, I'm better working solo in my branch, where everything is wonderful and works". Then, the lead dev has to merge branches with tons if changes each. A drama. I missing something here. Collaboration. Aka teaming. – Laiv Mar 17 '18 at 15:37
  • I'm voting to close this question as off-topic because **we are not your team.** If you want to know why *your team* is pushing back, **ask them.** If you want to know what the "cons" are for automated unit testing **ask that.** ... Please don't ask us to read minds -- especially minds you actually have some access to. – svidgen Mar 18 '18 at 16:45
  • ... that said, I can tell you from my own experience, if they've been working without automated testing for a long time, they *will not* make the transition quickly. This is often both a mindset and feasibility issue. **Be patient.** ... Also bear in mind, automated testing is only so valuable, and it's not worth the effort *at all* in some cases. Simplicity and good, disciplined QA generally trump automated testing. – svidgen Mar 18 '18 at 16:52
  • You can close it if you like, I was trying to give @gnat a chance to provide an answer to give them credit. gnat answered the question without reading any minds. – Bob Mar 19 '18 at 13:04

6 Answers6

8

Firstly, thank you to @Gnat for finding insight into the real issue here and giving me what I needed to know to get to the real issue.

We have agreed to isolate the unit tests and integration tests into their own projects to ensure that the added test code doesn't get compiled into the build.

We are also going to be running the tests in a non-blocking workflow until we are able to separate out the code into smaller repositories so that each build doesn't take 15-30 min.

And for now, we have agreed to not build the project on every Pull Request(PR), but instead to run the tests on a nightly build. This is what they really meant by the tests slowing things down... They actually meant doing a build on every PR. They do PRs dozens of times a day. Their main concern about tests blocking an urgent build, was really their concern that urgent builds often end up with dozens of PRs, when everyone is trying to merge their code at once, and they wanted to be able to just run the tests at the end of that process, and not require a separate build at every point along the way.

There wasn't one single answer which gave me all the information I needed, and most of the best answers came in comments, so I've created this answer to summarize up everything that made the most sense.

The important pieces of information I was missing were:

  • If the tests are too close to the code under tests, then its very time consuming to tell .net not to build the test files into the .dlls, and it's much easier to just keep them as separate projects.
  • The real issue was the build required to make the tests run. It wasn't the tests they didn't want to run, it was the build. They didn't want the build to prevent them from being able to merge code as frequently as they do. Locally, on their own computers, they are able to do a debug build and run the tests in under a minute, but on the CI servers, it takes much longer.
Bob
  • 846
  • 7
  • 16
6

I'm managing a team in .net.

Doesn't matter.

They are writing unit tests ... and only wants to run them once a week or so.

Unit tests are not run once a week. You run them before and after writing the tiniest amount of code you can to make them pass. Tests are usually good but whatever these are they ain't unit tests.

and love it.

Good.

However they keep pushing to have the unit tests kept as a separate project

Doesn't matter.

Our js teams are running tests on every commit.

That's all? Either you commit very often or you don't test much. I test locally first.

What's been left out of this story is how often you're refactoring the tests. Are these tests some entrenched guardian that takes herculean effort to change? Refactoring tests should be made as easy, and as often, as refactoring code.

I keep trying to explain to them the benefits of having the tests near the code, and running the tests often, but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

They should be allowed to put the tests anywhere they can work. They also should know how to localize a build to the classes that they actually touched so that this is a non issue.

It would be crazy to take tests designed to run "once a week" and shove them into continuous integration. Instead have them add new tests designed to run in the continuous integration that will be tuned to the performance needed there.

The fancy term for these new tests is: continuous integration tests. They are also not unit tests. The best way to categorize tests is based on the performance you need from them. Not the tools you use to run them. That way tests that behave the same way are together.

Do not take control of the tests away from the developers. Give them tools that let them control new kinds of tests.

candied_orange
  • 102,279
  • 24
  • 197
  • 315
  • 1
    "You run them before and after writing the tiniest amount of code you can to make them pass. " I doubt this team or many other follow this strict form of test first development. – JeffO Mar 16 '18 at 21:13
  • @JeffO which is fine. But don't call them unit tests if they aren't unit tests. – candied_orange Mar 16 '18 at 21:15
  • 3
    Based on the link for Unit Test, there's nothing in there about how or when they are written. it's about the scope of what they test and don't test. – JeffO Mar 16 '18 at 21:18
  • @JeffO which is true. But the longer you wait to write tests the harder they are to create. And if you don't use them for what they are for it's unlikely that what you write performs the way it should. In other words it would be foolish to assume these are unit tests since they've never been used that way. – candied_orange Mar 17 '18 at 01:41
  • `Unit tests are not run once a week.` Agreed. You should, at least, run them all every single time you build the project. IMO – Laiv Mar 17 '18 at 11:12
  • 1
    @Laiv If you mean rebuild the entire project rather than only what you just changed then that makes them regression tests. I don't classify tests based on their structure. I classify them based on what I can expect from them, how they're used, how they'll perform. How they're written is an implementation detail. – candied_orange Mar 17 '18 at 12:57
  • My comment was rather addressed to the frequence. Well written UT don't slow builds to the point Op's co-workers are afraid off. Even if them take minutes, the cost is negligible compared with the cost of pushing code untested. – Laiv Mar 17 '18 at 13:19
  • @Laiv well that's certainly true. The idea that tests will slow the build seems like a red herring to me. – candied_orange Mar 17 '18 at 13:22
  • @CandiedOrange: your answer is not bad, but honestly, your dogmatic points of what must be called a unit test and what not is neither consensus amoung professionals nor really helpful for the OP. The problem they have stays the same if they would call it just "tests coded with tool x". – Doc Brown Mar 17 '18 at 19:09
  • @DocBrown I'm truly sorry if it's not what the consensus thinks but it's at the heart and soul of the OPs problem. The op wants tests. The dev's think they have tests. The op wants to use them in ways the dev's know would not be good. What's wrong? The tests are not all things to all people. It is not useful to tell me the tests are "coded with tool x". I learned much more from "ran once a week". Tests should have useful names. If we haven't figured this out yet we need to. – candied_orange Mar 17 '18 at 19:40
  • 1
    @CandiedOrange: but the way you worded it, it gives the impression the OP has the wrong tests (and not that they use them wrongly) because they are no unit tests, so his team should write a different kind of tests. I think you agree that focussing on how to work differently with those tests is more important than how they are called. Currently, the naming focus might give some readers a wrong impression of what you are talking of. – Doc Brown Mar 17 '18 at 22:45
  • ... and please read my comment just as a suggestion for improvement, no more, no less. – Doc Brown Mar 17 '18 at 22:49
  • @CandiedOrange It could be an issue with the way I worded the initial question, but locally they run the unit tests all the time. The tests are not designed to run once a week, they just didn't want to run them on the CI server because of build times. And yes they commit dozens of times a day. Thanks for the insight regarding test location in compiled languages. – Bob Mar 18 '18 at 08:45
  • @Bob: *"locally they run the unit tests all the time"* is a direct contradiction of what I can still read in the first sentence of your question and it invalidates most of my answer ;-(( Shall I delete mine? – Doc Brown Mar 18 '18 at 09:02
  • @DocBrown and it invalidates mine as well. Pity, we were having an interesting debate about the nature of tests. Sounds like their real issue is build times. There are ways to deal with that that don't require avoiding CI tests. – candied_orange Mar 18 '18 at 11:44
  • @DocBrown My apologies for the confusion. I've updated the question to be more clear. – Bob Mar 18 '18 at 13:35
  • @DocBrown I would not delete your answer. It's still useful. – Bob Mar 18 '18 at 13:38
  • @Bob: it is an answer to a different question, especially now. Your team seems to have a completely different problem. – Doc Brown Mar 18 '18 at 13:46
5

but they keep bringing up fears of the builds being too slow, or what happens when the tests prevent an urgent build.

What happens if you should prevent an urgent build because you're about to commit some buggy code, but don't because you didn't run the tests that would tell you that you're introducing bad code?

It seems like your team is looking at the tests as an obstacle rather than a tool. The tests are their safety net, their canary in the coal mine. If the builds are too slow, then perhaps that's telling them something they need to know about the software. If they continue to sweep the slow build problem under the rug, it will continue to be a problem.

Bryan Oakley
  • 25,192
  • 5
  • 64
  • 89
  • [according to OP comments](https://softwareengineering.stackexchange.com/questions/367771/why-wouldnt-my-team-want-to-run-unit-tests-automatically#comment803332_367771) builds seem to be fast, not slow: "currently they take 3 seconds to run" – gnat Mar 16 '18 at 16:33
  • 2
    @gnat: right. I think the concern is that the overall build takes a long time, even though the testing part of the build is quick (ie: "build" is not synonymous with "test"). – Bryan Oakley Mar 16 '18 at 16:34
  • 1
    The problem looks to me as a issue of confidence. Devs don't trust in their own tests. Maybe these are not as "unitary" (deterministic) as they think. – Laiv Mar 17 '18 at 15:33
  • @gnat the build takes 15 min. The tests take 3 seconds. – Bob Mar 18 '18 at 00:14
  • I think you hit the nail on the head here. They’re worried they won’t be able to push out an “emergency” patch. The goal here is to convince them they *shouldn’t* be putting code on prod if the tests are failing. – RubberDuck Mar 18 '18 at 14:14
3

Firstly, separate projects for unit tests are good. Keeping the unit test code away from the main code is separation of concerns and all that kind of cool stuff.

Secondly, there is no getting away from the fact that unit tests, if run as part of the build will slow things down and queuing for builds can happen. You don't mention what you're using to build (TFS, TeamCity, Jenkins etc) but there are things you can do to mitigate this like multiple queues and priorities. Speak to your build manager.

If there is an urgent build that has to jump the queue, the build manager again should be your first port of call. They may be able to can lower priority builds and push the urgent build higher up the queue.

Finally, you mention CI but don't mention whether it is gated. That is: new code is kicked out in its entirety if the build or unit tests fail. This can undoubtedly be contentious as the code can in theory work, but some conditions may cause the tests to fail.

There is alas, no silver bullet here. When I was a build manager, I often encountered huge resistance when code got kicked out - especially if a deadline was looming. How often I heard the refrain: "TFS keeps crashing. Just turn the setting off and let me check the code in!". Without exception the developer was at fault. They'd either forgotten to include a file in the check in or simply failed to update the test when the code changed as it would have taken too long.

You seriously need to get into the habit of pushing back on the developer here otherwise they will run roughshod over you for ever. If another developer pulls broken code that you allowed through just because another developer starting whining, you will not be popular.

Robbie Dee
  • 9,717
  • 2
  • 23
  • 53
2

I believe its because of the typical "reluctance to change" problem we study in software engineering all the time. I can't think of a benefit to keep Unit Tests separate (I have worked in JS, Java, .Net and ABAP) but I can see the usual reason, "we have been doing it for a long time and it always worked, so need to change".

Hope that helped. i would also be interested to know if there is a genuine reason.

Talal Hussain
  • 39
  • 2
2

they keep pushing to have the unit tests kept as a separate project

Tests should go in a separate project. This is standard practice

mySolution
    myProject
    myProject.Tests

and only wants to run them once a week or so.

This is a little odd, although if they are UI tests you might run them overnight. However. Just agree and get the tests checked in.

You can have the 'why don't we run them on checkin?' and 'why don't we not allow merges when tests fail' conversations at a later date.

Ewan
  • 70,664
  • 5
  • 76
  • 161