-1

My question is similar to Leaving intentional bugs in code for testers to find:

We don't do this at our firm, but one of my friends says that his project manager asked every dev to add intentional bugs just before the product goes to QA...

...If you ignore the scenario were one of these intentional bugs gets shipped with the final product, what are the other drawbacks we should consider before even thinking of adopting this approach?

The difference is that I had different premises. I wanted to make a point without messing with anyone. Intended bugs not for motivation but for testing the testers, to point out that quality assurance process is dysfunctional.

I was working on a rather large and important feature. Critical part the of feature was that users could get a discount under certain circumstances. Problem that we had with that project was that QA was very sloppy and missed a lot of important bugs. On the other side project manager had a lot of faith in QA. In my opinion bad QA is worse than no QA, because it only adds illusions about quality. Couple of times I got into arguments with PM but I gave up and just decided that I should be my own tester and just assume that QA doesn't exist.

At some point a brilliant idea came to my mind - to test the QA. I found completely outrageous bug and decided that instead of fixing it I would leave it as is and see if the QA finds it. Bug was really bad. When user placed an order and got a discount, the browser redirects to payment provider and there user sees price without discount and is charged accordingly. This was a critical path bug.

Long story short, QA didn't find a bug, I fixed it before release of course and even created a ticket and politely announced it without blaming anyone to see the reaction. Both QA and PM didn't really cared, just said okay and that's it.

After that I finalized my conclusions about our QA but PM still had a lot of faith in him. I told some time after to assistant PM in private conversation, she told me to speak to his management, but I decided not to pursue and just carry on on my own because I really didn't know how to present the case.

So my questions are:

  1. Is it good idea to test testers by leaving bugs or even creating them?
  2. How to handle the situation where it happened, intentionally or not?
  3. How to otherwise point out low efficiency of QA?
  4. How to do it without forcing people losing face and creating tensions and conflicts?
Community
  • 1
Andrey
  • 1,159
  • 2
  • 10
  • 14
  • 2
    And then how do you test the tester-testers? ;) – Mason Wheeler Jun 04 '15 at 19:00
  • @SJuan76 to certain extend, I would even say to high. But I present a specific case which I failed to resolve and still don't know how I should have done it. – Andrey Jun 04 '15 at 19:02
  • @MasonWheeler yeah you need a way to test that tester-testers indeed left a bug for testers. Or maybe they made a bug in bug code so that it negated it. Complicated stuff indeed. – Andrey Jun 04 '15 at 19:03
  • 1
    @MasonWheeler sort of like taxes: who audits the auditors? I say, [turtles all the way down](http://en.wikipedia.org/wiki/Turtles_all_the_way_down). –  Jun 04 '15 at 19:03
  • @SJuan76 I think this is meaningfully different because here, the motivation seems to be proving a deficiency with QA so that management might change something, while that question is more about ensuring QA remains effective over the long-term. – Ixrec Jun 04 '15 at 19:04
  • @Snowman if speaking seriously it is exponentially easier to verify that bug exists than to verify that it doesn't, so testing the tester-testers is much easier and PM or teamlead can perfectly do it. – Andrey Jun 04 '15 at 19:05
  • @Andrey correct, it is possible to prove that bugs do not exist **however** for any non-trivial program the complexity explodes to the point it is equivalent to the [halting problem](http://en.wikipedia.org/wiki/Halting_problem). –  Jun 04 '15 at 19:07
  • @Snowman yeah in this case it is to check if the program halts (produces incorrect result) at given input which is trivial. – Andrey Jun 04 '15 at 19:10
  • In addition to the link to the related question (see my edit which improved on it) I also recommend explaining why the accepted answer does not answer this question. While not strictly required, it does help us better understand what you are trying to gain by asking your question that was not already said previously. –  Jun 04 '15 at 19:21
  • I think its a good idea. you'll get a measure 'percentage of bugs found' but not to do it without telling the testers!!!!!! – Ewan Jun 04 '15 at 19:28
  • Do you test the people gathering the requirements? Customers will tell you if the got it right. At what point is a programmer at fault for introducing too many bugs into the system whether QA caught it or not? – JeffO Jun 04 '15 at 20:18
  • Your concerns about the bugs are valid. Please consider a completely different approach to correcting the situation though. You'll notice that a LOT of different people are giving you similar answers and suggestions. Instead of defending why _you_ think your actions were ok I would suggest reading all these comments with an open mind and then thinking about it for a day or two. – Michael Durrant Jun 04 '15 at 20:55

3 Answers3

8

Is it good idea to test testers by leaving bugs or even creating them?

No, you don't want your employees playing petty games about who can stump who, rather than delivering actual value.

If your QA department is not delivering value you likely:

  • Have a company who assumes testing is not worth paying for and hires duds for QA roles
  • Don't have documentation about functionality to test
  • Have bad/nonexistent tests for key functionality

How to handle the situation where it happened, intentionally or not?

You are a professional.

Professionals don't play silly jokes to "gotcha!" their coworkers. So, don't do it again.

How to otherwise point out low efficiency of QA?

Do your customers complain? Most people with pay levels able to affect this sort of problem don't care if a dev is upset.. but they care if the customer starts having significant problems, bugs, or otherwise lost revenue.

How to do it without forcing people losing face and creating tensions and conflicts?

Document how this affects the end user and show management that insufficient QA processes exist.

Realistically, you probably need a combination of:

  • More automated testing
  • Better defined QA requirements (if you have user requirements, convert these into stories/activities QA must do before a release)
enderland
  • 12,091
  • 4
  • 51
  • 63
  • "don't do it again" I don't see any harm in playing such joke if I keep it to myself. I did it solely to understand if I can rely on QA or not. My job is to deliver quality product, from that moment I just assumed that I need to be my own QA, test more, write more unit tests etc. – Andrey Jun 04 '15 at 20:37
  • 3
    @Andrey you can do whatever you want. As a professional, I'm not going to intentionally submit bugs just to "gotcha" our test team, I'm going to approach problems in my workplace professionally. And ignoring that, just imagine you do this and are gone suddenly for some reason - illness, vacation, new job, etc - and those intentional bugs get into production. Imagine how _that_ conversation will go when your boss realizes you intentionally put serious bugs into the codebase... – enderland Jun 04 '15 at 20:43
  • There were no immediate customers, it was public web site. We always received a lot of feedback from users, so variations were not really significant. Very small majority of users were paying so it is not always trivial to judge whether many clients suddenly became significantly unhappy. Lost revenue is also not immediately evident. – Andrey Jun 04 '15 at 20:45
  • as a professional I must realistically assess the situation, if I don't frame anyone and keep those experiments private I think it may be useful instrument. In that particular case I knew that nothing would go live without me. Funny thing that in the end we still found couple of bugs actually after project went live. – Andrey Jun 04 '15 at 20:51
1

I think this is the critical part of the issue here:

I was working on a rather large and important feature. Critical part the of feature was that users could get a discount under certain circumstances. Problem that we had with that project was that QA was very sloppy and missed a lot of important bugs. On the other side project manager had a lot of faith in QA. In my opinion bad QA is worse than no QA, because it only adds illusions about quality. Couple of times I got into arguments with PM but I gave up and just decided that I should be my own tester and just assume that QA doesn't exist.

I will answer your questions in the context of that paragraph.

Is it good idea to test testers by leaving bugs or even creating them?

No, you test your testers by measuring legitimate defects that made it through QA. Sure, if they miss an esoteric bug that is nigh impossible to reproduce or find then go easy on them. If they report a bug that project management chooses to ignore then it blows up with the customer you absolutely do not make it into an issue with QA.

How to handle the situation where it happened, intentionally or not?

If your QA are lazy and let obvious or major bugs through and your testing procedures would have caught them then this becomes a performance evaluation issue. I would not advocate employee performance being strictly related to bugs (e.g. you let 25% of the defects through this year, your annual raise is being docked 25%) but clearly if a QA employee is not finding obvious bugs (you know, his job) then maybe he should not be employed anymore.

How to otherwise point out low efficiency of QA?

Measure it, but be reasonable. See my explanation above. Everyone makes mistakes. But if a manager is hesitant to assign a QA employee to a high profile project because of performance concerns, well, his gut feeling just figured it out.

Yes, I have seen this happen.

How to do it without forcing people losing face and creating tensions and conflicts?

Not going to happen. You can be polite and professional all you want. At some point, QA are simply allowing too many bugs to slip through and management must address it or face economic consequences (e.g. customers jumping ship to competing products).

The alternative is to use management tools to address the poor performance: training, assign to an easy project, paired testing, and eventually, the other path: written reprimands for poor performance, and eventually, termination.

0

Low-quality QA is wretched, and if you can't get Management to recognize that they're not doing their job, that's even worse. But the only person that's going to be blamed for intentionally putting in a terrible bug is you - for wasting everyone else's time, and for putting in a bug that, if QA is as bad as you say it is, will go into production and ruin the company.

Best case scenario - you get blamed and lose your job. Worst case scenario - you get blamed and everyone loses their job.

Zibbobz
  • 1,522
  • 3
  • 13
  • 20