Outsourcing Quality Assurance and Testing

Question

I was recently approached by a software firm that specializes in Quality Assurance and Testing.

Up until this point, the developers at our (small) company have been responsible for their own QA for the most part and we've had mixed results. We're at the point where we are ready to hire a full time QA guy, but I was curious as to whether or not other's have used teams like this in the past and what the results were? I'm a bit skeptical but trying to be open-minded.

tl;dr What was your experience with an outsourced QA/testing team? Pros and cons?

Answer 1

I would recommend against this for one main reason that has nothing to do with the competence and capability of the particular outsourcing firm which has approached you (which I can't judge).

For QA to work, once you get past the `developers do their own QA' stage, it needs to be a first-class member of your planning and management process. By nature, development and QA are going to be in a (hopefully friendly, but) oppositional relationship -- this follows naturally from their competing goals. After all, what are the main goals of the developers?

• to be done with the current release, so they can start working on the great stuff they've got planned for the next one

• to get every feature they've worked hard on and completed at least a rough cut of into the current release

• to spend time hacking neat new features instead of fixing newly discovered bugs in old code

meanwhile, the goal of the QA team (or the QA guy) is to not let any release out the door until all of the bugs are found, and all of the found bugs are quashed.

The problem, of course, is that they can't both be right, and -- as importantly -- neither is right all of the time. Sometimes that great, but not-quite-ready feature does have to wait for the next release. Sometimes the release does have to ship, even with known (and clearly documented!) bugs, so that the developers can move on to the next one. Compromise is going to have to happen -- and it can't happen if either of these two groups is subordinate to the other. For this reason, the developers and the QA team must have an equal seat at the table when these decisions are being made, and someone higher up must have the final say when considering their conflicting positions.

For this reason, I recommend avoiding any corporate structure where developers are subordinate to the decisions of QA, or QA are subordinate to the decisions of developers -- and this will be very hard to do if the developers are in-house, and the QA team is outsourced.

Answer 2

QA / Testing has an image problem. I've done my fair bit of freelance software testing and it has always been valuable to the clients. I've mostly worked with startups, small software houses and digital agencies.

The need for outside testing help often arises when these companies either have lots of work on or are not quite ready to commit to a full time resource. Having the ability to call upon a freelance tester at short notice can be extremely valuable.

The problem that many companies face is lack of really good testing and testers. Getting a good tester on board can really help a project launch confidently. Getting a bad one will just slow things down. I run a community of software testers so have come across my fair share of great and bad testers.

Asking a tester a few basic questions should easily filter out the incompetent.

Tell them about your needs and see if they ask questions or recommend a solution straight away. The good testers will ask the right questions.

Ask them about projects they have recently been involved in. What was their biggest challenge?

Ask for recommendations/references.

Ask them what their best skill is, or what their focus on testing is. Testing is increasingly becoming specialised, it's hard to become an expert in specific fields.

The best testers will also drill down super quick into important (not superficial) problem areas.

Hope that helps.

Answer 3

Good testing is as important, if not more important than development.

Not just anyone can test your stuff. You need to find people with lots of expertise in your problem domain. They should be people with as much, if not more, domain knowledge than your developers. If you're developing a health care app, for example, hire a nurse or health care technician that uses that stuff.

In short the best testers are expert users. They know the problem domain, they know the problems you are trying to solve, and they know them better than you do. Hire lots of these people, they will find an order of magnitude more real problems then the random guy off the street following a rote procedure. The random-guy-off-the-street may generate a lot of noise too, they have no idea what problem you're trying to solve. The expert user takes one look at your app and can get what you're trying to do right away.

Answer 4

I've had a mixed bag of experiences, most of them bad. In general you get what you pay for but that is not always true. I've worked with QA guys that made themselves so valuable to the team, that I'd say that they were a productivity multiplier.

If I were given the choice, I'd always choose a top notch professional QA guy with experience, over several out sourced resources.

Also, even if it is a good outsourced team. You still have to deal with the overhead of communication and time zone issues, that even with the best people on both sides can become frustrating and expensive.

Answer 5

I've found that outsourcing QA work can be effective, if your expectations are realistic and you have a fairly hands-on approach to managing the effort.

I was responsible for a pilot project at Microsoft to take advantage of 3rd party vendors in China for localization testing. In order to make this work, I conducted in-person training and post mortems every three months. I did pre-triaging of bugs that came in to avoid distracting our "war team" with poorly written or inadequately regressed bug reports. It took some effort, but we got very good value out of this process; the trained people rotated out into different teams as the effort expanded. Some of the better people in that first group that I trained are actually now doing executive-level and business development work. This effort was very successful, and I'd do it again in a heartbeat if I were in a position to start a similar project in my current situation.

On the other hand, I've also worked for a company which outsourced some QA effort to India. Their approach was to provide a vague guideline for what needed to be tested, a requirement that the test automation be written into Java, and no dedicated internal resource was assigned to manage the effort. Essentially, they hired the company and "threw the task over the wall." They got terrible, unmaintainable code with lots of static member variables, tight coupling, temporal issues, and a ridiculous rate of (irreproducible) false failures. We first spent about a month or two trying to salvage the code, but we ultimately had to throw away everything they did, and it cost a lot of money.

So, no matter how your vendor pitches it, don't think that outsourcing QA will completely free you from managing the effort. What it should do is provide an effort multiplier, if you manage it correctly. Expect to dedicate at least one full time person inside your organization, preferably someone with QA and Project Management experience.

You may save money over hiring an equivalent team internally, but that's not guaranteed. You will be insulated from the effort of qualifying talent and hiring and firing people, which is worth quite a lot if your vendor finds quality people. But you will have a fair amount of hands-on work to do. The best thing about outsourcing is that you can scale up or down the effort fairly quickly based on the state of your project. If there's not much to test, you're not paying as much.

Given the opportunity, I'd generally rather hire people committed to the success of our company to do QA. But if you don't have the internal expertise to do that hiring, or the budget for a large internal full-time team, it's definitely worth considering.

Footnote: The country that our vendors were in wasn't a major factor in the success or failure of our projects; the level of involvement in managing the effort on our side was the critical factor.