How to encourage client to do some in house QA testing?

Question

Update/Clarification My client understands the need for their in-house testing and he/they always swears they will "do better" (i.e. do something) but it just doesn't happen. They don't have the budget for outside testing. I guess I am asking (vaguely, I acknowledge) about what could instill a "test early, test often, test on target machines' ethos?

Question: how to encourage users to take the time to explicitly test and report issues with new releases, not to "test-as-they-go" in production projects.

Background: I have a small client for whom I have written a suite of multimedia presentation tools. They are a nice client and we have a good relationship. The project is on-going, adding features as we go along.

There are two issues I have:

1. Feature definition is done on-the-fly, often over the phone, subject to change, revision, reversal. (a bit of like Kennedy's "We will go to the moon and do the other things" – I've always been amused by the "other things" part of that)

2. Virtually no QA testing is done on their end.

I can deal with #1, more or less. This is not a client who would even read a spec before a meeting, let alone write up one. I'm used to it. It is item #2 which I have the issue with: they don't or won't test new releases. What they do is to use them for production so when bugs come up, they either find a workaround and don't report it, or they are so in a hurry to get on with the project, that bug reports are vague.

We have had many discussions about all this but I have only been able to nudge them a bit (e.g. we use github for issues tracking – though mostly I use it). The root reasons are twofold: they are a small consulting company and don't have (or don't think they have) the resources for testing (nor the budget to outsource it). And cultural: although they think of themselves as "developers" they really are just users of a multimedia software package. (e.g. they have none of the obsessive neurosis attention to detail of "real" developers).

This affects me as you would expect: without feedback I can't tell if a feature is complete (see #1), or if there are other consequences. It also is making me a bit lazy.

Answer 1

they have none of the obsessive neurosis attention to detail of "real" developers

Preface: The kind of language you used here is typically a red flag for me. When I hear people talk about "real" developers or the (one and only) "right" way of doing things, I start thinking of tunnel visionned cargo-cult developers.

Now, I'm not saying that you're definitely one of these developers (I don't have enough evidence to assert that), but if you are, then you might benefit from this answer.

Answer

It sounds like you and your client are optimizing for different things. It's an unfortunate fact in software engineering that often the needs of the business and the desires of the developers don't necessarily line up.

Software developers are often passionate people with a focus on improvment. They like to improve software performance, the development process, business processes, communication methods, etc. And that's great. Focusing on these things is what separates the craftsmen and craftswomen from the mindless keypushers.

But, your client isn't a software craftsperson. Your client is a business with a completely different set of priorities. And, sometimes, those priorities look ridiculous to we software craftspeople... but that's only because we're optimising for different things.

Business frequently want to optimize for things like early release to market and short term cost savings. In doing so they may need to sacrifice things like QA, User Experience, long term cost savings, and other things that make developers tick.

Is that a bad thing? Well, not necessarily. I can't speak for all businesses, but in my experience my clients do these things in order to increase their own ROI (return on investment). Doing things like QA, UX refinement, and long term planning offer them lower ROI. Even worse, many businesses have investment structures that only reward short term wins as opposed to sustainable approaches and long term wins.

So, while you could try to sell the idea of QA to your client you may be wasting your time and straining your relationship with them. In the best case you'll get someone eager to try out your ideas (unlikely). In the worst case you'll have to convince the entire company to rework its incentive structures so that long term investments such as QA are rewarded. In either case, your odds of success are low.

Answer 2

The interesting question is when you get paid, not whether your client does any testing of their own.

• if you get paid based on your time, no problem.
• if you get paid in advance, no problem.
• if you get paid when the client declares the project “done”, big problem.

The problem is how you can know when the client accepts the software and will pay you. This clearly doesn't work when the client continuously amends the project with vaguely defined new requests. If this means that pay day is always deferred – and becomes more unlikely by each request – this becomes untenable for you.

A fixed contract that carefully specifies all features and defines under which conditions the client will accept these features is clearly very uncomfortably strict, but it allows you to plan the project in advance (also the next project). It also guarantees you'll get your money for the software you delivered, if it lives up to the spec. In such a scenario, the only responsibility of a client is during the contract definition phase, and at the end for acceptance testing.

Such acceptance testing done by a client is separate from other forms of testing:

• unit tests
• system integration tests
• usability tests
• load tests
• pre-release tests

As far as possible, you would anticipate the acceptance tests and perform them yourself before delivering the functionality as to avoid any embarrassments. Aside from acceptance tests (which only measure contract fulfilment, not software quality), all Quality Assurance is your responsibility. In particular, your client does not necessarily have a QA mindset, the necessary technical background, or the contractual obligation to do QA. Also, I find outsourcing bug hunting to the client quite unprofessional.

That is not to say that bugs wouldn't happen. Assuming you have a project-based relationship to your client, you'll want to walk a line between being courteous and rapidly providing fixes, and explaining that they have accepted the current release as sufficient for their needs – large changes require a new contract. If you have an ongoing support contract, you'll of course have to stick to your agreed service level.

In an agile setting, responding to client needs is valued more than sticking to the letter of the contract, but you'll still want to get paid. Therefore, many agile-oriented project methodologies value close client interaction, to the point that the client might become part of the team. You could then always talk to this “product owner” to clarify any necessary points. Since the PO has the authority to grant you the time to work on any feature they find valuable, this can work even when starting with vague client needs. If you do not have such a close communication, you will need to follow a more formal approach.

• When you learn of new customer needs, work with the customer to translate them to requirements. This helps the client to get what they actually want.
• Requirements are objectively measurable – they are either fulfilled or not. This saves the client from half-solutions that only kind of work.

• All client requests must be provided in written form so that you can bill them. This protects them from being billed for stuff you just felt like working on – such as rewriting the entire interface when asking for a button to be aligned differently.

  A lot of communication can be done in person or over the phone, but at the end you'll want a piece of paper to document that the client wanted you to work on these requirements. In simple cases, it might be sufficient to recap a phone call and send them an email to verify what they asked you to do.

Bug reports are always difficult. If your clients are devs themselves that should help since they can understand your needs: having clear steps to reproduce. A simple way to get powerful insight is to enable logging in the deployed software. Provided that the data privacy issues can be worked out, requiring every bug report to have the current log attached not only guarantees some written communication, but also tells you what the user actually did (in contrast to what they thought they were trying to do).

Answer 3

The way to encourage communication of bugs is to encourage frequent, granular communication of features. If you train a company that they can ask for anything with zero ceremony, they'll use that feature for minor bugs, too. Give up on changing to your client's workflow unless these changes make their life easier.

Getting your client to do in-house testing is tough, but getting them to actually report bugs is not as difficult as it sounds. The way to get more feedback is to reduce user friction...even if that means transferring some of that friction to yourself.

1. Use simpler tools, even if those tools are inadequate and inappropriate. E.g., BaseCamp is a pretty awful bug tracker (because it's intended for project management), but people are actually willing to use it.

2. Since the bug trackers we were using did not support image copy-paste, I actually wrote a trivial program which copies the current clipboard image to disk (as a Guid), then copies the Guid to the clipboard. After minimal training, a user could attach clipboard images to issues by just hitting print screen, clicking a button, and then pasting into the file chooser dialog of the bug submission tool.

A screenshot (possibly edited in MS Paint with annotations) and 1-2 sentences is enough to pinpoint most features/bugs.

Both of these suggestions are targeting friction points which I experienced, and both of these suggestions increased reporting by a factor of more than 10. However, you will need to target your own friction points.

Answer 4

Make the testing for your client easy, but make it really hard for your client to use any new features in an untested version in production. This can be accomplished as follows:

Whenever you deliver a new feature, you implement this first in a "beta version", clearly marked with a sign "not for production". You make this beta version available to the client for testing. You also provide the latest "production version" he shall use for real production (without the new features, but with the latest bug fixes), and you refuse to transfer the new beta features into the production version until you got a feedback that someone at the clients side has at least tried it out first.

If the client starts to use the beta version on his real production data though it shows always a big message "Not for production use" whenever he starts the program, then you cannot help him, but at least you made clear that whenever he looses production work because he used the beta for wrong purposes that it is clearly his fault. If the client does not learn from that, you may consider to disable your client's ability to use the "beta" in production by deactivating some crucial functions like saving the results to disk in the "beta", if that's necessary.

Providing a separate "beta" will force you to establish proper version control / configuration management, so you can manage a production branch and a beta testing branch side by side without hassle. But since you are working with Github, I guess you already use something like GIT, which makes this kind of management very simple.