When is it okay to ship a product with a known bug?

Question

The question is probably too broad. The reasons why are infinite. — jmq, Mar 11 '11 at 08:14
You may want to clarify this question a little. The obvious answer would be never... but if perhaps you feel you have a situation where its ok then it could help us help you with some additional generalized information... — Kenneth, Mar 11 '11 at 08:14
I will rephrase... the obvious answer would be never knowingly... — Kenneth, Mar 11 '11 at 08:28
Define BUG. We recently shipped a product that wouldnt install. Great bug :D — Barfieldmv, Mar 11 '11 at 13:31
It's good policy When you are an entomology specimen supplier. — DKnight, Mar 11 '11 at 18:13
I've always called them "undocumented features"... Does that make it okay? — Andreas Johansson, Mar 11 '11 at 19:47
@Barfieldmv, and the product came through QA without it being discovered? — , Mar 12 '11 at 11:40
@Barfieldmv but then the customers only saw *one* bug. If you had fixed that, who knows how many others they might have discovered? — Tyler, Oct 30 '11 at 18:57

score 119 · Answer 1 · answered Mar 11 '11 at 08:12

119

It has to always be OK, because there is no such thing as bugless software.

answered Mar 11 '11 at 08:12

Matt Ellen

3,368
4
30
37

2

but it sounds like he's aware of the bug... so at that point it would seem that a programmer is just being lazy to not fix it being aware of it... – Kenneth Mar 11 '11 at 08:16
7

@Kenneth: You could see it like that, but products have to be shipped, and they will always have bugs. It would depend on the severity of the bug as to whether you hold off on your deadline. – Matt Ellen Mar 11 '11 at 08:19
1

@Matt this is true. However it would seem to me that in most cases you wouldn't want to knowingly ship a product with major bugs. That would mean that the remaining bugs that you know about would be minor which in most cases would be easily fixed or at least worked around. You can't handle bugs you don't know about but if the testing process is being done correctly most bugs should be caught... – Kenneth Mar 11 '11 at 08:26
@Matt, OK I will ship you my software with a bug that causes your hard drive to be completely erased ;) – DisgruntledGoat Mar 11 '11 at 12:13
@DisgruntledGoat: is it a bug if you know it will happen? If you tell me that's what the software will do, then it's a feature, if you don't tell me then it's just malicious! ;) – Matt Ellen Mar 11 '11 at 12:20
1

Maybe my sarcasm wasn't clear... but saying that it's "always OK" to ship buggy software is kind of irresponsible. It's like saying "there will always be murderers in the world, so it doesn't matter if I go and kill a few people". – DisgruntledGoat Mar 11 '11 at 13:17
This strategy seems to work well for Microsoft. ;-) – jonescb Mar 11 '11 at 13:28
7

@DisgruntledGoat Not all bugs are equal: some are easy fixes, some are project destroying disasters. Obviously those should be fixed. Some are very rare bugs, hard to find, based on unusual circumstances, and usually difficult to fix without major rewriting. Sometimes those just have to stay in because fixing them offers too little value and the software needs to ship yesterday. Its all about cost/risk/benefit analysis. – CodexArcanum Mar 11 '11 at 14:54
@Matt Ellen, but wait, you're telling me my professional hello world software has bugs? it's just one line: `print "Hello World!";` – zzzzBov Mar 11 '11 at 16:14
1

@zzzzBov I recall an amusing lesson on bug hunting that involved a faulty "hello world". I forget the details but it probably involved text encoding or multi-threading or something. The point was that working code is only "bug free" in the context you tested it in: a change in the runtime context can introduce unforeseen bugs. – CodexArcanum Mar 11 '11 at 16:40
@Kenneth Yeah it sucks and it always makes me sad inside, but when the boss says jump, I jump. But I always ALWAYS give the 'customer perception' speech and eyes of disapproval. – MetaGuru Mar 11 '11 at 20:16
The OP pretty much handed this one to you! – ElGringoGrande Mar 12 '11 at 02:32
+1 Even Toyota sells the cars with bugs, just to enter the market earlier than competitors with new models. AFAIK it is cheaper to sell and then repair, than spend years to finish it and sell w/o bugs – Genius Mar 12 '11 at 12:39
@CodexArcanum: yep, my point exactly. – DisgruntledGoat Mar 13 '11 at 21:12

GrandmasterB · Answer 2 · 2011-03-11T19:34:48.723

33

Its a judgement call. Remember, a bug can be many things. If its a major piece of functionality thats flat out not working, then you fix it first. If its something minor that has minimal or no real impact on the program's usefulnes, you might let it slide.

So look at it from a cost/benefit viewpoint.

You ship products with known bugs when the total cost and risk of fixing the bug outweighs whatever problems and negative impact would arise from the bug being out there.

So if you have a 2 week testing period before you release, and a small bug is found, the question is... is fixing that bug worth the 2 additional weeks a team might now have to spend to re-test the application and installation (an often forgotten about step in creating software)? What are the cost to reputation or in sales if the software is late? Are people going to be angry? They might be quite happy to live with a minor bug if the major functionality can be delivered on time.

Risks include the risk of introducing a new problem, not just in fixing the bug, but also things that might arise from creating a new installation.

Negative impact is both the time and effort dealing with customers reporting the bug, and things like reputation damage.

edited Mar 11 '11 at 19:34

answered Mar 11 '11 at 08:32

GrandmasterB

37,990
7
78
131

30

A typo in the 'about' box is something you really should fix. It will take about 0.7 seconds (assuming we both type at the same speed). However if you leave that typo in, _people can see it_. It's instant death for your users' confidence in your software if there are visible mistakes in the user interface. – Mar 11 '11 at 09:17
This seems about right to me. Most of the time there are a few known minor bugs in a product even when it goes to release. They tend to be things that are very unusual and have a negligable effect on the actual operation and use of the software or things that most users never see. – glenatron Mar 11 '11 at 10:40
3

Indeed, typos in your UI shatter faith in the quality of a product (wrongly, as many great programmers just don't speak good english), however I see your point - trivial bugs that cause no real harm and probably won't ever come up aren't worth the hassle of holding back a release. Leave it for a bullet point in 1.01. – Phoshi Mar 11 '11 at 10:58
Do not let spelling errors into your application. Quite frankly I'm more embarrassed by them then an actual bug. – ChaosPandion Mar 11 '11 at 19:04
1

I have no idea what you are talking aboot... ;) – Andreas Johansson Mar 11 '11 at 19:14
@Graham, ignoring that you are being too literal with my examples, you missed the point of my answer. A typo may take 0.7 seconds to fix. But a fixed typo generally requires a new build (and certainly a new installer build). So now your team goes through the entire pre-release process again, *hopefully not introducing some other bigger problem, costing who knows how much time and money to fix that 0.7 second typo. So you weigh not just the 0.7 seconds for the bug fix, but the ENTIRE cost vs the benefit of having that small bug fixed. Like I said, its a judgement call for each developer. – GrandmasterB Mar 11 '11 at 19:28
@GrandmasterB: apologies for taking a string literal and interpreting it literally as a really bad example, rather than as an approximation to your genius. Thanks for editing the bit I referenced out of your answer, so that I don't do it again. – Mar 12 '11 at 02:12
@Andiaz: And you're not even Canadian! 8-)} (Explaining the joke: the Canadian pronunciation of "about" sounds similar to "aboot" to US-English ears.) – Keith Thompson Oct 29 '11 at 22:23
@Keith Thompson: Haha yeah, you're right. I guess I just watch too many TV series. ;) – Andreas Johansson Oct 31 '11 at 07:42

score 12 · Accepted Answer · answered Mar 11 '11 at 11:32

I assume you are talking of a "known" bug (the question is meaningless otherwise). Well, the answer depends on these factors:

1) Who is the user and how will he/they accept the bug in case it is found?

2) What is the potential impact (damage) of the bug?

3) Is it feasible to delay shipment of the software in order to fix the bug?

4) Most importantly: what do you expect from your software? Time-to-market? Quality? Do you want to see if your customers use the software long enough to find the bug?

score 6 · Answer 4 · answered Mar 11 '11 at 12:00

Bugs come in different severities. At any software companies I've worked at we've categorized bugs in order of Priority from P0 to P4.

P0 Is the software does not work/crashes and could cause damage to the customers business. P1 It's not working as designed and crashes consistently in core functionality P2 It crashes occasionally and or a piece of side functionality may not work. P3 Some element of the software is not working as designed/expected P4 Cosmetic issue.

I've worked in places where P4's just don't get fixed because they have such a small effect on the software.

I'd say it's okay to ship if your software has known P3/P4 issues, I'd put this in the release notes and note that they are being worked on.

I'd never release software with a P0,P1 or P2 issue that I was aware of.

score 5 · Answer 5 · answered Mar 11 '11 at 17:17

5

It's called a "Known Issue". Google, Microsoft, Apple, etc all ship products with bugs, both known and unknown. Try to minimize them, but don't wait for perfection. Ship fast, ship often.

answered Mar 11 '11 at 17:17

Andrew Lewis

533
1
4
12

score 3 · Answer 6 · answered Mar 11 '11 at 08:21

3

You can't ship software without bugs. The advice I can give is that it always better to say to your customer: "This version can't do that and that but we are going to fix this" than the situation when customer tells YOU that you have a bug.

answered Mar 11 '11 at 08:21

Vladimir Ivanov

665
8
13

score 0 · Answer 7 · answered Mar 11 '11 at 11:05

when it's a "feature"! ;)

On a serious note, unless you are a perfect programmer with a perfect test setup, to perfectly test every single code path and eventually that could potentially exist, it's improbable that you will ship a bugless product.

Therefore, be pragmatic, if everything you've encountered in your testing has been fixed, anything extra should be fixed on an as needed basis.

score 0 · Answer 8 · answered Mar 11 '11 at 15:24

As long as you are honest to your customers you can ship with bugs. Telling them about all existing bugs shows them that you have good knowledge about your software, and that is indeed is well tested (if it is..). :)

Obviously the best thing is to avoid shipping with bugs.

score 0 · Answer 9 · answered Mar 11 '11 at 15:26

It is frequently better to have a product ship on time, with a list of known issues, than not ship at all.

One of the things in the programming world that gives people confidence in a project is whether they have release scheduled and whether the schedule holds.

This is why Ubuntu ships releases every half year, even if there are still open issues.

score 0 · Answer 10 · answered Mar 11 '11 at 16:35

I'd say that a good rule of thumb is, "is this bug a showstopper?"

If the bug causes a "happy-path scenario" to fail, then absolutely do not ship with that bug.

If the bug causes a "tangent-to-the-happy-path scenario" to fail and there is no workaround, then do not ship with that bug.

If a bug is documented and there is a known workaround, then it is probably OK to ship with that bug.

score 0 · Answer 11 · answered Mar 11 '11 at 16:55

From a consumers point of view... Never. My point is as long as you know there is a major bug in the software then you should never ship it. However forces of nature (business) override this if the production cycle of the software is now at the stage where it can harm the business model and they are minor bugs that wont: (i) compromise the security of the software (ii) impact usability

score 0 · Answer 12 · answered Mar 11 '11 at 18:15

As people have said, its a very broad question. It actually takes me to an interesting perspective: the so called "bugs" you claim is only the faults that were discovered by your testers. There could be an infinite more loopholes. Just reminded of an interesting story I heard from a respected Professor in one Graduate Seminar: When people in one of the Scandinavian countries used a "handwriting -recognizable" type voting machine, certain people hacked the entire system by writing malicious SQL code (which the system took in as normal input).

score 0 · Answer 13 · answered Mar 12 '11 at 11:37

0

There's something called the FMEA (Failure mode and effects analysis) it's very useful to decide when a known bug is important or not based on:

Occurrence
Severity
Detection

answered Mar 12 '11 at 11:37

score 0 · Answer 14 · answered Mar 13 '11 at 07:40

Another deciding factor can be how the defect relates to your last release. If the bug is part of a new, but broken feature, then the non-functionality doesn't represent a regression of functionality. Go ahead and ship.

On the other hand, if the defect causes a loss existing functionality that is known to be of use to existing customers, then it must block the release. Such a release would be a downgrade for your customers, and would serve neither your interests nor theirs.

There can be shades of grey in this. A regression in core functionalty should never ever go out the door. Some regression in peripheral features could go out to lead users if the release also contains new functionality they have expressed interest in. An obscure defect that is not likely to affect many customers can go into a release, so long as a work around is provided when it does affect those customers. Defects in highly experimental features that are turned off by default anyway should never cause the release to be delayed.

When is it okay to ship a product with a known bug?

14 Answers14