183

My first commit in my project resulted in the nightly build being broken and people are all over me as we are nearing the release. I want to send an apology email that should sound sincere and at the same time hinting that this was my first commit and this would not be repeated any more.

Being a non-native English speaker, I have difficulties coming up with correct words. Can someone please help?

gnat
  • 21,442
  • 29
  • 112
  • 288
rajachan
  • 733
  • 2
  • 9
  • 9
  • 103
    If the build never broke I'd begin to suspect a broken build process ;) – Lazarus May 25 '11 at 11:54
  • 7
    How were you to know that the build was broken? –  May 25 '11 at 12:01
  • 68
    How about : "I am very sorry for breaking the night build. If you want to withold my salary as a form of punishment, I will not take the company to employment tribunal". Nah, just kidding - do not apologize, this sort of stuff happens all the time. People who are "all over you" are f.king psychos who don't know how to properly restore the system to previous state. – Jas May 25 '11 at 12:44
  • Just make it clear that you have learned your lesson and that in future you will always do a test build before checking in changes. – Paul R May 25 '11 at 12:54
  • 14
    I broke the build on my first 10 commits! Dont worry about it – Nobody May 25 '11 at 13:02
  • 7
    If one breaks the build because of excessive slopiness then an sincere (but not seppu-level) apology might be in order. If not, shit happens. It's really bad when corporations adopt a culture of "check-in fear" where everyone's afraid to make mistakes and the check-in police roam the corridors. I've seen it many times and always lead to so much fear and check-in policies that it's impossible to get *anything* done. Make mistakes quickly, fix them quickly, repeat. – Homde May 25 '11 at 13:07
  • 5
    I try really hard not to break server builds with my commits. But if one day I do and somebody tells me to apologize for it I'll probably just tell them to go to hell. It happens, it's not a tragedy unless the configuration of your build system is particularly moronic – Dyppl May 25 '11 at 13:26
  • 1
    DO NOT APOLOGIZE! you did nothing wrong. A good working team would make light of the situation. The WordPress project [blames Nacin](http://twitter.com/#!/blamenacin) every-time the nightlies get broken. One time he knocked out the dashboard for 10 million users so don't feel bad. – Chris_O May 25 '11 at 14:26
  • Buy me a pizza for lunch. I'll forgive you. – MVCylon May 25 '11 at 14:32
  • 1
    @rmx I broke the build on my very first commit with the company! Now, we each have a "Get out of one broken build FREE" card :) – Travis Christian May 25 '11 at 15:07
  • 1
    @rajachan: in my team, people who "block" the rest of the team (breaking the build, the non-regs, the dev env) are expecting to fix it, perhaps with help if necessary, and to bring croissants on the next morning :) – Matthieu M. May 25 '11 at 15:28
  • 137
    I just wish i had a nightly build to break.. – Max May 25 '11 at 15:53
  • It could be worse, when we deployed my first project, it took out all of our web services (in production). – mikerobi May 25 '11 at 19:26
  • 5
    Tell them you were just testing to make sure the build process worked properly. – Joe Phillips May 25 '11 at 19:37
  • 7
    I broke the build, but I didn't break the deputy – WOPR May 26 '11 at 04:41
  • 1
    Why are they so mad? They must using some ancient source control like CVS with people never tagging or even writing commit message at all (my company). – kizzx2 May 26 '11 at 10:44
  • Nobody ever apologized to me for breaking the build. – Web May 26 '11 at 15:43
  • 1
    CRACK! whizzz... WHAM! ClonK. Cling, Clang, Doink. Wonk. *RiNg rInG*... BAM! BOOM! **KABOOM!** – Mateen Ulhaq May 26 '11 at 23:34
  • I agree you should not have to aplogize. Of course the solution do not commit anything unless it works. – Ramhound May 27 '11 at 15:03
  • Where I work, you have to wear a pink hat until you fix it. Takes the stress away from the error and everyone has a good laugh (even the responsible one) and no one has to apologize verbally. – dukeofgaming Mar 18 '12 at 13:39
  • Entrails. Entrails is always good. – mjfgates May 07 '12 at 00:04
  • "people are all over me as we are nearing the release". Haven't they read the Mythical man month http://en.wikipedia.org/wiki/Brooks%27s_law "Adding people to a late project makes it later.". Your project may not be late yet, but if you are nearing release, I'd think hard about bringing new people on board. – Jaydee May 07 '12 at 14:08
  • 4
    How about, "Sorry, you didn't catch these errors during the code review." – JeffO May 07 '12 at 14:44
  • As the old meme goes, there are exactly two kinds of programmers, those who HAVE broken the build, and those who WILL. Say "OOPS! I'm sorry!" and don't worry about it. – John R. Strohm May 14 '13 at 13:52

23 Answers23

308

Don't apologize!

  • Breaking the build once in a blue moon is not a big deal, and should never be a show-stopper.
  • It's your manager's fault for not configuring continuous, automated builds.

Also, I bet your team fails the 'Joel Test' and can't make a build in one step.

If so, this would be another thing that you shouldn't apologize for. Indeed, it's a team anti-pattern.

takrl
  • 439
  • 1
  • 8
  • 19
Jim G.
  • 8,006
  • 3
  • 35
  • 66
  • 32
    +1 Agreed! Don't apologize. LEARN what you did wrong. Don't do it again, at least not soon. Be polite when people are "all over you". Learn from what they say (even if it's just who's a prick and who's OK). – Peter K. May 25 '11 at 11:53
  • 13
    Well, you can apologize still... But I'm a bit surprised that people are all over you. If you are new on the team, it shouldn't be such a surprise that you made a mistake. At least it shows that you did something :) – Philippe May 25 '11 at 12:17
  • 1
    +1 but for a slightly different reason.. If one of my developers does something stupid I don't want them to apologize, it means nothing, I just don't want them to do it again. Now, I wouldn't consider breaking a build "something stupid", in fact I don't think it's a big deal, but every company is different and if they consider it a big deal then do not apologize. Just don't do it again. – Andreas Bonini May 25 '11 at 14:23
  • 43
    +1. The whole purpose of doing a nightly build is to catch problems as early as possible and before they go sailing out with a release. 95% of developers have made high-visibility mistakes during their careers. The other 5% are lying. – Blrfl May 25 '11 at 14:52
  • 26
    While I agree that this does not call for a "falling on the sword" level apology, I think it does call for an "oops, my bad" level apology. Not every 'I'm sorry' needs to be abject. – Matthew Scouten May 25 '11 at 17:15
  • @Jim: Woah, what a trip down memory lane: [Wasting Money on Cats](http://www.joelonsoftware.com/articles/fog0000000037.html) - I just installed BeeTag yesterday on my phone to do QRs !!! – mplungjan May 26 '11 at 08:18
  • It's not "your manager's fault" that you don't have a continuous build, this is just a lame excuse. Instead of worrying about apologies show initiative, and work with the team to set up one. – Adam Byrtek May 26 '11 at 13:57
  • 5
    I don't agree with this premise at all, there is nothing wrong with a **simple** "I'm sorry" apology, *I realize I screw up and will do my best to learn from my mistake*. I agree the best way to * amend* yourself is not to do it again, but you if you care, you might wan't to openly show it to let others know, not every time you screw up, but he clearly felt bad about it and there is nothing wrong with apologizing. – Trufa May 26 '11 at 14:37
  • @Trufa: In case you missed it, the problem with the OP's company is that it's sufficiently difficult to make a build to warrant an apology. They're failing the 'Joel Test', and the OP should feel no shame for the accident of his/her workplace that is "It's difficult to make a build." – Jim G. May 26 '11 at 16:17
  • 1
    @JimG.: I did not miss it, I'm not saying it is all his fault, or he should be crucified, the company is to blame too, I'm just suggesting, that even though it might not be entirely his fault, or maybe even his fault at all, and apology could mean, Hey! I care, I took note, and won't happen again. Not all companies are perfect, that doesn't mean you shouldn't be considerate. – Trufa May 26 '11 at 16:27
  • 1
    Why do builds break? Because someone merged code badly. But why he had to merge? Because the other guy pushed faster... Why should I say sorry, for just doing my work? – Pascal Mar 03 '17 at 15:04
  • @Pascal: 100% agree. – Jim G. Mar 03 '17 at 17:22
184

Bagels. Donuts. Etc. At one company I worked for in the past, checking in broken code or otherwise causing colleague disruption is generally resolved by the bringing in of apology foodstuffs the next day.

We had a guy blow away a production database one day, causing massive panic and a late night for the whole team. The next day he grilled burgers for lunch.

I love co-worker apologies. Tasty, tasty apologies.

Dan Ray
  • 9,106
  • 3
  • 37
  • 49
  • 84
    +1 for "I love co-worker apologies. Tasty, tasty apologies." – Jas May 25 '11 at 12:40
  • 33
    Breaking the nightly dev build is one thing but blowing away the production database is on a whole other level. If I ever did that I wish that grilling burgers was the worst of my punishment. – maple_shaft May 25 '11 at 12:42
  • 1
    @maple_shaft - I believe he also had a Serious Talk with his manager. I wasn't party to that; I just ate the burger. He was a good guy and a great dev who had a very bad day. Happens to all of us (right, OP??). – Dan Ray May 25 '11 at 12:50
  • 21
    You can almost taste the guilt. – MSpeed May 25 '11 at 13:14
  • 2
    @billynomates: It's even better than the BBQ sauce. – FrustratedWithFormsDesigner May 25 '11 at 14:21
  • 1
    I subscribe 100% to this (we do the same thing :p). It cultivates the team spirit as well, and people are much more amenable to your excuses when their mouth is full of good tasting food :D – Matthieu M. May 25 '11 at 15:30
  • Whether or not this is company policy I should remember this tactic. – jhocking May 25 '11 at 16:01
  • 42
    "Blowing away a production database" puts all kinds of hilarious images in my head as to what exactly happened to the database. Most of them involve everyone hearing a loud explosion from the server room. "Oh god, the database is reaching critical volume!" "Quick! Lower the control rods!" "It's too late, the data, she is doomed!" **BOOM** – Carson Myers May 25 '11 at 20:57
  • 3
    @Carson - I *wish* it had been that dramatic. The lesson he learned (which he already knew, actually) was to double-check your connection before issuing `drop database` commands. – Dan Ray May 25 '11 at 22:24
  • @Dan as in, make sure you're connected to exactly the right server? Or something else? – Carson Myers May 25 '11 at 23:10
  • @DanRay Do what everyone else does, and blame the backup guys! ;) – Tom Morgan May 26 '11 at 08:21
  • 1
    "The next day he grilled burgers for lunch." made me fell of my chair. It sounds like it was the only thing he could do, and he went straight doing it. – Eugenio Depalo May 26 '11 at 10:01
  • 1
    @Carson - Yep. He thought he was cleaning up a sandbox server, when he was actually cleaning up a production server. Needless to say, various policies got reviewed after that. @Eugenio - Funny! I didn't mean that. He's actually a super good and smart guy who had a costly moment of carelessness. – Dan Ray May 26 '11 at 12:06
  • 9
    `drop database` ... Oh the power of those two little words. It was years ago, but I remember it well ;) – Leigh May 26 '11 at 18:10
  • 2
    On some teams, if people brought in food every time they broke the build or checked in buggy code or something, everyone on the team would weigh 400 pounds. – Graeme Perrow Sep 22 '11 at 17:15
81

Two quotes for you:

The man who makes no mistakes does not usually make anything.--William Connor Magee

Anyone who doesn't make mistakes isn't trying hard enough.--Wess Roberts

I agree with Jim G., don't apologise but do learn from it and don't make the same mistake again... keep it DRY ;)

Lazarus
  • 381
  • 2
  • 6
  • 9
    Or there is the more general quote of: "let he who is without sin cast the first stone" If who ever is moaning about the broken build has broken the build before, they shouldn't be moaning – Richard May 25 '11 at 16:46
  • like the second one!! – Sufendy May 26 '11 at 02:24
  • 2
    Quoting myself to every Grad/junior I've ever mentored, *`"I expect you to make lots of mistakes. Own up to them, accept them, and learn from them. If you never make mistakes, you'll never really learn anything"`*. – S.Robins May 06 '12 at 23:37
  • Nice use of the "DRY" principle... :) – J. Allan Oct 21 '16 at 19:20
54

Don't apologize, just FIX IT as soon as possible. It is okay though, everybody breaks the build at least once, in my last company it was something of an initiation ritual. When a team member broke the build we would put a rubber duckie on his desk in the morning before he came in, this let him know he broke the build and he would fix it.

We called it the Continuous Integration Duckie and when you had it for the day people would tease you but it was all in fun, none of it was supposed to be mean spirited.

We took something like a broken build and turned it into a team building exercise.

maple_shaft
  • 26,401
  • 11
  • 57
  • 131
  • 2
    +1: not just what they care about - all they *should* care about. You haven't done anything wrong, as such - just flexed the boundaries of what's possible a bit too far ;). You're probably also the person best placed to fix it as well. Everyone does this every now and again. Everyone uploads something to live that shouldn't have gone. Everyone leaves the WHERE clause out of an UPDATE statement once in their lives. It's how you react that's important. Where we are, all the devs tend to close up, refusing to speak up about who individually was at fault if anyone asks, it just gets fixed. – Tom Morgan May 26 '11 at 08:17
  • That seems like a **really** great way to handle mistakes. – Trufa May 26 '11 at 14:39
  • 3
    **Continuous Integration Duckie**, Hahahaha – AttackingHobo May 26 '11 at 22:47
44

"Sorry! My bad!" is how I usually apologize when I've broken the build. It happens. But as others have said, this is an opportunity to improve your systems so that one person cannot so easily break the build for everyone else.

I would not make a formal apology in these circumstances, but if you actually feel that a more formal apology is appropriate, then your apology should do these things:

  • Express regret.
  • State the problem.
  • Take responsibility.
  • Make amends.
  • Save face.

That is, "I'm sorry [EXPRESS REGRET] that I inconvenienced you [TAKE RESPONSIBILITY] by accidentally [SAVE FACE] breaking the build [STATE THE PROBLEM]. Doughnuts are on me tomorrow. [MAKE AMENDS]"

Each part is necessary in a proper apology; if you don't state the problem then it is unclear. If you don't express regret, take responsibility, and make amends, then people feel like you're insincere. The face-saving part is the most overlooked part of an apology; the face-saving part is what reminds the injured party that you are a valuable coworker who sometimes makes mistakes, and not an idiot (or a saboteur!)

Finally, some thoughts on build breaking:

I work on the C#/Visual Basic compiler team. Of course today Visual Studio is now such a massive project that it has a team of its own just to manage build infrastructure, and a huge room with its own dedicated air conditioning system. Back in the mid 1990s when I started as an intern the Visual Basic build team was one intern -- me -- and a closet full of machines. Times have changed!

Back in those days before continuous integration and strong checkin processes, teams would have a variety of penalties for breaking the build. On some teams the penalty was that if you broke the build, you had to wear a funny hat every day at work until someone else broke the build. On some teams, if you were wearing that hat you were responsible for verifying that the nightly build was correct.

That last bit sounds maybe cruel, but it actually served a valuable purpose. Since almost everyone broke the build at one time or another, eventually the entire team would learn the processes for verifying the nightly build.

Eric Lippert
  • 45,799
  • 22
  • 87
  • 126
16

Don't apologize. It's your coworkers who are to blame for not reviewing your first commit and not having a quick feedback system for builds like a continuous integration server.

At my current job we have an informal rule that someone who sneakily commits just before leaving work and turns out to break the build has to bring candies/cakes/drinks for the whole team the next day. But we do have continuous integration that warns us of a broken build during the day. And the rule probably wouldn't apply to someone's first commit.

Anyway, a formal mail of apology is probably a bit too much.

guillaume31
  • 8,358
  • 22
  • 33
  • Excellent point - peer review should have caught this. Because you _do_ peer review changes before they're applied to master/HEAD/default, right? – Frank Shearar May 27 '11 at 20:11
11

A fundamental rule - when you're wrong, ADMIT IT :-| You don't have to grovel in apologies. Everyone makes mistakes. The pros admit it. That's teamwork. The other team members should be pulling together to help you over it. If they don't, ASK for help. The most that has to be said afterward is - what can we learn from it?

They say a successful marriage is based on three little words - "I was wrong".

If someone doesn't make an occasional mistake, they're not working, but a mistake one doesn't learn from is two mistakes.

Mike Dunlavey
  • 12,815
  • 2
  • 35
  • 58
  • I think this is a great answer and much better than the "Don't apologise" one that has the highest votes. You can apologise to everyone for breaking the build, but they also need to show you a bit of grace. They should also be saying "don't worry, we've all broken it before, after all, that's what it's there for". As long as you try not to break it again, they should be cool with that. If you ignore everyone and go and do the same mistake then that's a different story. – Rocklan May 07 '12 at 02:05
6

The best apology is fixing the break quickly

JaredPar
  • 997
  • 6
  • 12
5

If your company already has a way to test your build changes, then (A) your changes either failed (but you checked them in anyway) or (B) they succeeded (and you need to build a new test case).

If your colleagues gingerly test their changes and expect to find breaks on the nightly build, then (C) you have a brittle process (and you need to introduce testing like that found in Extreme Programming).

It is possible that (D) Your changes caused unforeseen changes in Bill's code that were either in place before or changed on the same build as yours.

Depending on how you and your company test, I would apologize on a case-by-case basis:

  • (A) I failed the build test but checked in my changes. Sorry, I am changing my process so I won't do it again.
  • (B) I passed the build test and added JUnit test XYZtest.java to reduce the chance of re-occurrence.
  • (C) Since we don't have a build test process, I am creating one for my changes. I'd like to share with you how we can improve our build process.
  • (D) I will work with Bill to write a JUnit test XYZtest.java to reduce the chance of re-occurrence.

I'm sure there's an (E) that I haven't thought of.

Notice that I am saying "to reduce the chance of re-occurrence" rather than "so it won't happen again." It will happen again. But you may improve your process to reduce that possibility. That, I think, is one of the marks of a winning programmer.

rajah9
  • 231
  • 1
  • 3
2

Don't apologize. You are human and you will make mistakes. Everyone will break the build occasionally, the key is to just fix it quickly.

As for the people jumping all over you...well I'm curious as to if they've ever written in a bug.

Corv1nus
  • 933
  • 1
  • 8
  • 13
2

How to approach this depends on the atmosphere in your group. If it's a blame culture, I'd be very careful about apologising and how you do it. If it's a collaborative, positive atmosphere, then yeah, something along the lines of "I messed up, I'm sorry. How can we avoid this in the future?" is probably a good idea.

In any case, a goof up like this should be accompanied by some sort of post-mortem to a) find out how it happened and b) how to minimise chances of it happening again.

I'm not familiar with your structure (I work in a very different environment) but ultimately, the reality is that occasionally, people make mistakes, and things get broken. You learn from the experience and move on.

temptar
  • 2,756
  • 2
  • 23
  • 21
2

In my environment if you broke the build you would get a good natured ribbing and some witty cometary. I'm not sure which English speaking country your in, but it could be that as a non native English speaker you are not getting the underlining nature of the comments.

Being from the other side as a Senior developer I once commented on a code review that some way of doing x "sucked" not because the code was bad but do to the structure of the project. It was more commentary to myself that I need to fix the structural problem. Only later did I discover that the Jr Dev though I was very angry at him due to my inaccurate flippant speech.

rerun
  • 2,045
  • 12
  • 24
2

Um. You get the broken build token. Pass the rabies on to the next lucky recipient. Happens all the time. Quality is important, but mistakes are inevitable. Fix them. Move on. Shame the next poor bloke.

Glorfindel
  • 3,137
  • 6
  • 25
  • 33
M. Tibbits
  • 131
  • 5
  • 1
    I've seen this a lot. The other is you get to "look after" the nightly build until someone else breaks it. This doesn't mean fixing the it, but figuring out why and who should fix it. – Stephen Darlington May 26 '11 at 08:48
1

The TEAM failed.

Your company needs more code review on a dev doing a first time build.

I just don't see how a team allows this to go on without them doing a review and offering some assurances along the way that you are doing things correctly.

Being close to release time is not an excuse, but a better reason to double-check new code.

If your release cannot be easily undone, there are even bigger problems with this group.

JeffO
  • 36,816
  • 2
  • 57
  • 124
1

As a general rule, I would say:

If your checkin causes a compiler error you would have caught yourself had you done a "get latest" before checking in, a simple "whoops, my bad" is in order. (especially if you stroll in at 10 the next morning, while everyone is deciding which changeset to roll back to)

If your checkin causes general unexpected behavior, even runtime errors, I don't think it should be held against you. It comes with the territory. As long as everyone's "get latest" generally gets passed their compilers, people really should not be throwing a fit (with a couple exceptions like deleting a database, deleting the server copy of the project and all the changesets, or anything else that is so dumb that people have to assume malicious intent).

Morgan Herlocker
  • 12,722
  • 8
  • 47
  • 78
0

Better a late answer than never...

As many others have said, don't apologize for breaking the build. Simply admit that it was you and get on with the job. This stuff would happen whether you were there or not, and nobody deserves to be treated poorly because of it. People react badly under pressure, so if you are able to remain calm and simply get on with the job, you'll stand out when it matters. My advice for when people give you a hard time is to simply avoid being defensive, defuse the situation by letting people know you are either on to the problem or you are quick to seek advice if you find you've become stuck.

Personally, I see a broken build as an opportunity.

  • If the build breaks and you can prove it's your fault, then it's likely to show that the continuous integration and build processes are doing their jobs. This is a good thing, as it validates your processes. If the build never breaks, I'd worry that there was something wrong with it.
  • If you've broken the build in a fairly common manner, and it happens to break in this way often, then perhaps something is missing from the CI process. This is an opportunity to improve your processes so that the common failure scenarios can be better managed.
  • If you've gone beyond the call of duty and really messed up the build with an obscure problem, then you have an opportunity to learn about something new that might be important enough to trigger a warning to the rest of the team.

So what I'm saying is that breaking the build occasionally means that people are doing their jobs. Sure you might have made a mistake, but if you use it as an opportunity to learn, you are doing your job simply by learning to do it better next time.

S.Robins
  • 11,385
  • 2
  • 36
  • 52
0

I don't understand why people are all over you. If the system is setup well, they should be able to tease out your changes / fix them very quickly.

But again, if you are one of those guys who broke windows build, Then ... I can't help. (It is incredibly hard to do btw - before anyone questions MS build philosophies, but now and then someone does it - and the whole company QA stops for a day).

And yeah - don't apologize. Just have a chat with your manager and make him understand that you learnt from what you have done.

0

Builds break all the time. Bugs and mistakes are a fact of life, and that's why you should have a process that minimizes effects of bugs and mistakes.

If a build breaking is that big a deal, it means your process is broken.

You should be doing continuous builds, not nightly builds.

  • +1 for "if a build breaking is that big a deal, it means your process is broken." On the other hand, you still have to deal with the broken process, and that means doing what you can to avoid breaking the nightly build until the process is improved. – Caleb May 07 '12 at 00:20
-1

Tell them they need CI builds per check-in. That way you wouldn't have to wait until the nightly to know it's broken. YEA!!! Tell them it is the process that is wrong, nothing else. You just identified a gap in their system.

But yea, make sure to fix it. Not a big deal. It is not in production. Just a worthless nightly.

Issa Fram
  • 101
  • 2
-2
  • I would not apologise.

  • I would blame CI lead for allowing me commit broken build.

  • There should be a CI process in place for stopping developers from committing broken code.

  • If local build fails, then it should not be allowed to enter build server.

CodeART
  • 3,992
  • 1
  • 20
  • 23
  • 1) Not all projects are set up to do continuous integration. It's nice to have, and can help with situations like the OP's, but it's far from a given. 2) It never hurts to apologize even when it's not really a big deal, even when there are tools that could have prevented the problem. 3) Blaming *someone else* for a problem that you caused, whether or not it's really your fault, is a lousy idea. – Caleb May 07 '12 at 00:16
  • There are two problems here: 1 - broken code on local machine. 2 - broken code on a build server. First problem is very minor as all developers make mistakes. Changes can be undone and there will be no impact on the system or the department. Second problem is likely to have a much greater negative impact on the system and the department. – CodeART May 07 '12 at 00:40
-2

While I think some kind of apology can be in place, please don't say: "I will make sure this doesn't happen again." Because it will. And if it does it will bite you.

Pieter B
  • 12,867
  • 1
  • 40
  • 65
-2

If you are working on an open source project,

just say "Sorry, I broke the build. May be I was too sleepy!"

and add it as a github comment.

It is because many open source developers write code during midnight.

linquize
  • 1,150
  • 1
  • 11
  • 20
-2

A usual practice in my company is:

  • Shouting "it wasn't me!" (usually CVS / SVN proofs otherwise)
  • Wearing a t-shirt saying "my-userlogin ist Schuld!" (yes, 50% of our developer's own such a shirt)
  • Claiming that it works in the local sendbox and all tests passed just fine

My company also has a nice way of handling such an "incident":

Robert Heine
  • 101
  • 2