Drawing out a document with strategies for coping with "disasters"

Question

Does anyone happen to know of some guidelines for writing up a document for coping with possible disaster scenarios ? By disaster I mean anything that could lead to loss of work, time or eventually a client.

I seem to remember reading an article along these lines but, unfortunately, I can't find it anymore. So I'd appreciate some feedback in this direction.

I think such an approach would be useful in the company I'm in right now as we've had the occasional scare which might have been avoided with some preventative thinking.

Answer 1

Be forewarned if you want to do disaster recover plans the right way they'll be more to it then you might imagine.

Put a budget to the making of that plan, both in man hours and in resources cost. Like are you planning to spend eight man hours writing your plan or 500 man hours. You might say '500 man hours!, no way!.' Powwowing with 10 employees to start and then talking about disaster at meetings add to man hours fast. If you are midsize or large company, then when managers take time to get info ready for you for the plan that you requested then that eats from the time budget also. All just for the writting a disaster recovery plan.

Some things to do for disaster; make an inventory and cost replacement of all your assets you need to make your clients happy. On items with warranties, guarantees, support contracts, licensing, make sure you have all manufacture, distributor, vendor contact info and current status of accounts with them. Think if a server drowns will you be able to replace that 10,000 dollar 10 year old server in a day? Will the vendor be happy to hear you want a replacement item right away (or is your account outstanding). You might say, oh I'll by a NEW machine off the shelf! Oh? will your old software WORK on that machine. Will a custom software app that needs a hardware key allow a new machine to run the custom app?

Have building evacuation plans, usually with walkie talkies assigned to personnel and their backup personnel.

For certain company types, for proper disaster recovery, make sure you've spoken with banks for emergency financing.

Check your medical plan support; ie 50% of staff is out from sickness cause by issue at building, like mold or dangerous fumes. Once I ran across 11% of staff out from getting very sick from DUCK POO (which had a nasty virus).

Have backup site for staff, have travel costs figured and budgeted to have staff go to backup site. Once one company planned their backup site to be just over one hour south of main site. WOW those mileage costs for employees (and sometimes travel time (hourly wage) costs for some staff) can add up in one week! Union people might have contracts for time traveled to offsite. Plan for people just not being able to travel to offsite and account for temps costs to replace them until main site is backup. You might ask "Who would do that?" Well pregnant employees for one as they may need to be near their doctor or hospital.

One odd disaster recovery thing is legal stuff. Like if your board of trustees gets charged for illegal stuff (like having a bunch of computers with illegally installed operating systems). When some company associates are unavailable for what ever reason certain authorities (read as signatures if you like) may become unaccessible.

The way you sell a disaster recovery plan to "those that want it" are:

First and foremost; Staff and client health protection (from disaster), such as evacuation procedures, call-chains/trees (ie who calls who, who is authorized to tell whom what), health issues of people

Second, financial and cash asset protections (ie info like access to your copyrights, patents, contracts (info), client account balances, lines of credit with financiers) without which recover from disaster is really hampered from lack of cash flow.

Certain "Information" assets fall between financial and asset categories.

Details on asset recovery of buildings, electronic equipment, paper assets like all those filing cabinets. (I bet a lot of companies decide to go paperless when they start on a disaster recovery plan)

There's more. But this post is long enough.

Answer 2

The method I'm most familiar with is Failure Mode and Effects Analysis. It can be generalized to any system.

The point of it is to enumerate all the specific parts that can fail, how they can fail, and in each case what the result would be. Then classify each item based on likelihood, severity, cost, etc. Then you go ahead and create mitigation strategies.

Answer 3

One of the things that we did when moving into a new building for our data center was to come up with a disaster recovery plan, and it was as simple as getting everyone together and just "what iffing", and then coming up with solutions for each event.

Things like "What if Larry gets hit by a car coming to work", "What if the primary and slave DB server for the local callouts die at the same time", etc.

Or even "What if the Qwest guy comes in and rewires 9 of our punchdown blocks because he thinks they look messy in the middle of a call cycle?" (Not that that has ever happened...)

Answer 4

What you're looking for is material about risk management. The general idea of risk management is to first identify the risk, determine what aspects of the project will be affected should it materialize, assess the likelihood of the risk materializing and what impact it would have to different aspects of the project, and identify methods to detect when the risk is beginning to materialize and what action(s) should be taken to counteract the risks.

The methods for dealing with risks is generally captured in a risk management plan. Risk matrices and risk registers capture information pertenent to risk management.

Managing risks is generally considered a project management topic. Hopefully the links to those particular Wikipedia articles might help provide a high level overview, and some links to additional resources. You might also be interested in the Project Management Stack Exchange, especially the questions tagged risk management and risk.