Does your disaster recovery plan work? You never really know until you test it. You’ll be surprised at the things you find out when testing your disaster recovery plan. It really doesn’t matter what the plan is, or what tools you are using to implement your plan. The point is, until you actually experience a simulated failure you have no assurance that everything you need can be brought back.
Don’t be sucked into false confidence using something like Always on technology. When your data is hacked, then all of your always on devices are corrupted at the same time. You need to have a complete disaster strategy, not just a strategy for hardware. That strategy is not complete until you have proven it works.
Personally, I have participated in three disaster recovery simulations for three different companies. For every one of those simulations, there were key aspects of our systems that were missing. We were confident of success. And we did have a great degree of success in the simulation. Still, there were key systems we were not able to bring online in the available time frame due to inadequate documentation, missing code, or missing data. For missing data we didn’t find that out until we started using the restored system during the simulation.
There were some great comments left from yesterday’s editorial on catastrophic systems recovery. I’d like to pull out the comment from AZ Jim because it speaks to the fact that this sort of thing really does happen. He writes:
Ben, it is a vital topic to not only IT (DR), but to the entire business (BC). When that fateful day struck on September 11, 2001, I was working for a financial services company that lost people in the towers, as well as the corporate headquarters half a block away where all the windows up to the 34th floor were blown inwards with dust and debris covering everything three inches thick. I and my IT colleagues were back in our home states trying to get our mind around what happened when we find out that a lot of our business partners in that building had contracted with outside IT resources to build systems in their offices. On top of the obvious loss, we were told that they never thought anything like this would happen. They urgently needed us to restore databases from old backups that (fortunately) were offsite. We were in a crisis mode for a month fixing something that should not have needed fixing. The real tragedy was that there should have been more awareness by our business. Seven and one half years earlier in the bombing of the World Trade Center, some businesses didn’t survive because their “IT” was in offices the police wouldn’t let them in to get even a backup device.
Whether your DR/BC plan is an offsite tape or whether it is a multi-site, multi-regional synchronous mirrored center, test your plan. Even if it is in the cloud, plan for a test. Murphy is alive and well for those who think that sort of thing is passé.
In short, make a plan, test your plan, perfect your plan, maintain your plan.
Cheers,
Ben