It’s smarter to learn from the failure of others than from your own. German telecom giant Deutsche Telekom was kind enough to teach us a lesson this weekend how NOT to prepare for a crisis. Early Saturday morning, a “database error” caused a major blackout of their mobile voice and data services, leaving millions of customers in Germany and abroad with no signal for hours, some for the entire day. What appeared to be a worst case scenario already for the cell phone carrier, was topped by a gaping hole in its notification plan. According to media reports, that is.
In its alarm plan, designed to notify the experts who could fix the IT problem, the Bonn-based tech firm relied on an infrastructure which became the subject of a critical situation itself: their own cell phones. Those alarm phones were not working either. And it’s hard to respond to a crisis one is not aware of.
“The company dealt with the problem, that its own notification cell phones were connected to the T-Mobile network, and thus were not reachable,” reported the Bavarian public radio station Bayerischer Rundfunk.
My two cents on this: You can as well keep the spare keys for your car in the glove box, in case you lock yourself out. It is hard for me to believe that the alarm plan of a world-leading corporation like Deutsche Telekom with some 40 million customers has a flaw like this. It is an imperative rule of crisis prevention to activate your resources as fast as possible and based on a robust infrastructure. Activation is the first chain link in a series of actions. At Telekom, it was also the weakest.“You can as well keep the spare keys of your car in the glove box, in case you get locked out.”
At least, the press office was apparently better prepared. Telekom spokesman Philipp Schindera used a backup phone and made calls from an Austrian phone number. Whether or not this was a Vodafone Austria account, we don’t know. I would have called to ask him, but my cell phone was not working, either. I’m on a Telekom contract.
So what is the takeaway from this real and methodical blackout? The notification should work. Period. No matter what. It’s apparently considered a minor detail, a given, which can be easily overseen, but it’s absolutely crucial to enable the organization to respond and activate its crises processes in the first place. The more affinity to risk, the more important becomes the redundancy.
This is why, as another example, aircraft have two altimeters and two flight sticks (or levers). A concept which has proven very helpful from time to time. Altogether, there is a lot to learn from aviation in terms of crisis management and prevention. But that’s a whole new chapter which I will tackle in another article, soon.
P.S.: A few days ago, I bitched with a colleague about those experts who always offer their advice and knew it all before, once an accident has happened. To make sure I’m not getting confused with these experts, let me fully disclose that everything I wrote about Deutsche Telekom’s failure is based on various but concurring media reports. Whether they are true or not (or only half): The topic is too important to let this opportunity to raise the subject pass. This is not about putting somebody on the spot, it’s all about the learnings.