Every system we ever use will fail. How we respond to the failure is a test of character and preparedness. This morning all of 37 signals products were out of action because of a problem with a load balancer at their hosting service. Basecamp, Backpack, TadaList, Campfire, Highrise . . . all down.Their reaction - communicate and fix.

37signals System Status via kwout

While the problem persisted the urls of the broken applications were pointed to a system status page. Once a fix was in place the product blog was updated with an apology and an explanation.

What happened this morning? - (37signals) via kwout

Given that Murphy’s Law was operating they handled the situation well; communicated clearly, found the problem, fixed it, and did so promptly. It underlines the need to plan for failure and recover from disasters.

Basecamp help: General via kwout

It is possible to export XML backups from Basecamp but I’m sure that many this morning felt a sinking feeling for a few moments, while some of those with XML files would have been wondering what to do with them.

37signals Product Blog: Exporting data from Basecamp via kwout

Web applications are tremendously convenient, with integration and sharing features that we’d never have dreamed of a few years ago. But still we need to plan for the worst.

Murphy’s Laws - End of the World Production, LLC. via kwout