How Much Downtime is Too Much?

I received an email about my datacenter move posts. Jeremy Kraybill asked:

I’m curious if you considered a zero-downtime move at all, where you would keep the “old bloglines” still running while data was transferred to the new bloglines datacenter, and then switch over to the “new bloglines” via DNS after the new site was up? And either users have data loss of several hours (arguably better than downtime of the same amount), or you replicate transaction logs for user-critical data.

That’s an good question. We didn’t consider a zero-downtime move and the reasons why illustrate some of the tradeoffs to consider. One was the engineering effort involved. It would have required a substantial amount of additional work to pull off a zero-downtime (or close to zero) move. It would have also greatly increased the risk of something going wrong. We did enough work ahead of time to reduce the downtime to a 4 hour window, which we believed our users would accept. The benefits of moving to the new datacenter sooner rather than later also factored into our decision.

Those aren’t the only things to think about when considering downtime. Bloglines is a free service and isn’t currently monetized. If Ask Jeeves generated a significant amount of revenue from us, that would have factored into our thinking. But even then, the ’net is littered with examples of sites like eBay. I haven’t checked recently, but eBay, at least in the past, had a policy of regular, scheduled downtimes.

In any event, with scheduled downtime, it’s important to communicate with your users. For us, that meant a post to the Bloglines blog 24 hours in advance. The blog post had specific times listed, along with links to a site that converted the times into all other timezones. We also added a link at the top of every page on the site alerting users to the downtime. During the downtime, we displayed a page that explained exactly when we’d be back on-line. And we updated that page when we went half an hour over our scheduled time. Finally, people appreciate humor. If you’re going to have downtime, make the down page fun. For the datacenter move, for example, we gave the Bloglines plumber a pirate makeover.

I’ll continue my tale of the Bloglines datacenter move tomorrow, with a post that could be titled ‘Rho Rho Rho Your Boat.’