Also See Parts One, Two, Three, Four. The move itself went almost perfectly. At 2pm, we took the crawlers off-line and started copying many of the databases. At 4pm, we took the entire site down and started copying the remaining databases. Around 5pm, wandering around barefoot, I broke my toe, but that didn’t affect things (other than my toe). Around 7:30pm, it became clear that we would require an extra half an hour to complete things, so we updated the plumber page with the new estimate.

With two weeks to go before the move, we started having daily status meetings with all the people involved with the move: people from site ops, net ops and the entire Bloglines team. These only ran about 10-15 minutes each, but were invaluable in getting issues taken care of quickly. We were still working through issues with blog article migration, but we thought we could still make the December 16th date.

As it happened, the new datacenter was built out before the custom blog article replication code was completed and tested. This was ok, because we wanted to stress test the new datacenter machines. After configuring the new machines, we started running some test crawls against an older version of our feed database. To differentiate this test crawler from the Redwood City production crawlers, we changed the User Agent. Many people noticed a crawler with the User Agent “Bloglines/3.

I received an email about my datacenter move posts. Jeremy Kraybill asked: I’m curious if you considered a zero-downtime move at all, where you would keep the “old bloglines” still running while data was transferred to the new bloglines datacenter, and then switch over to the “new bloglines” via DNS after the new site was up? And either users have data loss of several hours (arguably better than downtime of the same amount), or you replicate transaction logs for user-critical data.

The simplest (and safest) way to move a site is to take it completely down, copy all the data to the new machines, and then bring the site back up at the new datacenter. We could have done that, but the length of downtime required would have numbered in the days, and we didn’t want to do that. Actually, an even simpler way to move a site is to physically take the machines and move them to the new datacenter.