Behind the Scenes of the Bloglines Datacenter Move (Part 1)

One week ago, we moved the Bloglines service from the AT&T datacenter in Redwood City, CA to MCI in Bedford, Massachusetts. This was a challenging and complex undertaking that required months of preparation by many groups. Now that the dust has settled, over the next couple of days I’ll explain some of the process involved.
We had been at AT&T since Bloglines first went on-line in June, 2003, and had been very happy with them. AT&T is a tier 1 colocation facility. They aren’t the cheapest, but we never had to worry about power outages or other issues that can crop up with other facilities. After we were acquiried by Ask Jeeves in February, we started talking about moving the Bloglines service to the main Ask facility, which is in Massachusetts. This made sense for a number of reasons: it would be easier for operations, it would be easier for us to quickly expand in the future, and it would be easier for us to tie into other parts of Ask Jeeves.
Once the decision was made to move, we had two tasks: figure out how many machines to build out in Bedford, and figure out how to do the move with the minimum amount of downtime. In my experience, estimating how much hardware you’ll need at some point in the future can be difficult, especially when you’re growing quickly and you don’t have a lot of history to use in estimating. I believe in the concept of overwhelming firepower (when in doubt, double or triple it), so we overestimated everything. In the end, the new system has 3 times the number of machines that we were running in Redwood City, and each of those machines is probably twice as fast as any of the old boxes. Once operations had the configurations, they set about ordering, installing, and configuring the machines. That left us with having to figure out how to move the site across the country while minimizing downtime.
I’ll continue that part of the story tomorrow.

Advertisements

Markspotting

I’ll be at the Always On conference this Thursday, July 21, on the Open or Closed Web panel, moderated by that shrinking wallflower Marc Canter. I’ll be arguing for a Closed Web. Of course, I am kidding with those last two statements. As I am continually reminded, sarcasm doesn’t translate well to the written page.
On August 8th, I’ll be on the Vox Populi: Understanding the role of consumer-generated content panel of the SES San Jose conference.
Most likely unrelated to the above two events, this Saturday I’ll be trying my hand at flying a 1945 Boeing PT18 Stearman, which is an open-cockpit WWII era bi-plane. While I do not have the leather cap and goggles, I do have a scarf and jacket, so I’ll be at least somewhat dressed for the occasion. I can’t wait!

Google Launches My Google, World Does Not End

On Thursday Google launched their My Yahoo competitor. It’s not really called My Google, but it might as well be. As has been pointed out by many others, this marks a reversal for Google and their ‘no portal’ policy. Honestly, who couldn’t have seen this coming for some time now?
The surprising thing, at least to me, is that they’re trying to copy My Yahoo, which is the wrong thing to do. As many people have found out, the My Yahoo metaphor of a customizable page displaying static information doesn’t scale. It may have worked in the mid-1990s. But in this particular century, with millions of blogs and other sites of interest, you need a different interface paradigm to deal with all that information.
My Yahoogle doesn’t track what information you’ve already read, and what bits are new. So, each time you visit your My Yahoogle page, it takes time to scan the page to see if there’s new information. This is a complete waste. If you only show new things, the amount of information that needs to be displayed decreases greatly. There’s less information, and it’s all new. It’s a much more efficient way of dealing with many information sources.
Another flaw in the My Yahoogle model is the idea of placing everything on one page. Besides forcing the user to become a web page designer (should I place this information source in the right corner, or left?), this again reduces the number of information sources that can be followed, to a number that can be reasonably placed on a single web page.
The Bloglines user interface was developed partially in response to these flaws. Only show new articles. Provide a mechanism (the tree display in the left pane) that allows you to easily select a subset of information sources to display at any one time.
I have over 200 subscriptions in my Bloglines account (many of which you can see in my blogroll on the left). There’s no way I could follow that many sites in My Yahoogle. Sometimes I’m asked if I consider My Yahoogle competition. There’s no way that they can compete without completely changing their interface.

Syndicate Conference and Business Week

I’m back on the West Coast again, at least for a few weeks. I moderated a panel at the Syndicate Conference in NYC this past Wednesday. We arrived Tuesday evening, in time to catch a fun blogger dinner at Gallagher’s Steak House in Times Square. For this particular carnivore, it was like heaven. Large slabs of cow and other mammals all over the place. Very yummy.
Wednesday morning, I had a meeting with Business Week’s Stephen Baker, as described here. Stephen asked some great questions, and I hope I was able to give coherent answers (thanks to jet lag, I was operating on about 2 hours of sleep).
After that, I moderated a panel at Syndicate. The panelists did a great job, and I thank them for making the panel a success. Unfortunately my flight was at 6pm, so I had to scoot right after the panel ended. With a seat in the last row of the plane, and after having the plane sit on the tarmac for an hour due to weather delays at SFO, I finally got back home around midnight.

Weather Forecasts

As part of our ongoing Universal Inbox strategy, we launched weather forecasts in Bloglines this evening. Simply click the ‘Add’ link from your My Feeds display, and then select the ‘Weather’ link. You can add a weather forecast by city/state or zipcode. Forecasts are available from throughout the world, and like all of Bloglines, this new feature is localized in the 8 different languages that Bloglines supports.

Updates

It’s been a busy few weeks; here are some quick highlights:

  • Early this week, we hired Garrett Rooney. Garrett’s super smart and we’re really excited to have him join the team. And most importantly, he’s another Sony PSP player, and we need fresh meat for our Ridge Racer and Wipeout Pure games.
  • On Wednesday, Bloglines announced the ability to track UPS, USPS and FedEx packages. We were working on this when the Askquisition happened, and I’m really happy that we’ve finally been able to release it, as it’s my favorite new feature. Testing this feature required the ordering of many items (ie. toys) from Amazon and other places. Sigh, the things I have to do for my company.
  • On Friday, we announced that Bloglines had been translated into Klingon. Yes, this was for April Fools, but Paul actually did translate the site into the Warrior Tongue. For those keeping score, Paul also did this for ONElist back in 1999.
  • Also on Friday, I made my acting debut in one of the videos promoting the Jeeves9000. I’m in the video titled ‘Integration‘. No, I have no plans on quitting my day job to become a master thespian.
  • Finally, I’ve been having terrible luck with cars lately. Don’t let me drive your car; it’s guaranteed to break down.

We’re Hiring!

Bloglines is looking for help as we continue to innovate and lead the news aggregation space. We’re looking for a Senior Product Manager, several Software Engineers and several Senior Software Engineers. All jobs will be based out of the Los Gatos, Ca office of Ask Jeeves. To apply, you must fill out the on-line application. Other jobs with Ask Jeeves are available here.
The environment is fast paced; we’re looking for self-starters who thrive in an environment where they have a lot of freedom along with a lot of responsibility. We’ve got a lot of exciting things planned; help us continue to innovate!

Thanks!

This past week has been hectic. I’m gratified by the overwhelmingly positive response to Bloglines’ acquisition by Ask Jeeves. As I’ve said, we’ve got great plans ahead, and once the dust settles a bit, we’ll be able to start rolling out new features again. We will start hiring in the next couple of weeks. As a first step, we’ll be hiring a couple of engineers and a product manager to augment our existing team. I’ll post a pointer when that happens
I’d also like to extend thanks to everyone that emailed me congratulations over the past week. I’m sorry if I haven’t responded, I’m suffering email overload at the moment. So if you don’t hear from me, please know that I do appreciate the emails.

Nobody Expects The Bloglines Askquisition!

Yes, the rumors are true and we’re all really excited. Now that that is out of the way, I’ll try to
answer some questions.

  1. I won’t be going anywhere. I’m fully committed to Bloglines, and we’ve got great things in store.
  2. The Bloglines web site will keep on going, business as usual. One of the things we liked
    about Ask Jeeves is their multi-brand strategy, and we’ll be operating as one of their independent brands.
  3. So what will change?
    We’ll have a lot more resources available to us. For example, we’ll be integrating Ask’s killer Teoma search engine technology within Bloglines. This will vastly improve our blog search capabilities. We don’t think that
    world-class blog search exists yet; with Teoma and Bloglines that will happen.

Since we launched in June 2003, we’ve had an organic growth policy, both for the web site and for the company. And that’s worked very well for us. Over the past year and a half, we’ve had many conversations with several great VC firms about funding Bloglines. We’ve also had conversations with many companies about acquiring Bloglines. We’ve been in a very fortunate position where we did not have to take any money, and we turned down all offers. But Ask Jeeves was different than the others that approached us. They wanted us to continue to run Bloglines as a stand-alone property, and also integrate Bloglines into their other properties where it made sense. And they were willing to commit a lot of resources to
Bloglines to help us expand our features and capabilities. Just as important, it was clear from day one that the Ask team understood us, and our service. In fact, many of the execs at Ask Jeeves were already addicted Bloglines users. More subjectively, we thought they had much more of the start-up/fast moving mentality than any of the other companies we talked with, and that approach made them feel like the right partner for us.

Speaking from experience, I know that the acquisition of a service that you use and depend upon can be unnerving. I also know that after acquisition some services wither.I am confident that won’t happen here. There is a shared passion and vision for Bloglines, and I’m very excited about the future. Update: Here’s Jim Lanzone’s take on the acquisition. Also, thanks for all the great emails and comments! I will try to respond to them over the next couple of days.

Aggregator Market Share, User Behavior, and Revenue Models

Richard MacManus has been trying to figure out aggregator market share based on stats from his own blog. It’s really interesting reading, and the comments are very good as well. He puts Bloglines at 50% of the aggregator market, although he implies (and I agree) that hard numbers are difficult to come by.

Seperately, a lot can be gleaned from the live traffic stats that BoingBoing publishes. Kirk Scott ran the numbers, and he came up with the following. BoingBoing is the 3rd most popular feed on Bloglines, with 13,533 subscribers as of this morning. In the “Connect to site from” section of BoingBoing’s stats report, the stats show that 90% of all traffic is directly linked from a bookmark or URL. Of the remaining 10% that is referred to BoingBoing, Bloglines is delivering more than 2.7x the number of hits than Google.

Referrals from search engines:

  Pages Percent Hits Percent
Google 277780 51.5 % 291357 51.4 %
Yahoo 216005 40 % 225746 39.8 %

Referrals from other sites: (Bloglines has 13 entries in this list)

  Pages Percent Hits Percent
display 19662 4.4 % 660776 32%
display 3390 0.7 % 114402 5.5%
topblog 1559 0.3 % 1559 0 %
recs 236 0 % 236 0 %
toplink 231 0 % 231 0 %
search 165 0 % 165 0 %
topblog 159 0 % 210 0 %
related 146 0 % 146 0 %
preview 103 0 % 3697 0.1 %
subs 58 0 % 60 0 %
cites 53 0 % 53 0 %
public 53 0 % 3248 0.1 %
myblog 53 0 % 54 54
  —-   —-  
  25868   784837  

What this tells us is that Google, through search, delivers more page views on BoingBoing than Bloglines, but Bloglines audience share is 2.7x larger than Google’s in the number of overall impressions.

The hit count (images retrieved) show that many more people are reading the content on Bloglines, and relatively few clicking through to the site. Even so, our clickthrough rate is higher than any other site with the exception of Google and Yahoo! search.

Nutshell: Bloglines is the 3rd largest source of referring pageviews on BoingBoing, and the largest generator of referring hits to BoingBoing content, larger than Yahoo! and Google — and all other search engines — combined. Bloglines generates 30% of all referring hits to BoingBoing.

In another post, Richard MacManus points to a blog post by Jupiter Research analyst Eric Peterson based on a conversation Eric and I had last week. Eric was interested in the business model behind Bloglines. Not accepting my usual stock answer of “Volume!”, I detailed that we will integrating highly targetted contextual advertising into Bloglines next year, or “Adwords on Steroids” as Eric puts it (I like that description!). To reiterate what I told Eric, when we do start to roll out advertising, we will be very sensitive to user feedback, and we will be looking to our users to help guide us in this area.

Back to Richard’s commentary on Eric’s post, Richard says “Bloglines currently has only a fraction of the quantity of users that Google has”. True enough. But if you look at the number of hits BoingBoing is seeing (as detailed above), Bloglines has the #1 share of any site. We certainly drive fewer single-visit unique visitors, but based on these numbers, we are the largest source of repeat visitors, which are the most valuable kind of audience to have. These numbers highlight just how sticky an aggregator is; we have incredibly high page view per user numbers and active unique numbers.

It’s fascinating to examine all of these stats, and we’re grateful that BoingBoing provides a live glimpse into the traffic on one of the most popular blogs on the Internet. On our end, Bloglines currently only provides subscriber counts to publishers. What other statistics, demographics or bits of information do publishers want from us about the traffic we’re aggregating?