Persistent Storage for Amazon EC2

Last night, Amazon announced that they’re adding a persistent storage capability to their EC2 service. To review, EC2 provides the ability to create virtual servers on the fly. These servers are a bit ephemeral, however. They can fail at any time and don’t provide any persistent, local storage of their own. If an EC2 instance fails, you have to completely restart it, losing any data it may have been working on. Amazon’s S3 service is persistent storage, but it is not designed to be accessed as local storage by EC2 instances. The newly announced persistent storage capability is designed to solve this issue. It’s like an on-demand S.A.N., but with more flexibility. One of the really nice things about it is the ability to checkpoint a persistent volume to S3. This is great for database backups, among other things. No performance numbers have been published yet, but those who have been using it say the performance is good. This makes Amazon Web Services even more interesting, because it’s now easier to run a normal MySQL instance without having to do something like running some kind of replication just to deal with the non-persistent local storage. And it scales up.
See Werner Vogels’ announcement of the persistent storage service, and RightScale’s analysis of it, for more information.


Laptop Bag Recommendations?

Dear Lazyweb,
I’m in need of a new laptop bag, something on the smallish side. It needs to fit my Macbook Air and its power adapter, my Kindle and its power adapter, my Bose headphones, possibly a small mouse, and a couple of cords. So, not much. Any recommendations? I’d like to see it in person before I buy it, and I’d like to get it this weekend, so that eliminates mail-order places like WaterField.

Database Developments (new post on Startupping)

I just wrote a new post over on Startupping about two items related to databases and Internet services. I talk about SSDs and the launch of Amazon’s new SimpleDB, which I think is a very big deal.

Keyboard of Choice

As someone who has done a considerable amount of typing in his lifetime, I’ve become a bit of a keyboard connoisseur. My first computer was one of the original IBM PCs and one of the great things about that computer was the keyboard. It was this heavy, steel beast of a thing. The keys were well spaced and had the perfect amount of tactile feedback and travel. Maybe it was because I learned to type on that keyboard, but I haven’t found a better keyboard since. Luckily, a company called PC Keyboard purchased the technology/designs of this keyboard and are selling new versions. I just ordered my third ‘Customizer 104/105’ keyboard from PC Keyboard for a new iMac. If you’re unsatisfied with your current ‘board, I recommend checking one of these keyboards out.

AlwaysOn Open Media 100

I’m flattered to be included in the AO/Technorati Open Media 100 in the Toolsmiths category. Thanks to everyone involved!

HP and Sleepycat

Previously I had mentioned how we were having problems with Seagate drives. On the flip side, I’d like to point out two exceptional companies, HP and Sleepycat Software.
In production, we use several HP Procurve switches. These are devices that connect the various machines in the Bloglines cluster. Like some other aspects of our cluster, we’ve bought several of these off eBay, which even several years after the bubble burst, is still a good source for cheap computer hardware. Recently, a newly purchased used HP switch died on us. There were two exceptional things about this. First, the switch continued to operate, but in a reduced way. Specifically, we couldn’t access the management functions of the switch.So even though the switch had ‘lost its brain’, it continued to do the basic functions of keeping the network going. This highlights exceptional design on HP’s part. The second exceptional thing about this event was HP’s support. They overnighted us a replacement switch, no questions asked, no receipt needed. The switches have lifetime warranties, which apparently apply even to secondary owners. Amazing. They made true believers out of us. At least for our networking gear, we’re HP purchasers for life.
The second company I want to mention is Sleepycat Software. Sleepycat makes the database software that powers large parts of Bloglines. Sleepycat perhaps isn’t as well known as MySQL or Postgres, but their software is very fast, bulletproof, and their support is top notch. They deserve more attention. Some people dismiss their database because it doesn’t have a SQL query engine. SQL is fine for ad-hoc queries. But I can guarantee that not a single query that we run on our databases is ad-hoc, by definition. We have a defined set of database APIs, and we want them to run as fast and reliably as possible. So why take the 10x or more performance hit of a SQL engine or risk the bugs inherent in a more complicated system? You get a fully ACID compliant database system, with hot and cold backup capabilities, as well as a full replication system. And it’s open source.
I don’t want to start sounding like a commercial, but I thought both of these companies deserved to be highlighted based on my experiences.

Seagate Has A Problem

At Bloglines, we have 3 classes of machines in our cluster. We’ve got web boxes, which are pretty lightweight. We’ve got storage class machines, which as you can guess have big drives and medium speed processors. And we have database class machines, which have fast processors, fast disk, and lots of ECC memory.
Fast disk, in general, means some form of SCSI. The database machines use Ultra SCSI drives, specifically Seagate Cheetah Ultra320s in a RAID configuration. Unfortunately, we’ve experienced something like a 40% failure rate on these drives. Because of the RAIDs, this hasn’t resulted in any loss of data or downtime, but it’s still extremely unacceptable.
The drives have a 5 year warranty, so we’ve been shipping them back to Seagate. In return, we receive ‘repaired’ drives from Seagate. Recently, one of those repaired drives failed within one minute when installed in a machine. My suspicion is that part of the problem is that Seagate isn’t doing much of a job to fix drives that are sent back for repair.
Speaking of which, when sending a drive back to Seagate for replacement, you can call them up and ask for the ‘advance replacement option’. This means that they send out a ‘new’ drive before they receive your old drive. This speeds up the replacement process. Before today, we were able to get a customer support rep on the phone directly and specify the advance replacement option immediately. But now, apparently Seagate is outsourcing their first-tier customer support, so now when you call them up, they ask for your details and then say someone will be in touch within 24 hours. Which, if calling on a Friday, probably means Monday.
We’ll never purchase Seagate Ultra SCSI drives again. The risk is too high.


So I guess that Google’s new Gmail web-mail service isn’t a hoax after all. Kudos to them for the publicity stunt of announcing on April Fools.
More importantly, it sounds like they’ve got the right idea about storage, giving each user 1 gigabyte of storage. I think this is absolutely the correct thing to do. Economically, it doesn’t cost Google much to provide this (storage approaches free over time, and most people won’t use up that gig, at least not immediately). And it really ties the user to Google’s service. If I’ve got a gigabyte of old email on Google, that’s a very strong incentive to continue to use the service.
It will be interesting to see how Yahoo and MSN/Hotmail respond to this. They’ve both made a business out of charging extra for more than a very small amount of storage.
I’ve said it before, but I’ll say it again, modified slightly. When designing a service, assume hardware is free. Assume processing power and storage are infinite. Because they approach that over time, and limiting them does your service more harm than good. In addition, I think at this point you can also assume that bandwidth is free. That certainly wasn’t the case in the mid-1990s. But there’s now a glut, you can get very good deals on bandwidth these days, and it’s only getting better.


I attended the Future Salon presentation by Dave Sifry on Technorati last night. This was the first time I had met Dave, and he struck me as one of those instantly likeable types. It was a good presentation and I have a lot of respect for what they’re doing at Technorati. Even though there is some functionality overlap between Bloglines and Technorati, which will probably increase over time, I think we serve different audiences. Anyways, Dave had some very nice things to say about Bloglines, and I was quite flattered.

Better To Be Lucky Than Good?

During the recent move of Bloglines from Equinex in San Jose to AT&T, we retired a couple of machines and added several new machines. Yesterday, we were reconfiguring what used to be one of the primary database machines at the old co-lo. While swapping out drives, I noticed that the wire for the speaker was completely melted. Looking more closely, the speaker had shorted against the metal chassis. That couldn’t have been good for things. And now, for whatever reason, that machine is completely flakey and crashes every hour or so. While it was functioning as one of the database machines at the old co-lo, it crashed a grand total of once in about 1 year of heavy operation.
This was one of the machines I originally bought used off eBay. I got it cheap and it worked really well for a year, so no complaints. Ebay is still a good source for cheap gear, although it seems like there’s less good stuff (computer wise) these days than say a year ago.