The Groups.io Engineering Philosophy

A thundering herd of wildebeest, Serengeti, Tanzania

But what a fool believes he sees
No wise man has the power to reason away
What seems to be
Is always better than nothing
And nothing at all keeps sending him…
What A Fool Believes, Doobie Brothers

Ask my kids and they will tell you, I am the party of no. No dessert before you eat your vegetables. No skipping out on 3rd grade to day trade stonks. No feeding your gremlin after midnight. I apply this principle to Groups.io devops as well.

One thing to note before we begin is that Groups.io is a small company with steady, predictable growth. For reference, we have about 60 Linodes of various sizes in our production cluster. Traffic is of the order of 10s of millions of page views and emails a day. Some of the following is not applicable or appropriate for fast growing or large services (but I think it is appropriate for most services).

KISS Is Not Just A Band Although KISS Is Also A Band

My guiding principle is:

Eschew recrementitious convolution

That means I have a thesaurus. It also means, keep things as simple as possible. The fewer technologies we use, the fewer things there are to go wrong.

When I’m getting paged at 3am because something is wrong with the site, I am not going to be at my sharpest and I’m going to be under a lot of pressure to get things fixed quickly. The simpler the system is, the easier it is going to be for me to figure out what’s wrong, and to fix it.

For us, this policy manifests in several choices for our tech stack. And that starts with the programming language.

Compile The World And Copy It Everywhere

Everything is written in Go. I am on record as being a huge Go fanboy. I will not extoll all the virtues of Go in this post, but I want to point out three important aspects of the language. First, it is not complex. There is no magic. This is important when trying to figure out what a piece of code you wrote in the last decade is doing. The second important aspect is that it is a compiled language. I believe in compiling all the things. The third important aspect is that it produces binaries that are (mostly) statically linked. Which means that it’s super easy to just copy them around as needed, without needing any support scaffolding.

In Which Our Villain Says No To A Bunch Of Useful Tech

Groups.io does not use containers. And that means no Kubernetes. The way we do releases is that we create a tarball of new executables, copy that to all the machines, untar them into a new directory, and then move symlinks to point to them. After that is completed, we restart the services using the new executables. Our production instances typically have uptimes in the hundreds of days.

We don’t do any autoscaling. I realize that means that we’re not fully utilizing our paid-for production capacity at all times, and I’m ok with that. I’ll take that tradeoff, for less complexity. We have a utility that uses the Linode API to spin up new machine instances and configure them when we need them. But because we have predictable growth, we have plenty of warning and can add machines at our leisure.

We don’t use DNS to access our machines internally. We use an SSH config file with ProxyCommands for each machine. Again, this is to keep things simple.

We have a monorepo, hosted by Github. We do not do automatic deployments on commit. We deploy all the time (including Friday afternoons!), it’s just the idea of automatic deployments makes me twitchy.

Structured Logs And JSON Configuration Files

We keep all our log files on the machines they were produced on. The way that Linode prices instances, we end up having a lot of extra, unused disk space on each instance, so we’re able to keep copious logs (see also above where we don’t autoscale). We have a program, unimaginatively called research, that will grep through appropriate log files on each machine, when we need to debug or trace an event. This works really well for us.

Configuration data is kept in JSON files, and is copied to each machine. We do not have a centralized configuration database.

Distributed Messaging Is Great

I don’t always say no. One core piece of our tech stack is NSQ, a distributed messaging system. It has been rock solid for us. Our large but not-exactly-a-monolith web server will off-load tasks that can be done in the background or that take a long time, to various microservices via NSQ. Some of these include uploading files to Amazon S3, generating group or user exports, sending notifications and webhooks, and sending emails.

One thing I wish NSQ had was a request response system. When interacting with some of our microservices, we need a response. For those services, we currently use HTTP JSON RPC calls. It would be interesting if instead we could just use NSQ in a way where we could receive a response (currently NSQ is basically fire-and-forget). That way, we could take advantage of NSQ’s distributed nature, which would make managing those microservices easier. I believe NATS may have something like this, although I haven’t investigated it.


Groups.io is the best tool to get a bunch of people organized and sharing knowledge. Start a free trial group today.

Why We’re Not Leaving The Cloud

Mars Camp, Wadi Rum, Jordan

You’re A Rich Girl, And You’ve Gone Too Far
‘Cause You Know It Don’t Matter Anyway
You Can Rely On The Old Man’s Money
You Can Rely On The Old Man’s Money
Rich Girl, Hall & Oates

David Heinemeier Hansson recently posted about Basecamp’s decision to leave Amazon AWS and move to self-hosting their own servers, Why We’re Leaving The Cloud. It’s a good post and you should check it out. Basecamp is spending over half a million dollars a year just on database and search hosting.

Renting computers is (mostly) a bad deal for medium-sized companies like ours with stable growth. The savings promised in reduced complexity never materialized. So we’re making our plans to leave.

David Heinemeier Hansson

David frames the question of hosting as a choice between two extremes. One the one hand, you buy and run your own servers. On the other hand, you use a full service cloud provider, like AWS, including their managed services, like RDS. But that’s a false dichotomy. There are plenty of options in the middle, both in terms of providers and in terms of how you utilize them. By going this middle path, you can get many of the advantages of the Cloud without incurring the added expense.

Not Just The Big Guys

When people think of the cloud, Amazon AWS immediately comes to mind. There’s also Microsoft’s Azure and Google Cloud. But there are other options as well, such as Rackspace, Digital Ocean, and Linode. At Groups.io, we have used Linode for over 8 years to great success. The reason that we use Linode instead of AWS is price. They cost less than Amazon or Microsoft or Google.

Typically these companies don’t offer as many services as the big guys. Linode is just now starting to offer a managed database service, for example. But that’s fine for us because we run our own database. And that brings up my second point. By managing our own database, we again save money. Basecamp is currently using Amazon’s expensive managed database offering. When they switch to their own servers, they will need to manage their own database anyways. So there’s no advantage to moving to their own servers over how we’re using the cloud.

But even with managed services, we come out ahead with Linode. We currently use Amazon S3 for a lot of our storage. But Linode recently started offering a compatible service that will allow us to cut our storage bill in half. So we will be moving to that soon.

Sure, But Renting Still Costs More Over Time

Yes, regardless of where you host, renting machines will cost more over time vs owning and hosting your own machines. It’s worth it for us, because even though, like Basecamp, we are a company with stable growth, it’s still important for us to be able to bring up new machines quickly and/or for short periods of time. Let me give you a recent example.

We use Postgres for our database, in a three machine cluster configuration. We needed to upgrade from the older version we were using to the latest version. For us, the best way to do that upgrade was to bring up three new database machines, copy the database over, test it, switch the site to the new cluster, and then retire the old cluster. With Linode, we were able to bring up three beefy new machines quickly and just as quickly retire the three old machines. If we were self hosting, we would have to purchase those new machines, at considerable expense. We would have ended up with three large machines sitting idle after the upgrade.

The Pain Of Owning Your Own Servers

I’ve had to run to the datacenter at 3AM to fix a broken machine. I’ve had to deal with datacenter (lack of) cooling issues causing hard drives to prematurely fail (thankfully SSDs moot this). I’ve had to deal with flaky server BIOSes. I’ve had to deal with running out of rack space in a datacenter that’s full. I’ve had to deal with negotiating co-location and bandwidth contracts. I am more than happy to pay more to never have to deal with any of that again. I do not consider hardware to be one of our core competencies, nor do I want it to be.

Decentralize All The Things

One final point that David makes is that many services are hosted on Amazon, and that is not a good thing:

It strikes me as downright tragic that this decentralized wonder of the world is now largely operating on computers owned by a handful of mega corporations. If one of the primary AWS regions go down, seemingly half the internet is offline along with it. This is not what DARPA designed!

I completely agree with David on this. And I like to think that we’re doing our part.


Groups.io is the best tool to get a bunch of people organized and sharing knowledge. Start a free trial group today.

Re: Introduction

Puppy by Jeff Koons, Bilbao, Spain

I’ve been one poor correspondent
And I’ve been too, too hard to find
But it doesn’t mean you ain’t been on my mind
Sister Golden Hair, America (the band)

After a long dormant period, I am reactivating the blog. I figure this would be a good time to re-introduce myself, and to do so, because I generally find these types of posts boring and self congratulatory, I will borrow a concept from Scalzi, the “fake interview.” Let’s begin.

So, dude, what’s with the domain name?

That’s how you want to start this? Ok, fine. A long time ago, I had the nickname of ‘pig’, because I put a picture of a pig on my resume.

That was dumb.

I mean, yeah, sure, now that you mention it and also, looking on 25 years later. In my defense, I did get the job, and made a bunch of friends.

I’m not liking your tone. Can we get back to the intro now?

Whatever, pig boy. Ok, besides that bad decision, who are you?

For the purposes of this blog, I’m a software engineer (I’m also a dad to 9 year old twins and husband to Suzanne). I’ve started three companies. I started ONElist, an email groups service, in 1997, and it was acquired by Yahoo in 2000 (where it was renamed Yahoo Groups). I started Bloglines, an online RSS aggregator before Google Reader was a thing, in 2003, and it was acquired by Ask Jeeves in 2005. I started Groups.io, an email groups service, in 2014, and I still run it. Also, you’re really mean.

Wait, you did the email groups service thing like 25 years ago, and you’re doing it again?

I am.

Out of ideas, eh?

Nope. I still think it’s a good idea and is something that should exist. Fortunately, many people agree with me. Also, it just passed its 8th anniversary. I believe it is a net positive in the world, and that’s important to me.

So, gonna take lots of VC money, hire a ton of people, grow the service and then pawn it off on an acquirer where it’ll then languish and die? I mean, that does seem to be your look.

Again, nope. I did that with ONElist, ’cause that was the style of the time. Bloglines never got to the point where I was ready to take funding before Ask came in with an acquisition offer. I am keeping Groups.io lean and focused, and I have no plans to take outside funding or to sell it.

You know, I didn’t think this interview would be so adversarial.

Hey, I’m trying my best to keep this from being boring, but you’re not giving me much to work with here.

Fair. How about we wrap this up?

So, what will you be talking about on this here flying pig blog?

I’ll be focusing on two things: the business aspects of running a non-VC funded Internet business from the viewpoint of a tech founder, and the technical aspects of running a site like Groups.io. Expect a post about every two to four weeks.

Don’t kill yourself with the posting frequency there, Mr Post Master. Why are you doing this?

The same reason anyone does a blog like this, marketing, of course. And while I’m here, please check out Groups.io for all your email groups needs. Email groups are a great way for groups of people large or small to stay in touch. We have a complete collaboration suite, including: group calendar, files, photos, and wiki.

Corporate shill. You probably didn’t even write this. I’m afraid to ask; what’s with the America song lyric at the top?

What can I say, I’m a fan of the smooth sounds of yacht rock. And yes, I wrote this.

Yacht Rock? Seriously?!? That’s it, I’m out.


Groups.io is the best tool to get a bunch of people organized and sharing knowledge. Start a free trial group today.

Best And Worst Decisions – Mark Moore

A few years ago, for Startupping, I asked several entrepreneurs about their best and worst decisions. With the shuttering of that site, their answers vanished from the web. I’ll be reposting them here. This was originally posted on February 22, 2007:

Mark Moore is CEO and co-founder of One True Media, where he is responsible for the vision, business and product strategy as well as the teams and systems needed to execute on them. Previously, Mark was a founding member of three start-ups, Listen.com, MilleCom, and Diba, Inc., where he built successful development and web operations teams. At Oracle Corporation, Mark began his career developing that company’s Media Server and database kernel. (ed. disclosure: I am an investor in One True Media).

Best Decision:
I’m preaching to the choir here but the best decision a startup can make is to get the initial product out quickly, take feedback, and iterate rapidly. I understand that not all companies can work in this mode (some require longer time to create and iterate due to complexities around the product, etc) but the closer you can work to this methodology than the better chance you will have for success. In my experience, I have found that what you envision in your original business plan will not be true after six months. This is because you will understand the market better and have a better feel for what will really be successful after being in business for a while. So, get started as soon as possible by introducing your product to the market and then you can get to writing a “real business” plan that works and has been tested in the marketplace. For the entrepreneur who wants to be stealth for a year, I say good luck and make sure you have an extra year of funding.

Biggest Mistake:
Ahhh, the list is so long. Well, I would have to say some of my biggest mistakes involve the business partnerships that are made while the company is just starting out. Every entrepreneur would love to sign a deal with a large company which gives them access to a large market, or simply creates a solid revenue stream for the company. The problem is that “there is no free lunch”. Large companies will simply not hand a big opportunity to a small company. And if they do, they will make sure they will take the lion’s share of the returns (or will be able to replace the startup quickly with their own). In most cases, these relationships turn sour because they don’t work (they are just an experiment by the large company and little effort is expended causing failure) and at the same time they can cause a very large distraction for a fledging startup. So, my advice is beware of large partners – they know their business very well and are not about to give you a “free lunch”. There are exceptions to this rule but they are far and few between – just make sure you are getting something valuable in return from a partner before you take the leap.

My cats in the New York Times

Well, not really, but I am quoted in this article on straight, single men who own cats. It’s a silly piece, which the author Abby Ellin fully admitted when we talked. And actually, I think I laughed through the entire interview, which I’m sure didn’t make me a good interview subject. Having never, you know, been interviewed about my cats before, it was an interesting experience.
I can’t say that I necessarily agree that it’s only now becoming socially acceptable for a straight single man to own a cat (or two). But who can argue with the quote that “[t]hey make the best boyfriends…” and that “straight men with cats seem to be really secure and stable.” Hey, if it’s in the New York Times, it’s true!

Reaction to Stealth Start-Ups Suck

Greetings from Tokyo. The response to Stealth Start-Ups Suck has been fantastic so far, both pro and con. Thanks to everyone who’s been commenting on it. You can track people talking about it through this link for Bloglines citations.
I didn’t mean to specifically pick on 24 Hour Laundry and I didn’t realize others had already been doing so. I don’t know them and I’m sure they’re nice and smart people. The CNET article about them was just the trigger that got me to write that piece. Actually, a meeting ealier in the week with a friend who has his own stealth start-up really got the ball rolling. I implored them to publically launch their service and I hope they do soon.

Miscellaneous

  • Yesterday I took another step in my flying career. I completed my second supervised solo. This now means that I can go practice without having my instructor around. Of course, I’m subject to many limitations, on weather and other things. But still, I’m very happy.
  • Bloglines continues to gain momentum. I’m getting an amazing amount of great feedback. Everyone who uses it seems to really like the service. I just need to continue to focus on getting the word out, and continue to roll out new features.

Pigs Fly

Today, after two months of flying lessons, and dozens upon dozens of supervised take-offs and landings, I soloed for the first time. I did 3 take-offs and landings on my own at Palo Alto, in Citabria N5054B. Below is a picture of the plane and I after the flight. Right after that picture was taken, as per tradition, the back of my shirt was cut off and marked up to commemorate the event. On the shirt now is a drawing of a pig landing at an airport. It now graces a wall in West Valley’s Palo Alto office.
I’m probably about half-way towards my private pilot’s license. This flying thing is pretty fun.