Monthly Archives: March 2011

Why Amazon Reserve Instances Torment Me

We’ve been using over 100 Amazon EC2 instances for a year now, but I’ve just now made my first reserve instance purchase. For the untutored, reserve instances are where you pay a yearly upfront per instance and you get a much, much lower hourly cost. On its face, it’s a good deal – take a normal Large instance you’d use for a database.  For a Linux one, it’s $0.34 per hour.  Or you can pay $910 up front for the year, and then it’s only $0.12 per hour. So theoretically, it takes your yearly cost from $2978.40 to $1961.2.  A great deal right?

Well, not so much. The devil is in the details.

First of all, you have to make sure and be running all those instances all the time.  If you buy a reserve instance and then don’t use it some of the time, you immediately start cutting into your savings.  The crossover is at 172 days – if you don’t run the instance at least 172 days out of the year then you are going upside down on the deal.

But what’s the big deal, you ask?  Sure, in the cloud you are probably (and should be!) scaling up and down all the time, but as long as you reserve up to your low water mark it should work out, right?

So the big second problem is that when you reserve instances, you have to specify everything about that instance.  You aren’t reserving “10 instances”, or even “10 large instances” – you have to specify:

  • Platform (UNIX/Linux, UNIX/Linux VPC, SUSE Linux, Windows, Windows VPC, or Windows with SQL Server)
  • Instance Type (m1.small, etc.)
  • AZ (e.g. us-east-1b)

And tenancy and term. So you have to reserve “a small multitenant Linux instance in us-east-1b for one year.” But having to specify down to this level is really problematic in any kind of dynamic environment.

Let’s say you buy 10 m1.large instances for your databases, and then you realize later you really need to move up to an m1.xlarge.  Well, tough. You can, but if you don’t have 10 other things to run on those larges, you lose money. Or if you decide to change OS.  One of our biggest expenditures is our compile farm workers, and on those we hope to move from Windows to Linux once we get the software issues worked out, and we’re experimenting with best cost/performance on different instance sizes. I’m effectively blocked from buying reserve for those, since if I do it’ll put a stop to our ability to innovate.

And more subtly, let’s say you’re doing dynamic scaling and splitting across AZs like they always say you should do for availability purposes.  Well, if I’m running 20 instances, and scaling them across 1b and 1c, I am not guaranteed I’m always running 10 in 1b and 10 in 1c, it’s more random than that.  Instead of buying 20 reserve, you instead have to buy say 7 in 1b and 7 in 1c, to make sure you don’t end up losing money.

Heck, they even differentiate between Linux and Suse and Linux VPC instances, which clearly crosses over into annoyingly picky territory.

As a result of all this, it is pretty undesirable to buy reserve instances unless you have a very stable environment, both technically and scale-wise. That sentence doesn’t describe the typical cloud use case in my opinion.

I understand, obviously, why they are doing this.  From a capacity planning standpoint, it’s best for them if they make you specify everything. But what I don’t think they understand is that this cuts into people willing to buy reserve, and reserve is not only upfront money but also a lockin period, which should be grotesquely attractive to a business. I put off buying reserve for a year because of this, and even now that I’ve done it I’m not buying near as many reserve as I could be because I have to hedge my bets against ANY changes to my service. It seems to me that this also degrades the alleged point of reserves, which is capacity planning – if you’re so picky about it that no one buys reserve and 95% of your instances are on demand, then you can’t plan real well can you?

What Amazon needs to do is meet customers halfway.  It’s all a probabilities game anyway. They lose specificity of each given reserve request, but get many more reserve requests (and all the benefits they convey – money, lockin, capacity planning info) in return.

Let’s look at each axis of inflexibility and analyze it.

  • Size.  Sure, they have to allocate machines, right?  But I assume they understand they are using this thing called “virtualization.”  If I want to trade in 20 reserved small instances for 5 large instances (each large is 4x a small), why not?  It loses them nothing to allow this. They just have to make the effort to allow it to happen in their console/APIS. I can understand needing to reserve a certain number of “units” but those should be flexible on exact instance types at a given time.
  • OS. Why on God’s green earth do I need to specify OS?  Again, virtualized right? Is it so they can buy enough Windows licenses from Microsoft?  Cry me a river.  This one needs to leave immediately and never come back.
  • AZ. This is annoying from the user POV but probably the most necessary from the Amazon POV because they have to put enough hardware in each data center, right?  I do think they should try to make this a per region and not a per AZ limit, so I’m just reserving “us-east” in Virginia and not the specific AZ, that would accommodate all my use cases.

In the end, one of the reasons people move to the cloud in the first place is to get rid of the constraints of hardware.  When Amazon just puts those constraints back in place, it becomes undesirable. Frankly even now, I tried to just pay Amazon up front rather than actually buy reserve, but they’re not really enterprise friendly yet from a finance point of view so I couldn’t make that happen, so in the end I reluctantly bought reserve.  The analytics around it are lacking too – I can’t look in the Amazon console and see “Yes, you’re using all 9 of your large/linux/us-east-1b instances.”

Amazon keeps innovating greatly in the technology space but in terms of the customer interaction space, they need a lot of help – they’re not the only game in town, and people with less technically sophisticated but more aggressive customer services/support options will erode their market share. I can’t help that Rackspace’s play with backing OpenStack seems to be the purest example of this – “Anyone can run the same cloud we do, but we are selling our ‘fanatical support'” is the message.


Filed under Cloud

DevOps In Action Podcast

While at SXSW, I recorded an episode of the IT Management and Cloud Podcast with Michael Coté and John Willis. You can find it on Coté’s People Over Process blog! Sorry about the background noise, we were doing it in the upstairs bar at the Driskill.

Leave a comment

Filed under Cloud, Conferences, DevOps

SXSW Interactive 2011 Day Two

Day 2 started off a bit rocky but then got really good.  I tried to get to the Lean Startup: King of the Apps Showdown session, but the shuttle buses take a long route and are few and far between, so I got there pretty late.  I saw, snapwebsites, Planzai, Icebreakr,and mapomatic all get their app and pitch critiqued by a panel of folks – Robert Scoble, Eric Ries, Dave McClure, Bill Boebel, and Stacey Higginbotham. They demanded mapomatic release right away, and told snapwebsites to remove half of his features.  I was ambivalent about this advice – I can see streamlining the UI to hide advanced functionality, but “remove functionality”? Sure, a simple app has a wider reach, but if you can effectively provide an advanced mode/version with more functionality that professionals need, wouldn’t that be better? I just don’t like how all these apps are degenerating into one very narrow function “plus it checks you into Foursquare!!!!”  Bah.

They joked about how many of the candidates had “Snap” in their name, and then on the bus on the way back I met a girl from TeamSnap. They have a site that lets you arrange sports teams, even take Paypal payments for team dues, very slick.

And congratulations to friend of the blog Lenny Rachitsky (@lennysan) of Localmind, who has been singled out by Scoble as “best SXSW app so far!” You should all go download it from the App Store now.

Then I tried to go to the session on How To Innovate At Big Companies, with Gene Kim of Visible Ops fame – I left the AT&T center early and went all the way to the Hyatt. But it was so full they had 30 people standing outside only allowed in if anyone left. So I went over to the Hilton to try to catch Mistakes I Made Building Netflix for the iPhone. And it also had a line of “one in, one out” standby people. Son of a bitch.  They better be videoing this stuff. Anyway, at this point it was too far into the slot so I just went to the Screenburn Arcade, which was fun. Got my picture taken with Loren Wiseman of Traveller fame – apparently the venerable RPG is being turned into an iPhone MMO!

To recuperate from our disappointing morning, Peco and I went and had a burger at Casino El Camino, which takes a ridiculously long time but is worth it, they have the best burgers in Austin. We hooked up with Cote and John Willis there, and then went to the Etsy Code as Craft: Moving Fast At Scale event which was brilliant and DevOpsey. Continuous deployment, only coding off trunk, logging and metrics, “dashboard driven development”, and more. A lot of the topics have info on them on the Etsy Code as Craft blog – they didn’t record the talk but promise to record it the next time they give it and put it up.

I took a lot of notes but the high points were:

  • They have open sourced their log collection/graphing tool, Logster, and their stats collection daemon, statsd.
  • They perform hundreds of releases a month but only had six “bad deploys” over the course of a year by combining one-button deploy, testing, feature flags, and a  theory of roll-forward. No more organized releases, release managers, rollbacks, all that stuff.
  • They have a cute “deployinator” dashboard that lets people (and not just coders, they have more people deploying than they have developers) do the deploys – they’ll open source it, but stressed that it’s not much technically, the culture and practices are 99% of the work to get to this.
  • They use a lot of chef but don’t use it for code deploys because that’s a simple task that they need more control over.

That was a two hour event, so then after a little VIP partyin’ atop Fogo de Chao, we packed it in for the day.

Leave a comment

Filed under Conferences

DevOps at CloudCamp at SXSWi!

Isn’t that some 3l33t jargon.  Anyway, Dave Nielsen is holding a CloudCamp here in Austin during SXSW Interactive on Monday, March 14 (followed by the Cloudy Awards on March 15) and John Willis of Opscode and I reached out to him, and he has generously offered to let us have a DevOps meetup in conjunction.

John’s putting together the details but is also traveling, so this is going to be kind of emergent.  If you’re in Austin (normally or just for SXSW) and are interested in cloud, DevOps, etc., come on out to the CloudCamp (it’s free and no SXSW badge required) and also participate in the DevOps meetup!

In fact, if anyone’s interested in doing short presentations or show-and-tells or whatnot, please ping John (@botchagalupe) or me (@ernestmueller)!

Leave a comment

Filed under Cloud, Conferences, DevOps

SXSW Interactive 2011 Day One

We started out the week gently, with a light Friday (why am I still so tired then?).  Two of your favorite Agile Admins, Peco and Ernest, were down at the Austin Convention Center to experience SXSW Interactive!  The third Agile Admin, James, was busy giving a talk at the nearby Security B-Sides Austin.

After getting down there early, getting badges, and getting oriented, we saw Jason Calacanis interview Tim O’Reilly. He gave a lot of interesting insight into the development of innovation, tracing the Internet, open source, Web 2.0/social media/user contributed content, cloud, and Big Data.

The single most interesting thing he said, though, was a side note that illustrated how hard it is for companies to maintain a real vision over time, especially once they get big and various stakeholders’ needs conflict – he talked about how many people have become billionaires off of O’Reilly ideas and how there’s been pressure for him to sell out or cash in – like Cisco offered to buy them, noting that “You guys are always first on the scene with the cool stuff but you fail to exploit it.” Tim always rejected those offers, though he did note he is often conflicted because of all the people he has working for them – and not making all those great people the money that would bring.

Next, Peco saw Google’s Marissa Meyer talk (he’ll have to share what went on there) and I kinda played hooky by going to a session about “daddy bloggers,” as that’s a personal interest of mine.

Then we both went to a session that was supposed to be about “The Connected Car: Driving Technology” and automobile telemetrics, but it sucked. It was some car guys and a lady from Pandora telling us over and over again that “cars hooked up to networks and stuff are cool.” I am willing to put up with about 5 minutes of telling me something’s cool, but if it doesn’t give way quickly to you SHOWING me that it’s cool, I’m out.  We bailed 30 minutes in, along with a lot of the other attendees.

Then we went down to the Austin Music Hall for four hours of Ignite, a format where presenters get 5 minutes and auto-advancing slides to make a point. The topic was “2021 Visions of the Future.”  Besides the talks and some bands, there were also a bunch of Arduino and robotics and various weird Maker type booths, which was fun. The crowd was really varied, in fact there were other people there from NI who had been invited via various completely different vectors (UT School of Engineering, Austin Ventures startup stuff, SXSWi). [Side note – there was a contest to drop an egg from the balcony safely using only 4 sheets of paper and a couple feet of tape, and I was one of the winners and got a Roku for my troubles! Engineering education FTW.]

Some presentations were good, others incoherent, but a common theme throughout the day was people doing things they have a passion for, and worrying about the money later.  This was a theme in Tim O’Reilly’s talk and it pervaded the presentations at Ignite. Really most people aren’t in the field they’re in because it’s the thing with the highest ROI they qualify for, they are in it because they  have some kind of passion for it.  A lot of life then tries to stomp that passion out, but the enabling factors of the Web, DIY, etc. are making it so you can flip off  “the Man” and pursue your passion yourself if you want to. As a large company we struggle with that, and try to promote people following their passions and enable internal entrepreneurship at a high level, while however the natural grinding wheels of a large organization grind that into meal at the middle levels.

Then we got free ice cream from the Free Ice Cream Man and headed home. I’m getting sick and feel whupped, but it was energizing to see so many people doing so many great things – and let me tell you, SXSWi is getting HUGE.  It’s nothing like the first year I went, when there were maybe ten sessions and the conference center was largely empty. The place was packed; companies have bought out and transformed nearby buildings – a bar across the street is now the Playstation Lounge, with huge video wall on the roof and stuff; CNN had a giant neon sign installed at one restaurant… I can tell the economy’s looking up and that Interactive is hot because Lordy there’s money getting thrown at this thing.

See the rest of my pictures from Day One of SXSW here!

Leave a comment

Filed under Conferences

SXSW Tips If You’re From Out Of Town

I just went to lunch with a visitor in town for SXSW (@lennysan from Localmind) and it reminded me of some of the ‘gotchas’ that someone not from Austin may not know.

  • If your hotel doesn’t have a shuttle, and it is not immediately inside the downtown area, you will need a car. (Unless you are real close to the one light rail line, and I wouldn’t bet on it not being totally overloaded). Austin is big and not pedestrian friendly outside of IMMEDIATELY adjacent to the river.
  • Here’s an awesome downtown parking guide from Community Impact. Shows where they are, what the rates are, whether they’re lot or structure.
  • You can rent your gun at the airport when you land, to avoid spending time finding another place to do it.
  • Just because a place has a big “BBQ” sign doesn’t mean you should go eat barbecue there.  We were eating real BBQ at Rudy’s and looking across the highway at the Bone Daddy’s that has a big ol’ sign on it saying BBQ.  I’m not saying Bone Daddy’s doesn’t have things to recommend it, but BBQ isn’t really its strength.  If you’re downtown, eat at the Iron Works, and if anyone is organizing a field trip out of town to the really golden places in Taylor or Llano, go along.
  • Someone asked on Localmind about breakfast during SXSW – here, breakfast is breakfast tacos, except for hotel restaurants and Denny’s.

Any more tips for furriners?  Post them here!

1 Comment

Filed under Conferences

SXSW Interactive Is Here!

SXSW Interactive is going on in Austin tomorrow through next Tuesday, and loads of great cloud and DevOps folks will be in town for it. Looking forward to talking with @cote, @lennysan, @botchgalupe, @davenielsen, @ehuddleston, and many more.

Here’s what I think the best cloud/devops/high tech related tickets are, let me know what I’m missing! A lot of the off premise events don’t even require badges and are mostly free.

Random Off Premise SXSW Interactive Stuff

Other events – not job related but make me happy:


Sessions are less important than the other stuff so they’re on here second!  No time for links, search on ’em.

What’s the good stuff I haven’t mentioned?  DevOps, Cloud, noSQL, and other cool stuff report below!

Leave a comment

Filed under Conferences