OpsCamp Debrief

I went to OpsCamp this last weekend here in Austin, a get-together for Web operations folks specifically focusing on the cloud, and it was a great time!  Here’s my after action report.

The event invite said it was in the Spider House, a cool local coffee bar/normal bar.  I hadn’t been there before, but other people that had said “That’s insane!  They’ll never fit that many people!  There’s outside seating but it’s freezing out!”  That gave me some degree of trepidation, but I still racked out in time to get downtown by 8 AM on a Saturday (sigh!).  Happily, it turned out that the event was really in the adjacent music/whatnot venue also owned by Spider House, the United States Art Authority, which they kindly allowed us to use for free!  There were a lot of people there; we weren’t overfilling the place but it was definitely at capacity, there were near 100 people in attendance.

I had just heard of OpsCamp through word of mouth, and figured it was just going to be a gathering of local Austin Web ops types.  Which would be entertaining enough, certainly.  But as I looked around the room I started recognizing a lot of guys from Velocity and other major shows; CEOs and other high ranked guys from various Web ops related tool companies.  Sponsors included John Willis and Adam Jacob (creator of Chef) from Opscode , Luke Kanies from Reductive Labs (creator of Puppet), Damon Edwards and Alex Honor from DTO Solutions (formerly ControlTier), Mark Hinkle and Matt Ray from Zenoss, Dave Nielsen (CloudCamp), Michael Coté (Redmonk), Bitnami, Spiceworks, and Rackspace Cloud.  Other than that, there were a lot of random Austinites and some guys from big local outfits (Dell, IBM).

You can read all the tweets about the event if you swing that way.

OpsCamp kinda grew out of an earlier thing, BarCampESM, also in Austin two years ago.  I never heard about that, wish I had.

How It Went

I had never been to an “unconference” before.  Basically there’s no set agenda, it’s self-emergent.  It worked pretty well.  I’ll describe the process a bit for other noobs.

First, there was a round of lightning talks.  Brett from Rackspace noted that “size matters,” Bill from Zenoss said “monitoring is important,” and Luke from Reductive claimed that “in 2-4 years ‘cloud’ won’t be a big deal, it’ll just be how people are doing things – unless you’re a jackass.”

Then it was time for sessions.  People got up and wrote a proposed session name on a piece of paper and then went in front of the group and pitched it, a hand-count of “how many people find this interesting” was taken.

Candidates included:

  • service level to resolution
  • physical access to your cloud assets
  • autodiscovery of systems
  • decompose monitoring into tool chain
  • tool chain for automatic provisioning
  • monitoring from the cloud
  • monitoring in the cloud – widely dispersed components
  • agent based monitoring evolution
  • devops is the debil – change to the role of sysadmins
  • And more

We decided that so many of these touched on two major topics that we should do group discussions on them before going to sessions.  They were:

  • monitoring in the cloud
  • config mgmt in the cloud

This seemed like a good idea; these are indeed the two major areas of concern when trying to move to the cloud.

Sadly, the whole-group discussions, especially the monitoring one, were unfruitful.  For a long ass time people threw out brilliant quips about “Why would you bother monitoring a server anyway” and other such high-theory wonkery.  I got zero value out of these, which was sad because the topics were crucially interesting – just too unfocused; you had people coming at the problem 100 different ways in sound bytes.  The only note I bothered to write down was that “monitoring porn” (too many metrics) makes it hard to do correlation.  We had that problem here, and invested in a (horrors) non open-source tool, Opnet Panorama, that has an advanced analytics and correlation engine that can make some sense of tens of thousands of metrics for exactly that reason.


There were three sessions.  I didn’t take many notes in the first one because, being a Web ops guy, I was having to work a release simultaneously with attending OpsCamp 😛

The second was interesting.  Adam Jacob from Opscode moderated a talk on “DevOps – Is It The Devil?”  That’s my version of the title, I think he said “anti-pattern”.  Anyway, opinions on devops were mixed, as were opinions on what it means exactly.  Is it business alignment?  Sysadmins getting into the product code?  Better automation on the sysadmin side?  I have lots of opinions on this for later blog posts.  Also see the “dev2ops” blog for related info.

Adam says there’ll be more on this at Velocity, they’re planning an unconference the day after on this topic.

The third session I attended was kinda cool.  I forget what it was supposed to be about, but what it turned into was a back room Mafia-style sit-down between all the major players that came to the event, including Luke, Adam, and Damon, to talk about how to work together, in that a comprehensive model for automated infrastructure would be of joint value to everyone.

Big thoughts from the session:

Controltier did a previous diagram and white paper showing how some of the tools fit together – it is well regarded and it helped me personally when I first started trying to figure out the CM landscape.

It’s really here where big companies like HP and IBM beat out open source.  Their software isn’t better by any stretch of the imagination.  I’ve personally used HP Deploy Management, for example, and it’s really not as good as some of the open source offerings.  But when they come in, they are able to provide you a comprehensive picture of how everything fits together, what you need, etc. that makes doing business with them easier.

My thoughts – guys, work together to get better.  You shouldn’t be worried about your piece of the miniature current pie, you should be looking to cut into the IBM/CA/HP business and get part of the huge pie.

The products need APIs so they can be integrated.  Especially, people need to be able to integrate their system provisioning and monitoring off the same configs.

In the end, everyone agreed that something big was in the wind with this “devops” movement and it was time for something to happen. The assembled, after some negotiation, told DTO they trust them to take a first shot at an architecture diagram and then they’d be happy to edit it.  Woot!

Random Tips

“DevOps” – new buzzword for the new role Ops folks are finding themselves in, this resonated with us as we’re having to combine ops and dev in our new cloud projects.

Visible Ops – A good book on ops recommended highly to us.  This, I think?

git – I wasn’t all that enthused about this new revision control system (“yet another one,” I thought), but Luke went on about it for a long time and I think I see some of its cooler points now.

One of the organizers (I forget which one) will start an “opsforum” google group so we can further collaborate online.  This is great – one of the biggest problems in the Web ops space is that there’s no good single place to go to bring all this under one umbrella.  We’ve had the Velocity conference for a couple years and now we have OpsCamp, but between events it’s all following people’s blogs, there’s no real community or collaboration space.

Here in Austin there’s other semi related entities like the LOPSA Austin chapter, cactus (unix), and geekaustin.org, but they’re more Austin only and not focused quite on point for Web ops.

cobbler, a Linux install server – I hadn’t heard of it before.  (I’ll be honest, I try not to stay down at the OS level too much any more…)  But it sounds cool.

ControlTier, a cool open source automation tool/company we met at Velocity and liked, has changed focus somewhat and corporately has become DTO Solutions, more of a consultancy around the whole automation area.  Seems like a good move for them.

People are getting a little disgruntled with Velocity – it seems to be leaning real heavy towards the front end performance thing and losing any meaningful focus on operations.  I tend to agree – it needs a wider focus – performance overall, not just front end, and more ops stuff.  And I love open source but let’s get the Splunks etc. of the world there too.

After Party

We tried to pay Spider House back adequately for providing the venue gratis by drinking the rest of the OpsCamp budget away there.  Mmm, Jameson.

Notable open source evangelist whurley showed up for dinner at Ruby’s later; he even got Luke to loosen up a bit.  Talking over drinks and dinner revealed that for many of them, it was their first time in Texas “besides the airports.” I thought it was a little funny that Texas still provokes a somewhat-joking fearfulness among visitors.  Being a native Texan, I have to admit on some level that pleases me.  Allow me to quote from Ulysses S. Grant’s memoir:

“The journey was hazardous on account of Indians, and there were white men in Texas whom I would not have cared to meet in a secluded place.”

I hope all the visitors had a good time in Austin, and I am excited to have some more OpsCamps!  I think they’re planning for it to be yearly, but I’d be happy to have an “Austin only” thing more frequently.  A number of admins I know couldn’t make it but would totally be down for such a thing.


Filed under DevOps, Uncategorized

10 responses to “OpsCamp Debrief

  1. I’d like to hear more about this: “People are getting a little disgruntled with Velocity – it seems to be leaning real heavy towards the front end performance thing and losing any meaningful focus on operations.”

    I’m part of the conference program committee, and I’m knee-deep in reviewing proposals for 2010’s conference, and there are really great non-frontend proposals. Having said that, given that client-side response time can be improved by >70% just by changing client-side code, I’ve always enjoyed the mixture of the two crowds. 🙂

    Is there a view that >50% of the conference is geared towards client-side performance?

  2. Ernest,

    I’ve heard a few folks say that they want this year’s Velocity 2010 to include more on the traditional backend Ops problem space. I completely agree, and we’re addressing that for Velocity 2010 & the Velocity Online Conferences (OLCs)


    -Jesse Robbins

  3. Josh

    Hey John and Jesse! First off, thanks for commenting. I didn’t get to go to Velocity, but being on the same team as Ernest, I heard plenty of what he had to say about it. I think the gist of what was saying was that there was just a heavy focus on things like page design, caching, CDN, and the rest of the Web Performance piece and little to do with anything Operations. I believe Ernest submitted a proposal to the conference that was turned down on a pure operations topic. I submitted two proposals, one on operational security and another on troubleshooting using logging volume and both of those were turned down. I believe another one of our team members submitted a proposal as well. All great topics, but passed by in favors of the same old talk from Souders and the like (as much as we do like him and his philosophies on performance). When it came time to submitting proposals to Velocity this year, I don’t think any of us even bothered because we figured it’d have been met with the same response as before. I kinda regret that decision now that you say you’re attempting to address the lack of Operational presentations from previous conferences. I definitely think it’ll be a change for the better.

  4. Hey y’all! I want to first say I love Velocity and was unhappy to hear that some of the major ops guys were considering not going. I’m not going to narc out who exactly but you know who all was there at Opscamp, and I was standing in a circle of “named” guys who were saying “I don’t know, I don’t know if they want us there, seems like their focus on ops is less and less.” So to a degree I’m just reporting what I heard.

    From my point of view, though, we had noticed even last year that the Velocity agenda was leaving us with fewer tangible takeaways. What I’d like is something holistically useful. Front end performance is important, and there was a big “revolution” in that space with Souders etc.’s work a couple years ago, but it’s a small piece of the entire puzzle.

    Even just the performance track needs to be about more than just front end performance and it needs to focus on specific takeaways that a Web ops kind of person would implement. There’s other venues for pure Web designers to go to IMO – that’s not this crowd; this crowd is a specific mix of Web generalists – part UNIX admin/open source wonk, part developer, small part designer. The more you go toward stuff only a pure Web designer who does nothing but CSS all day would care about, the more people like us will say “Hmmm, I think this year I’ll pass.”

    Also some of the stuff was very speculative (browsers, the weird proposed google protocol, quickling) that have no practical takeaways for someone. It also seemed like some of it was about highly esoteric fixes that would be a) for a super Web designer JS/CSS hacker and b) only relevant to the “top 5 guys”. It’s somewhat interesting to see the things that Google/Yahoo/Twitter/Facebook have had to do to operate at their scale, but 99% of the audience is smaller scale and won’t be doing that. “We spent 6 months optimizing those last two pixels!” Remember that the conference shouldn’t be for the top 1%, if it wants to make $ it should appeal to Joe Internet Startup/Jane Enterprise Web IT Person.

    Anyway, here’s what I’d propose as a mix.

    Performance – covering front end, back end, caching, memcached, hadoop, distributed computing, etc. (I liked some of the non-FE stuff in these veins at last Velocity)
    Operations – config management, monitoring, control, logging, security (OWASP type), etc.

    Focus on the core. There’s already cons for developers. There’s cons for designers. There’s cons for Web marketeers. But this is the first venue that brings together Web technologists specifically, especially devs/ops/devops.

  5. Pingback: Coté's People Over Process » Links for February 4th through February 8th

  6. Pingback: Agile Operations | Web Admin Blog

  7. Pingback: Come to CloudCamp Austin 2! | Web Admin Blog

  8. Pingback: Velocity and DevOpsDays! « the agile admin

  9. Pingback: Come to CloudCamp Austin 2! « the agile admin

  10. Pingback: Agile Operations « the agile admin

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.