Velocity 2010 – Infrastructure Philharmonic

Whew!  This is a marathon.  Next, we have John Willis and Damon Edwards on The Infrastructure Philharmonic: How Out of Tune are Your Operations? I really wanted to see Lenny’s talk as well but had to make a hard decision.  Shout out to him, transparentuptime.com rocks!  As always, my personal comments will be in italics below.

Note – download the DevOps Cafe podcasts by John and Damon and give them a listen!  They rock!

What separates high and low performing IT organizations?  If you’re average, the leader is 2-3x better than you.  There are a lot of “good” qualities that are found in both.

What do high performing organizations specifically share?

Pretty simple:

• Seeing the whole – holistic vision, common goals.
• Tune the organization for maximum business agility.

What gets in the way?

Specialization.  We value deep specialization because that gets us paid more. And allows people to act like a-holes with impunity.

Why change?

Competition, durrr.

And on a personal basis, the new specialization is integration and people want that.

Our Analogy: The Philharmonic

Highly skilled individual contributors that need to contribute to a seamless whole.

1. The sponsors – business & marketing.
2. The musicians – network, systems, database, etc etc.
3. The audience – users!
4. The conductor – leadership.  Coordination, bridging.

I like them talking about the conductor as an occasional manager myself. Seems like a lot of the time people talk about this stuff like it should  magically emerge from among peers and that’s not the way the world works.

Antipatterns – But sometimes there’s not one conductor – there’s a dev manager and an ops manager.  And if the person responsible for the full lifecycle is more than 3 degrees away from the actual process, it doesn’t work.

An IT organization’s musical “ear” evaluates output, shares understanding of goals, impacts individual decisions, and is tuned for your specific business needs.

Antipatterns – individual focus, script based, limited reusability, etc.

Good patterns – ops as code, team focus, reusability, method/process, sournce control.

Developing your “ear” starts with what you can measure.  “Ear” isn’t all subjective, there’s some science to it.  You need to start at the top with measurements that are meaningful to the business.

Antipattern – There’s not enough visibility!  Time for a metrics project!  We’ll get a sea of information and suddenly via BI or something we’ll get the Matrix.  But really you end up with a bunch of crap data.

We did this at NI just recently!

Measurement: a set of observations that reduce uncertainty where the result is expressed as a quantity.  Doesn’t have to be perfectly precise.

It’s about the high level KPIs and not the low level metrics.  Start with THREE TO FIVE.  Don’t be a fool with your life.  To get KPIs:

1. Step 1 – Get everyone together and pick ’em
2. Step 2 – Tie back to lower level metrics
3. Step 3 – Tie back to performance (and compensation!)
4. Step 4 – Profit

High performing organizations:

1.  Automate as a way of life.  Check out the Tale of Two Startups post by Jesse Robbins on Radar.

Infrastructure as code requires:

• Provisioning (gimme boxes!)
• Configuration Management (add roles!)
• Systems Integration/Orchestration (crossconnect!)

2.  Test as a way of life.  Testing as a skill -> testing as a culture -> quality as a culture.  Check out what kaChing does.  They built a “business immune system” with testing and monitoring deviation – in INTERVIEWS they have people write code to go to production.

3.  DevOps culture.  Get past the disconnects in culture, tools, and process.

Batching up deploys turns your agile dev into a waterfall result.  DON’T BE A FOOL WITH YOUR LIFE!!!

In my previous role in IT, I kept hearing some higher-ups wanting “fewer releases.”  Why release more than once a quarter?  Those monthly releases are so costly!  I always tried to not cuss people out when I heard it.  If your new code is so worthless it can wait an extra two months to go out, just don’t roll it out and save us all some hassle.

Operations wants…

To get out of the muck!  People want to add value and implement things and not just fight fires.  We’re in an explosion right now where ops gets to come out and play!  We want to be agile and say “yes”!

Amen.