Velocity 2010 – Getting Fast

The first session in Day 1’s afternoon is Getting Fast: Moving Towards a Toolchain for Automated Operations.  Peco, Jeff, and I all chose to attend it.  Lee Thompson and Alex Honor of dto gave it.

I have specific investment in this one, as a member of the devops-toolchain effort, so was jazzed to see one of its first outputs!

A toolchain is a set of tools you use for a purpose.  Choosing the specific tools should be your last thing.  They have a people over process over tools methodology that indicates the order of approach.

Back in their ControlTier days they wrote a paper on fully automated provisioning.  Then the devops-toolchain Google group and OpsCamp stuff was created to promote collaboration around the space.

Discussion on the Google group has been around a variety of topics, from CM to log management to DevOps group sizing/hiring.

Ideas borrowed from in the derivation of the devops toolchain concept:

  • Brent Chapman’s Incident Command System (this is boss; I wrote up the session on it from Velocity 2008)
  • Industrial control automation; it’s physical but works similarly to virtual and tends to be layered and toolchain oriented.  Layers include runbook automation, control, eventing, charting, measurement instrumentation, and the system itself.  Statistical process control FTW.
  • The UNIX toolchain as a study in modularity and composition; it’s one of the most durable best practices approaches ever.  Douglas McIlroy FTW!

Eric Raymond (The Art of UNIX Programming, The Cathedral and the Bazaar)

Interchangeable parts – like Honore Blanc started with firearms, and lean manufacturing and lean startup concepts today.

In manufacturing, in modern automation thought, you don’t make the product, you should make the robots that make the product.

Why toolchains?

  • Projects are failing due to handoff issues, and automation and tools reduce that.
  • Software operation – including release and operations – are critical nonfunctional requirements of the development process.
  • Composite apps mean way more little bits under manangement
  • Cloud computing means you can’t slack off and sit around with server racking being the critical path

Integrated tools are less flexible – integratable tools can be joined together to address a specific problem (but it’s more complex).

Commercial bundled software is integrated.  It has a big financial commitment and if one aspect of it is weak, you can’t replace it.  It’s a black box/silo solution that weds you to their end to end process.

Open source software is lots of independent integratable parts.  It may leave gaps, and done wrong it’s confused and complicated.  But the iterative approach aligns well with it.

They showed some devops guys’ approaches to automated infrastructure – including ours!  Woot!

KaChing’s continuous deployment is a great example of a toolchain in action.  They have an awesome build/monitor/deploy GUi-faved app for deploy and rollback.


Then they showed a new cut at the generalized architecture, with Control, Provisioning, Release, Model, Monitoring, and Sources as the major areas.

Release management became a huge area, with subcomponents of repository, artifact, build, SCM, and issue tracker.

In monitoring and control, they identified Runbook Automation, Op Console/Control, Alarm Management, Charting/History/SPC, and Measurement Instrumentation.

Provisioning consists of Application Service Orchestration, System Configuration, Cloud/VM or OS install.

This is all great stuff.  All these have open source tools named; I’ll link to wherever these diagrams are as soon as I find it!  I must not have been paying enough attention to the toolchain wiki!

Hot Tips

  • Tool projects fail if the people and process aren’t aligned.
  • Design the toolchain for interoperability
  • Adopt a SDLC for any tool you develop
  • Separate the dev release process from the package process
  • Need better interchange boundaries (the UNIX pipe equivalent)
  • No one size fits all – different tools is OK
  • Communication is your #1 ingredient for success

All in all an awesome recap of the DevOps toolchain effort!  Great work to everyone who’s done stuff on it, and I know this talk inspired me to put more time into it – I think this is a super important effort that can advance the state of our discipline!  And everyone is welcome to join up and join in.

Leave a comment

Filed under Conferences, DevOps

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.