Tag Archives: change management

Velocity 2010 – Always Ship Trunk

Always Ship Trunk: Managing Change In Complex Websites by Paul Hammond (Typekit)

No rest for the wicked.  More sessions to write up.  Let’s find out how to do feature switches, Flickr-style.  My comments are in italics.

Use revision control. Branching is sad because of merging.  But Mercurial and git make it all magically delicious.

Revision control is nice but what it doesn’t answer is what is running on a given Web server.

There are three kinds of software.

  1. Installed
  2. Open Source installed
  3. Web apps/SaaS

Web apps are not like installed apps.  Revision control is meant to deal with loads of versions.  With a Web app there’s about 1 version of your app in use.  If you administer every computer your software is installed on, you don’t have to worry about a lot of stuff.  Once you upgrade, the old code will never be run again.  It has a very linear flow.

But not really.  Upgrades don’t happen on every box simultaneously.  And shouldn’t – best practice is rolling to a subset.

And you push to a staging/QA environment first.  So suddenly you have more “installs.”  And beta environments.

You have stuff (dependencies) outside your control – installed library dependencies, Web service dependencies – all that change has to be managed.

Coordinating lots of peopel working at the same time is hard.

Deep thought alert: Nobody knows you just deployed unless you tell them.

You can separate the code deployment from the launch.  You can rewrite your infrastructure and keep the UI the same and no one knows.

Deep thought alert 2: You can run different versions in production at the same time.

Put it out.  Ramp up usage.  Different people can see different UIs and they don’t know.

What we need is a revision control system that lets up manage multiple parallel versions of the code and switch between them at runtime.

Branches don’t solve that problem for us (by themselves).  And they don’t help with dependency changes that affect all branches at once – if someone changes their Web API you call, it affects every version!

revision vs version.

Manage the different versions within your application – “branching in code.”  You know, if statements.

This is really dangerous if you don’t have super duper regression testing right?  I’m rolling a new version but not really…  Good luck on that.

This is the “switch concept.”  It allows for feature testing on production servers.

Join it with cookies and you can have a “feature flip” page!  You can put all kinds of private functionality into the app and rely on whatever if statement you wrote to make sure no one bad gets to it!  Good Lord!

There are benefits to production testing (even if it’s not from end users) – firewall stuff, CDN stuff, et cetera.  It’s very flexible.  You can do dark launches.  Run the code in the background and don’t display it.  Now that’s clever.

There are three types of feature flags

  1. user facing feature development
  2. infrastructure development
  3. kill switches

Disable login!

They have loads of $cfg[‘disable_random_feature’] = false

The cost of this is complexity.

Separate your operational controls from development flags.

Be disciplined about removing unused feature flags so it’s not full of cruft.

If you’re going to do this,  just go all in and always deploy trunk to every server on every deploy and manage versions with config.

Definitely daring.  I wonder if it’s appropriate for more “real” workloads than “I’m uploading my pics to a free service for kicks” though.

Joel Spolsky sayeth:  This is retarded.

With new style distributed merge, instead:

  • Use branches for early development. Branches should be merged into trunk.
  • Use flags for rollout of almost-finished code.

Is there a better alternative?  Everyone who makes revision  control systems makes them for installed software not Web software – what would one for installed software look like?

Q&A Tidbits: Put all the switches in one place… Not spread through the code.

What about Sarbanes/Oxley division of labor?  Pshaw.  This is for apps that are just for funsies.

You have to build some culture stuff to about devs not jsut hitting deploy and wandering off, but following up on production state.

1 Comment

Filed under Conferences, DevOps

Velocity 2010 – Change Management

Andrew “Dice” Clay Schafer of Cloudscaling on Change Management – A Scientific Classification.

The Dunning-Kruger effect says we can’t always judge our own level of competence.
Often we prefer confidence over expertise.

You have probably confidently deployed a disaster.

Premise:  Everyone is working to enable value creation for a business.  (Or should be out the door.)

If you don’t understand what creates value for your organization, how can you be good at your job?

Change Management: A structured approach to transitioning individuals, teams,  and organizations from a current state to a desired future state.

You look into doing this in the technology sphere.  Soon you get to the ITIL framework and you crap your pants.

Web Operations Renaissance

Process innovation is a differentiator that is driving value creation.

You can add value with performance, scalability, performance, and agility.

The 6 Laws of Reliability by Joe Armstrong of Ericsson (all this comp sci theory stuff was worked out in like the 1960s, just go look it up.)

  1. isolation
  2. concurrency
  3. faulire detection
  4. fault identification
  5. live upgrade
  6. stable storage

One approach – Cowboysaurus Rex.  Make changes live to production!

It’s fast and flexible.  But we suck at doing it safely.  Throwing more meat at it is a poor way of scaling.

The next approach – Signaturus Maximus – laying in the bureaucracy.

It adds adult supervision and has fewer surprises, but it slows things down.  And people don’t follow the process.  When there’s a crisis, it’s back to cowboy.

Next approach, ITIL Incipio!

Now, there are some good guidelines.  But no one knows what it really says.  And the process can become an end unto itself.  And when there’s afire, you break out of it and go back to cowboy.

Condo Adeptus.

Computers are better than people at doing the same thing over and over.  But, nothing can bring down your systems like automation. If you don’t test it, you jack yourself.

Novus Probator.

Your system is an application.  Run it through a dev process, and testing.  But we suck at testing.  Testing is its own skill and is often an afterthought.

Prodo Continuus.

All production all the time- “continuous deploy.”  If you build up the scaffolding to support it, you can push to prod without fear (almost).  You get feedback from prod very fast.  But we still suck at testing.  But you don’t break the same thing twice.

What are we neglecting? ell, what are you building?

You have some things that are less important (twitter) and ones that are more critical (your bank) and crazy critical (medical).

When we talk about changes, we assume it’s mostly apps, kinda some OS as those are less sensitive.  Maybe CPU, bot not really storage and network.  There is a criticality path but as much as you can do infrastructure as code all the way down the stack, the better.

There’s also a continuum of situation – from building to firefighting.  Perhaps different approaches are somewhat merited.

Dependencies aren’t just technology.  There are people in the value chain who need to know, and they are in different roles from us.  We need to be transparent to all those other folks and communicate better.

The wall of confusion between roles (like dev and ops) makes people sad if there’s not communication.  Hence, DevOps!  But there are many groups and many walls.

David Christensen’s FSOP (flying by the seat of your pants) cycle describes the problem – we don’t even know what best practices are yet.  Find some, use the, but go back to flying.  Never calcify anything you don’t have to.

Change your organization.  “But it’s hard and there’s politics!”  Strive to make the place where you work the place you want to work,  And if that doesn’t work out, there’s a place you should come work!

Come to DevOpsDays on Friday for more on how to do this!

Leave a comment

Filed under Conferences, DevOps