Hey all! There are some stories that were foundational in my path to DevOps that I think illustrate larger issues pretty well. I use them a lot when I do talks or whatever but I figured I might as well put some of them down here to inspire readers.
I was working at National Instruments in the IT department back in the early 2000s. I ran the Web Systems team, alongside a bunch of technology focused teams – the UNIX Admins, the Windows Admins, Notes Admins, Network Admins, DBAs, and various others.
It took 6 weeks to get a new server for the Web. We’d say what we wanted, but then the UNIX admins would spec it out, and then order it from Dell, wait 2 weeks for fulfillment, then it’d come, then the network team would rack and jack it in the server room, the UNIX admins would put the OS on it, various other teams would get their pound of flesh, and then eventually it would get sent to us so we could put our software on it and then let the app teams use it. Not exactly quick turnaround.
Then came VMWare, the shining new star! Their sales rep showed up and said “You know, you can have a new server in 15 minutes.” We went out to a live sales demo and were like “this is very cool.” I asked the sales team, “so I could make the root read only for security reasons and write everything to external storage right?” They thought a minute, never having considered that. “Yes? That’s actually a really good idea.” A coworker and I fist-bumped each other.
Anyway, so we bought it, yet another technology vertical team was formed to own it, and after some degree of enterprise folderol it was in place.
So now when I needed a new server, guess how long it took me?
Four weeks. All we had removed was the 2 weeks of Dell fulfillment. Not 15 minutes.
When the procurement was slow anyway, it was just kind of accepted that 2 weeks turned into 6 weeks in the process. “Well, someone has to carefully assign an IP address, or else everything would be chaos!” But then when it was 15 minutes turning into 4 weeks purely based on our own internal friction (the actual work involved being much less than a day end to end) – suddenly it became clear how much of our own problem we were generating.
But it was hard to fix. There is inevitably niche protection in an environment like that. “Hey UNIX admins, you put cfengine on the boxes to manage them? Can we use that to load on our software too?” “No.” “Hey, can we get our own Web DBA so Web initiatives aren’t always bottom-feeding off the big ERP initiatives?” “No.” Tossing a ball over walls from silo to silo inevitably resulted in complex handoff processes and wait times and resource contention.
So then when we moved to the R&D department and started working up some SaaS products, and we were faced with the option of using our own integral IT department – we said “Absolutely not. There’s this new Amazon Web Services thing and we’re going to do it all ourselves.” “But what about all the value those specialists bring?”
Well, it’s not the specialists’ fault – but the one day worth of specialty benefit they brought was stapled to 4 weeks of sitting on hands, and no one felt it was their role to fix that endemic process problem in the org. I don’t care if you’re offering me Linus Torvalds to fix my Linux problems, if you offer me “a day of Linus and then 4 weeks in solitary,” I’ll do without, thank you very much.
This was 2009, it was before “DevOps,” but we immediately realized that having a product team responsible end to end for the product – code, systems, security, etc. – allowed us to deliver high quality in a fraction of the time that we were used to. It was intoxicating. That’s actually where we Agile Admins met and why we still hang out together today, because we all went through the experience of the gig actually becoming fun again!
the agile admin 