I was just reading the newest “state of DevOps” post on Agile Web Operations by James Turnbull and he mentions a set of rules some mainframe guys taught him back in the day:
- Think before you act
- Change one thing at a time not ten things at once
- If you fucked up then own up
This reminded me of the standing orders I had as manager of our Web Ops shop here for many years. Mine were:
Web Operations Standing Orders
- Make it happen
- Don’t fuck up
- There’s the right way, the wrong way, and the standard way
This was my trenchant way of summing up the proper role of an innovative Web operations person.They were in priority order, like Asimov’s Three Laws of Robotics – #1 is higher priority than #2, for example. I told people “when in doubt as to how to proceed on a given issue – always refer back to the Standing Orders for guidance.”
First, make it happen. Your job is to be active and to get stuff out the door, and to help the developers accomplish their goals, NOT to be the “Lurker at the Threshold” and block attempts at accomplishing business value. Sure, we were often in the position of having to convince teams to balance performance, availability, and security against whatever brilliant idea they pooped out that day, but priority #1 was to find ways to make things happen, not find ways to make them not happen – which seems to be the approach of many a sysadmin. In the end, we’re all here to get things done and create maximal business value, and though there is the rare time when ‘don’t do anything’ is the path to that – I would be willing to say that 99% of the time, it’s not.
Second, don’t fuck up. All of Turnbull’s mainframe-guy points elaborate on this core value. (I imagine most of the friction between him and them was that they didn’t share Standing Order 1 as a core value.) As a Web operations person, you have a lot of rope to hang everyone with – you control the running system. If you decide to cowboy something without testing it in dev, that’s bad. For right or wrong, developers mess up code all the time but operations folks are expected to be perfect and not make mistakes that affect production. So be perfect. Test, double check, use the electronic equivalent of OSHA ‘red tags”… Be careful.
And finally, there’s the right way, the wrong way, and the standard way. (Unstated but implied is “do it the standard way.”) Innovation is great, but any innovation (a “better” or “more right” way) has to be folded back via documentation, automation, etc. to become the standard way. If there’s a documented process on how to build a system, you follow that process to the letter, I don’t care how experienced you are, or if your way is “better” for whatever your personal definition of “better” is. If the process seems “wrong,” then either a) there’s some subtle reason it’s done that way you need to figure out, or b) congratulations you can improve the standard procedure everyone uses and you’ve bettered the human race.
As a side note, there are explicit standards and implicit standards. If you go into a directory and see a lot of rolled log files called “file.YYYYMMDD”, and you decide to manually make one, for God’s sake don’t call it “file.MM-DD-YY.” You never know what automated process may be consuming that, or at least you’re pissing someone off later when they do a “ls”. Have standardization as a core value in your work.
If I were to sum them up in one word, the three item list basically reduces to three core values:
But of course it’s not Web Ops if you don’t have some salty language in there. And I fended off attempts over the years to add various options as a #4, mainly from Peco. “4. Profit” as an homage to the Underpants Gnomes was probably the leading contender.
the agile admin 