All three Agile Admins were at this Austin event last Saturday; GeekAustin had a set of back to back presentations on puppet, chef, and bcfg2 downtown at Elysium. A good crowd was there, maybe 50 people.
Matt Ray presented on chef, and has posted his slides. Jeff McCune spoke on Puppet and Sol Jerome spoke on bcfg2, I’ll link their slides here whenever I become aware of them being posted.
I had to leave before bcfg2, but the chef and puppet presentations were interesting. One of the main audience comments was “these are pretty similar,” which is true. Both are real good.
Probably the one problem with the event was that it wasn’t really DevOps. It was just tool presentations for sysadmins. Our developers took one look and didn’t come; we met one or two developers at the event that weren’t getting a lot out of it.
It’s fine to have an event looking at these tools – but I want to caution the DevOps community that the value of DevOps is the different approach to life. I think the excellent dev2ops post on “DevOps is not a technology problem, DevOps is a business problem” explains this perfectly. PEOPLE over PROCESS over TOOLS. If you don’t have engagement from the developers, you don’t have DevOps, no matter what whizbang gadgets you have. Frankly I’d like to see the tool vendors address this in their presentations. A little bit of “you’re a sysadmin, this saves you work, isn’t it cool” is fine but how about talking about how this can be used to collaborate with the developers? How does that model work?
At DevOpsDays US I warned about adopting new tools without adopting new tactics, using the example of the deployment of the Minie ball to muskets used in the American Civil War. Greater accuracy and range, but everyone still just stood their in lines and charged in formation, and the slaughter was profound. You may have a shiny new weapon, but if you are just using it back in your dark sysadmin cave, the problems that beset you will never go away. You’ll still be the bottom of the food chain, only taken seriously when someone can’t get their email.
I know the guys at Puppet Labs, Opscode, etc. are all big into DevOps – but I guess I’d like to see the tool presentations and value props speak to developers as well somehow. What would that look like? Ideas?
It’s not a mandatory part of DevOps, but I believe that DevOps works a lot better if operations teams adopt Agile. But all that most systems teams know about Agile is that “it’s that thing that makes the development teams not plan worth a damn any more.” Well, though there may be some truth to that, a well run agile process is very effective and not uncontrolled at all. Constructing and maintaining infrastructure in an agile manner is very possible – we’ve done it. In the beginning it seems daunting, just as it did initially to the legions of software developers who were steeped in waterfall based thinking, but once you adopt it I think you’ll see a lot of traditional pain points get a lot better – that was our experience. This is the first in a series of posts about using Scrum for Web operations, and I thought I’d start with explaining what Scrum is from an ops guy’s point of view.
While developing our first two products that we delivered on the cloud using DevOps, we used an “agile-ish” methodology, which is to say not a formal agile approach. This time, we’ve decided to run Scrum by the book, not just for the feature developers but for our systems engineers, operations staff, security engineers, and system automation developers. Some folks are talking about hybrid models like Scrumban (Scrum + Kanban/Lean) to better incorporate ops work, but we’re going to start with Scrum first and see where we get to.
As a system administrator, that’s scary, because we don’t know Scrum. But Scrum is pretty darn simple.
Here’s two short videos that give a pretty good intro to Scrum.
(P. S. Scrum master lady from this video… Call me!)
But if like me you think videos are not an efficient method of conveying information, here’s the short form.
The Team
In Scrum, you form a small crossfunctional team to all work together on a product instead of having to cross organizational boundaries and fill out forms to get any work done. The roles consist of:
product owner – the “business guy” who has say over what features get greenlit and when
scrum master – the project manager who keeps everyone on track
team – ideally 5 to 7 developers writing code (this is where Ops will plug in, in DevOps)
testers, security, other folks – probably should be included too, but adoption of that varies
Of course in a mid to large sized organization there are other stakeholders, like customers and management and legal and whatnot. But you form a team that has everything it needs to complete its work.
The Backlog
All needed features are brainstormed and put into a master list of features called a “product backlog.” This backlog contains everything – including your systems tasks. The backlog is then broken up into smaller chunks, like release backlogs (features targeted for a specific release) and sprint backlogs (specific tasks for a sprint). The sprint backlog is basically your work breakdown structure, if you’re more comfortable with that terminology, except that the tasks are worded more in terms of what feature they provide instead of in terms of what specific things you need to do; this fosters communication with the product owner.
The developers (and, ideally, operations staff) on the team help generate the backlog and provide time estimates in hours for each task. The product owner owns the prioritization and ordering (as constrained by things that are actual dependencies).
The Sprint
Work is performed in month-long iterations called “sprints.” Requirements are frozen for the sprint and the development is time-boxed – it must end at the termination of the sprint. At the end of the sprint, whatever features were put in that sprint should be complete and “ready to ship.”
The Standup
As the scrum, or sprint, progresses, there are daily “standups” – 15 minute meetings where everyone stands up, reports what they have done since the last standup, what they plan to accomplish by the next standup, and any roadblocks they are encountering. By keeping this meeting short it doesn’t waste time, but by having an actual face to face meeting you get very rapid and effective collaboration that cannot be achieved via managing project plans, sending emails, generating status reports, or the like. It cuts out all the busywork and keeps the kernel of coordination that lets a team keep up velocity.
The Burndown
The progress of the team against the sprint backlog is tracked by a “burndown chart.” As each team member completes their tasks for the sprint you can easily see whether you are on track for successful completion or not.
And that’s Scrum in a nutshell. Five things. Small integrated team, backlog, sprints, standups, burndown.
Next Time
I’ll talk about each of these areas more in depth later in the series, as we go through them ourselves as we develop a real product, and explain how a system administrator (aka systems engineer, infrastructure admin, operations ninja) can fit their work into this structure. But first, I will explain why Agile/Scrum is not just “crazy talk.” To a hardened system admin, or really to anyone used to working in a waterfall environment, it is very counterintuitive that this approach doesn’t just degenerate into the IT equivalent of orcs pillaging a city. But agile has several interesting practices that make it work, and should be very interesting to an operations person.
I was just reading this interesting InfoWorld post on The Most Common Turf Wars in IT – very relevant to what we talk about around here. It’s mainly about stakeholders not properly integrated into the product/project/release planning process. Their most common turf wars are so true:
“Ops vs. Dev” – A cry for DevOps. One of the reasons we have this blog.
“Admin on Admin” – And a good bit of the challenge with that is that admins themselves are a bit grumpy by nature and need them a little “OpsOps.” Being chronically marginalized and given a vision of “contain costs!” is to blame, but it means we sysadmins have a little more personal-skills work to do in order to go from BOFH to team collaborator.
“Security vs Everyone” – Bringing security to the DevOps table can be challenging; if there’s one group more elitist and cynical than sysadmins, it’s security professionals. The job obviously requires a little bit of that mindset. But there’s some movement in the security community to understand the need for collaboration in a DevOps style.
“Management vs. Staff” – Well, that’s more of a question for the ages.
There are others, but these are definitely the “big four” in my experience. What do y’all think?
Recently, I delivered a presentation at the Austin Cloud User Group introducing them to Microsoft Azure. I’m a UNIX bigot and have been doing the Amazon Cloud and open source thing, but we are delivering a product via Azure next so our team is learning it. It’s actually quite interesting and has a number of good points; it’s mainly hindered by the Microsoft marketing message trying to pretend it’s all magical fairy dust instead of clearly explaining what it is and what it can do. So if you want to hear what Azure is in straight shooting UNIX admin speak, check it out!
I went to the local Austin annual IT convention, Innotech, a while back. No, it’s not a coincidence that it sounds like the company from Office Space.
It was pretty good, at least for a couple hour visit. It’s somewhat disappointing that more of the Austin-based tech companies don’t show up to recruit if nothing else… All the show floor is little consulting companies and printer vendors, no Zenoss/BazaarVoice/HomeAway/etc. Although there were an interestingly large number of booths around “helping startups” in general –
I went to two sessions. The first was the Beta Summit, where you get 10 minute pitches from some of the hot new Austin startups about what they’re doing.
First up was Matt Curtain of Socialsmack. Yelp/fb/five star ratings are pointless for brands, so they’ve come up with a “props/drops” rating system people can do for them as well as ask questions and rate answers. It’s kinda stack exchangey if there was a “Random Consumer Brands Stack Exchange.” You can think of it as “Bazaarvoice lite.” They did one for Kona Grill in the Domain that got onto the news. Seems like a fine concept, the question is “why would I want to go use it.” Seems not quite focused enough. Like Stack Exchange, maybe a “cars Socialsmack” et al. would have enough focus to bring people?
Chad Ferrell of Recyclematch talked about their site, which matches up things people have and want to recycle with people that want them. It’s “Homeaway for trash.” Or more so than Craigslist, anyway. Seems like a good play into the green space.
Next up was Ricochet Labs! Who hasn’t played Qrank on the iPhone, it’s a sweet game. Fascinatingly, they are not a game company. Rodney Gibbs says they are developing a location based social platform to target verticals and Qrank was just like a demo proof of concept.
They expect that the OS will own “location checkin” eventually, instead of it being something 200 apps all provide. They are a cloud-based SaaS model using a distributed SOA deployment. Next on their plate is Yelp integration, and then they want to add:
Content channels
Offers/redemptions
Platforms
I have to say I love Qrank and these guys seem like they know what they’re doing.
Eric Katerman introduced Hurricane Party, another iPhone app that lets people define ongoing parties for people to come to, it makes little hurricane icons on the map that show magnitude of the party. They hope to parlay it into locations providing group deals. So it’s like a flash mob for partyin’. I put the app on my phone but haven’t gone to a party yet – they only really happen in Austin (I was bored in Houston one day but no luck).
Next up was Workstreamer. They collect/analyze/deliver info on businesses off social media and whatnot to perform “many to many brand analysis.” Seems like there’s a metric assload of all these “evaluate your brand by grepping twitter” plays, we’ll see which ones excel and survive.
Finally we had the HBMG Vector. I am torn on this. It’s supposed to be a private cloud-in-a-box. The presentation was very 1980s though and it seemed like an old school consulting company that has some frankly not very aligned products.
Then I went to a presentation on “IBM Smart Planet,” as it seems relevant to what we do at NI. The premise is that the world is becoming “Instrumented, interconnected, intelligent.” He talked about partners like Johnson Controls, Eaton, and Siemens in doing this, and noted that just the average building nowadays is kicking out a lot of data. I agree with all this but there weren’t many good takeaways or new insights.
Two of the Agile Admins, Peco and Ernest, are at the new Strata conference in San Jose this week. It’s about “Data Science” and “Big Data” – the confluence of the NoSQL movement, cloud computing, and the Petabyte Age. We now have the ability to gather more data than ever before, and even process it effectively, and this will be transformative to business and society.
We’ll be bringing you interesting things we find out from the conference, inasmuch as the shaky wireless allows.
Yesterday, we attended a variety of tutorials, and I’m sitting in the keynotes right now on the first day of the “main” conference. You can follow along with the keynotes at strataconf.com/live and most presenters are getting their slides and materials up on the site as well. It’s been good so far, stand by for more!
As a Web ops guy, I’ve used the Stack Exchange sites, especially Server Fault, a lot. They’re the best place to go do technical Q&A without having to go immerse yourself in a specific piece of open source software and determine what bizarre way you’re supposed to get support on it (often a crufty forum, mailing list with bizarre culture, or an IRC channel).
However, they have started to, in my opinion, come apart at the seams. They started with the “holy trinity” of Stack Overflow (coders), Server Fault (admins), and Super User (users). But lately they have expanded their scope to sites for non-techie areas, but have also started to fragment the technical areas. So now if I have a question, I am confronted with separate communities for Server Fault, Linux & UNIX, Ubuntu, and more. Or even worse, Stack Exchange vs. “Programmers” vs language specific lists. This basically heavily segments the population and leads to the same problems that the weird little insular mailing lists have. It makes me use the SEs a lot less. I don’t want to have to somehow engage with 10 different communities to answer my everyday questions (and I sure as hell am not going to follow 10 to answer questions), so in my opinion they are cannibalizing their success and it will implode of its own weight and become no better than any Internet forum site.
Recently I started seeing tweets about Quora from @scobleizer, and it said stuff like “is it the future of blogging?” and was being pitched as some twitter-blog hybrid which of course caused me to ignore it. But then it started getting a lot more activity and I thought I’d go check it out. But if you go to the Quora page, you can’t see anything without logging in. And of course if you log in with Twitter or Facebook it wants “everything from you and your friends, ever.” So I wandered off again.
Finally I gave in and went over and logged in, and it’s actually pretty neat – it’s Q&A, like Stack Exchange, but instead of segmentation into different sub-communities, it uses the typical tag/follow/etc. Web 2.0 paradigm. So “Stack Exchange plus Twitter” is probably the best analogy. Now on the one hand that more unmanaged approach runs the risk of becoming like “Yahoo! Answers” – utter crap, full of unanswered questions and spammers and psychos – but on the other hand, I like my topics not being pushed down into little boxes where you can’t get an answer without mastering the arcane rules of that community (like the hateful Cygwin mailing list, where the majority of new posters are chased off with bizarre acronyms telling them they are using email wrong). The simple addition of up/down voting is 80% of the value of what SE gives over forums, so will that carry the day?
Now maybe it’s because they’re having capacity problems, but the biggest problem with Quora IMO is that you don’t get to see any of it when you go there until you log in and give them access to all your networks and whatnot, which I find obnoxious. But if they fix that, then I think given the harmful direction SE is going, it may be the next big answer for Q&A.
Hey, looks like I got quoted in Adrian Cole’s new “State of DevOps” presentation. Some quick thoughts on current state and futures of DevOps from a bunch of important DevOps people and then little old me.
DevOps: Cleans and Shines Without Harsh Scratching by Julian “The Build Doctor” Simpson is a little bit of history, a little bit of prediction, and a little bit of dirty-sounding British phrases like “Discovering jam at the bunfight.”
DevOps: These Soft Parts by John Allspaw reminds us that the core “soft” skills of communication and collaboration lie at the heart of DevOps practice.
The Implications of Infrastructure as Code is the really meaty one, by Stephen Nelson-Smith. This is one of the best of the batch, it’s full of highly specific best practices derived from combining development with operations using Extreme Programming. Take your time and read this – if you are working on implementing DevOps in your shop, this one’s the essay that you need to be taking notes from to build into your own processes. Bang up job!
Props to Matthias Marschall for putting this series together; it’s really great to hear the different takes on this emerging area! There’s two more coming, so I’ll be staying tuned. If you haven’t been following these, please go read them all!
This reminded me of the standing orders I had as manager of our Web Ops shop here for many years. Mine were:
Web Operations Standing Orders
Make it happen
Don’t fuck up
There’s the right way, the wrong way, and the standard way
This was my trenchant way of summing up the proper role of an innovative Web operations person.They were in priority order, like Asimov’s Three Laws of Robotics – #1 is higher priority than #2, for example. I told people “when in doubt as to how to proceed on a given issue – always refer back to the Standing Orders for guidance.”
First, make it happen. Your job is to be active and to get stuff out the door, and to help the developers accomplish their goals, NOT to be the “Lurker at the Threshold” and block attempts at accomplishing business value. Sure, we were often in the position of having to convince teams to balance performance, availability, and security against whatever brilliant idea they pooped out that day, but priority #1 was to find ways to make things happen, not find ways to make them not happen – which seems to be the approach of many a sysadmin. In the end, we’re all here to get things done and create maximal business value, and though there is the rare time when ‘don’t do anything’ is the path to that – I would be willing to say that 99% of the time, it’s not.
Second, don’t fuck up. All of Turnbull’s mainframe-guy points elaborate on this core value. (I imagine most of the friction between him and them was that they didn’t share Standing Order 1 as a core value.) As a Web operations person, you have a lot of rope to hang everyone with – you control the running system. If you decide to cowboy something without testing it in dev, that’s bad. For right or wrong, developers mess up code all the time but operations folks are expected to be perfect and not make mistakes that affect production. So be perfect. Test, double check, use the electronic equivalent of OSHA ‘red tags”… Be careful.
And finally, there’s the right way, the wrong way, and the standard way. (Unstated but implied is “do it the standard way.”) Innovation is great, but any innovation (a “better” or “more right” way) has to be folded back via documentation, automation, etc. to become the standard way. If there’s a documented process on how to build a system, you follow that process to the letter, I don’t care how experienced you are, or if your way is “better” for whatever your personal definition of “better” is. If the process seems “wrong,” then either a) there’s some subtle reason it’s done that way you need to figure out, or b) congratulations you can improve the standard procedure everyone uses and you’ve bettered the human race.
As a side note, there are explicit standards and implicit standards. If you go into a directory and see a lot of rolled log files called “file.YYYYMMDD”, and you decide to manually make one, for God’s sake don’t call it “file.MM-DD-YY.” You never know what automated process may be consuming that, or at least you’re pissing someone off later when they do a “ls”. Have standardization as a core value in your work.
If I were to sum them up in one word, the three item list basically reduces to three core values:
But of course it’s not Web Ops if you don’t have some salty language in there. And I fended off attempts over the years to add various options as a #4, mainly from Peco. “4. Profit” as an homage to the Underpants Gnomes was probably the leading contender.
the agile admin 