From the “sad but true” files comes an extremely insightful point apparently discussed over beer by the UK devops crew recently – that we are talking about dev and ops collaboration but the current state of collaboration among ops teams is pretty crappy.
- Internal Borders by Graham Bleach
- DevOps is a good cause, but what about OpsOps? by the Build Doctor
- DevOps, SecOps, DBAOps, NetOps by Kris Buytaert
This resonates deeply with me. I’ve seen that problem in spades. I think in general that a lot of the discussion about the agile ops space is too simplistic in that it seems tuned to organizations of “five guys, three of whom are coders and two of whom are operations” and there’s no differentiation. In real life, there’s often larger orgs and a lot of differentiation that causes various collaboration challenges. Some people refer to this as Web vs Enterprise, but I don’t think that’s strictly true; once your Web shop grows from 5 guys to 200 it runs afoul of this too – it’s a simple scalability and organizational engineering problem.
As an aside, I don’t even like the “Ops” term – a sysadmin team can split into subgroups that do systems engineering, release management, and operational support… Just saying “Ops” seems to me to create implications of not being a partner in the initial design and development of the overall system/app/service/site/whatever you want to call it.
Here, we have a large Infrastructure department. Originally, it was completely siloed by technology verticals, and there’s a lot of subgroups. Network, UNIX, Windows, DBA, Lotus Notes, Telecom, Storage, Data Center… Some ten plus years ago when the company launched their Web site in earnest, they quickly realized that wasn’t going to work out. You had the buck-passing behavior described in the blog posts above that made issues impossible to solve in a timely fashion, plus it made collaboration with devs/business nearly impossible. Not only did you need like 8 admins to come involve themselves in your project, but they did not speak similar enough languages – you’d have some crusty UNIX admin yelling “WHAT ABOUT THE INODES” until the business analyst started to cry.
But are our developers here better off? They are siloed by business unit. Just among the Web developers there’s the eCommerce developers, eCRM, Product Advisors, Community, Support, Content Management… On the one hand, they are able to be very agile in creating solutions inside their specific niche. On the other hand, they are all working within the same system environment, and they don’t always stay on the same page in terms of what technologies they are using. “Well, I’m sure THAT team bought a lovely million dollar CMS, but we’re going to buy our own different million dollar CMS. No, you don’t get more admin resource.” Over time, they tried to produce architecture groups and other cross-team initiatives to try to rein in the craziness, with mixed but overall positive results.
Plugging the Dike
What we did was create a Web Administration group (Web Ops, whatever you want to call it) that was holistically responsible for Web site uptime, performance, and security. Running that team was my previous gig, did it for five years. That group was more horizontally focused and would serve as an interface to the various technology verticals; it worked closely with developers in system design during development, coordinated the release process, and involved devs in troubleshooting during the production phase.
In fact, we didn’t just partner with the developers – we partnered with the business owners of our Web site too, instead of tolerating the old model of “Business collaborates with the developers, who then come and tell ops what to do.” This was a remarkably easy sell really. The company lost money every minute the Web site was down, and it was clear that the dev silos weren’t going to be able to fix that any more than the ops silos were. So we quickly got a seat at the same table.
This was a huge success. To this day, our director of Web Marketing is one of the biggest advocates of the Web operations team. Since then, other application administration (our word for this cross-disciplinary ops) teams have formed along the same model. The DevOps collaboration has been good overall – with certain stresses coming from the Web Ops team’s role as gatekeeper and process enforcement. Ironically, the biggest issues and worst relationships were within Infrastructure between the ops teams!
OpsOps – The Fly In The Ointment
The ops team silos haven’t gone down quietly. To this day the head DBA still says “I don’t see a good reason for you guys [WebOps] to exist.” I think there’s a common “a thing is just the sum of its parts” mindset among admins for whatever reason. There are also turf wars arising from the technology silo division and the blurring of technology lines by modern tech. I tried again and again to pitch “collaborative system administration.” But the default sysadmin behavior is to say “these systems are mine and I have root on them. Those are your systems and you have root on them. Stay on your side of the line and I’ll stay on mine.”
Fun specific Catch-22 situations we found ourselves in:
- Buying a monitoring tool that correlates events across all the different tiers to help root-cause production problems – but the DBAs refusing to allow it on “their” databases.
- Buying a hardware load balancer – we were going to manage it, not the network team, and it wasn’t a UNIX or Windows server, so we couldn’t get anyone to rack and jack it (and of course we weren’t allowed to because “Why would a webops person need server room access, that’s what the other teams are for”).
Some of the problem is just attitude, pure and simple. We had problems even with collaboration inside the various ops teams! We’d work with one DBA to design a system and then later need to get support from another DBA, who would gripe that “no one told/consulted them!” Part of the value of the agile principles that “DevOps” tries to distill is just a generic “get it into your damn head you need to be communicating and working together and that needs to be your default mode of operation.” I think it’s great to harp on that message because it’s little understood among ops. For every dev group that deliberately ostracizes their ops team, there’s two ops teams who don’t think they need to talk to the devs – in the end, it’s mostly our fault.
Part of the problem is organizational. I also believe (and ITIL, I think, agrees with me) that the technology-silo model has outlived its usefulness. I’d like to see admin teams organized by service area with integral DBAs, OS admins, etc. But people are scared of this for a couple reasons. One is that those admins might do things differently from area to area (the same problem we have with our devs) – this could be mitigated by “same tech” cross-org standards/discussions. The other is that this model is not the cheapest. You can squeeze every last penny out if you only have 4 Windows admins and they’re shared by 8 functional areas. Of course, you are cutting off your nose to spite your face because you lose lots more in abandoned agility, but frankly corporate finance rules (minimize G&A spending) are a powerful driver here.
If nothing else, there’s not “one right organization” – I’d be tempted to reorg everyone from verticals into horizontals, let that run for 5 years, and then reorg back the other way, just to keep the stratification from setting in.
Specialist vs Generalist
One other issue. The Web Ops team we created required us to hire generalists – but generalists that knew their stuff in a lot of different areas. It became very hard to hire for that position and training took months before someone was at all effective. Being a generalist doesn’t scale well. Specialization is inevitable and, indeed, desirable (as I think pretty much anything in the history of anything demonstrates). You can mitigate that with some cross-training and having people be generalists in some areas, but in the end, once you get past that “three devs, two ops, that’s the company” model, specialization is needed.
That’s why I think one of the common definitions of DevOps – all ops folks learning to be developers or vice versa – is fundamentally flawed. It’s not sustainable. You either need to hire all expensive superstars that can be good at both, or you hire people that suck at both.
What you do is have people with varying mixes. In my current team we have a continuum of pure ops people, ops folks doing light dev, devs doing light ops, and pure devs. It’s good to have some folks who are generalizing and some who are specializing. It’s not specializing that is bad, it’s specialists who don’t collaborate that are bad.
So I’ve shared a lot of experiences and opinions above but I’m not sure I have a brilliant solution to the problem. I do think we need to recognize that Ops/Ops collaboration is an issue that arises with scale and one potentially even harder to overcome than Dev/Ops collaboration. I do think stressing collaboration as a value and trying to break down organizational silos may help. I’d be happy to hear other folks’ experiences and thoughts!
6 responses to “Before DevOps, Don’t You Need OpsOps?”
One way to deal with the verticals problem is to pull together task-specific teams from staff in the existing silos, sometimes as a permanent assignment, sometimes as a temporary assignment. As I read your description of how your Web Administration team was formed, I could visualize the hackles rising on managers from other verticals with which it overlapped.
If you build teams to deal with your new tasks out of staff from existing units it both allows their managers to feel involved, if only peripherally, and therefore less threatened; it solves some of the access problems you ran into because you are using staff who already have rights to those server rooms, databases, etc because that’s their ‘day job;’ and if you do it frequently enough and rotate the posts, it’s an excellent way to get everyone in the business familiar with the concepts and in the habit of communicating and cooperating first. You break down some of those entrenched fiefdoms, and you do it in a way that is not so explicitly threatening to the current powers-that-be.
All of this really speaks to the necessity, at some point, of getting management on board. You can only go so far taking these techniques to heart in any company before you start running into significant organizational issues, and no amount of discussion and goodwill between the guys on the ground is going to overcome those systemic issues.
Yeah… I’ve never seen a matrixed org like that work well though. “A man can’t serve two masters.” We tried various riffs on it – there would be dedicated “Web reps” on the DBA and UNIX teams for example – but they were always spread too thin and would get quickly consumed by whatever non-Web problem came around. In my opinion a scheme like that can at best provide decent support, but won’t be able to effectively innovate and create a better environment. It’d be better than nothing but probably not better than a dedicated org (which people could of course rotate on to/off of as part of normal internal job changes).
There’s a couple arguments against organizing by service rather than by technology, but I don’t really find them compelling. I think systems orgs everywhere are still organized by technology mostly because low expectations have not challenged them to break out of organizational habits from 50 years ago.
There are certainly some trade-offs to that organizational style; I’ve seen it work extraordinarily well, and very poorly. As with most things, I suspect it’s mostly dependent on culture. One serious hurdle is that, outside a few sectors where it is the norm, almost no one has done it, so it’s difficult to find people capable of pulling it together effectively. But why should that stop us? Every new technology we adopt presents the same challenge at first.
Which is more or less the same factor that, IMHO, makes dedicated organizations less than ideal. They can work well for a time, while the intersection of technology and business need happens to coincide, but eventually they turn into their own sort of silo with most of the problems that you have identified in the existing technology silos.
I think you already identified the primary argument against organizing by service, which is economic; the good news is that I think it’s going to be getting easier to deliver at the service, rather than the technology level. When you can black box your services to the same extent as technologies (when “a web service is a web service” becomes as true as “a Windows machine is a Windows machine”) then it’s just as cost-effective for the IT department to organize at the service level as the technology level.
Well, that’s a fair point. I think the cost could be mitigated by doing “coarse grained” business/service alignment so you’d have three parallel infra orgs instead of 12… But yes, everything eventually needs to change (our dev orgs are consolidating over time and spawning larger-scope architecture and support-programmer groups to avoid their own fine grained silo problems).
Good point about how it’s getting easier to deliver services though – one of the key things we used to ingratiate ourselves to the business was the functionality we could deliver “without programmers” – there are a lot of solutions nowadays that require only configuration and setup but no “lines of code” and though a programming group can certainly “take charge” of them, they’re not a more appropriate choice per se than a sufficiently business-focused ops organization.
Pingback: Our First DevOps Implementation | Web Admin Blog
Pingback: Our First DevOps Implementation « the agile admin