Tag Archives: DevOps

SRE: The Biggest Lie Since Kanban

There is a lot of discussion lately about how SRE fits into or competes with or whatever-s with DevOps.  I’m scheduled to speak on a “SRE vs DevOps Smackdown” panel today here at Innotech Austin, and at the exact same time I see Bridget tweeting Liz Fong-Jones’ slides from Velocity on using SRE to implement DevOps. And the more I think about it, and see what people are doing, the more I’m getting worried.

The Big Lie

Just to get the easily provoked to put up their pitchforks, I don’t dislike SRE and I don’t dislike Kanban.  The reason I call Kanban a “Big Lie” is because really doing Kanban correctly and getting the value out of it requires even more discipline that doing something like Scrum.  But it looks so close to doing nothing new that many lazy teams out there say “they’re doing Kanban” and by that they mean they’re doing nothing, but they’ve turned on the Kanban view in JIRA for your convenience.  They have no predictability, they’re not managing WIP, they’re not identifying bottlenecks – they just have a visible board now and that’s it. I strongly believe from my experience that most teams “doing Kanban” are really doing mostly nothing.  There’s articles on this blog about how I make my teams I’m teaching Agile do Scrum first if they want to get to Kanban to build up the required discipline.  And I’m not just a crank, David Hawks from Agile Velocity just told our management team the same thing yesterday, which brought this back to mind for me and spurred this article.

Because I’m starting to see the same thing with SRE.  It’s not surprising – there was and is plenty of “DevOps-washing” of existing teams out there.  Rename your ops team DevOps, done. Well, at least DevOps was able to say “it’s a methodology not a job description or group name stop it” to force deeper thought – it’s why my team at work is the “Engineering Operations” team not “DevOps”, Lee Thompson insisted on that when he set it up! But SRE – yeah, it’s a team just like your own ops team, from an “org chart” viewpoint it looks the same. So doing SRE can – and in many shops does – mean doing nothing new. You just call your existing ops team SRE and figure you’re done.

A brief personal history lesson – my last job before DevOps hit was running the Web Systems team at National Instruments, an ops team.  That’s where we agile admins met, Peco and James were both ops engineers on that team! (Karthik was a dev we worked with.) We had smart people and did ops all right.  We had automation, monitoring, we had “definition of done” standards for new services. You wouldn’t have to squint too hard to just call that team a SRE team and call it a day. But, I wouldn’t wish that job on my worst enemy. It was brutal trying to do ops for just 4-5 dev teams, and that’s with business support, some shared goals, and so on. Our quality of life was terrible, we weren’t empowered, and no matter how hard we tried, success was always right out of our grasp. When we actually started a team using DevOps thinking at NI after that, the difference was night and day, and we actually began to enjoy our jobs as ops engineers. I would hate for anyone to deceive themselves into thinking they’re getting the goodness they should be able to get from a DevOps/”real” SRE approach while still just doing it the way we were doing it.

I have a friend at a local legal software firm, who told me they’re going through and just renaming all the QA folks to SWET (Software Engineer In Test), whether they can code or not, and all the Ops folks to SREs in this manner. One might be charitable and say they’re leaning forward and they intend to loop around and back that up with retraining or something, but… will they? Probably not, it’s just a rename to the hot new term without any of the changes to help those engineers succeed more in their jobs!

SRE isn’t “an implementation of DevOps” if you just apply it as a name for a hopped-up ops team.  Properly understood, it can be an implementation of one of the three parts of DevOps, Infrastructure As Code, Continuous Integration/Deployment, and Site Reliability Engineering. But note that reliability engineering doesn’t start with deploy to production; so much of it is Michael Nygard-esque techniques to write your app reliably in the first place; reliability engineering, in usual DevOps fashion, requires dev and ops work both way back in the dev cycle and out in production to work right. It doesn’t need to be a different team.  If it is, and that team doesn’t get to decide if it takes over ops for a given app, and it’s not allowed to spend 50% of its time on reducing toil and you’re not comping SREs like you do dev engineers – it’s not SRE and you’re a liar for calling it SRE. If you don’t keep DevOps principles in mind, you’re just going to get your old ops team with its old problems again.

That’s why SRE is a Big Lie – because it enables people to say they’re doing a thing that could help their organization succeed, and their dev and ops engineers to have a better career and life while doing so – but not really do it.  Yes, there have been Big Lies before, which is why I cite Kanban as another example – but even if the new criminal is pretty much like the old criminal, you still put their picture up on the post office wall.

Frankly, anyone pushing SRE that doesn’t put warning labels on it is contributing to the problem.  “Well but it mentions in chapter 20 of the second book,” said someone responding to the first version of this article on Twitter.  Not good enough. If something you’re selling is profoundly misused it’s your responsibility to be more up front about the issues.

The Little Problems

Now there are legitimate issues to have even with the “real SRE” model, at least the way that it’s usually being described.  The Google books kinda try to have it both ways, describing it as an engineering practice (how I describe it above and in the SRE course I did for LinkedIn) and describing it as “a team that works this way.”  Even among those not SRE-washing classical ops, the generally understood model is that SRE is a org/job title for a production operations team.

There’s an issue here, the problem of specialization.  If you are Google scale, well, then you’re going to have to specialize and a separate ops team makes sense.  But – first of all, you are not Google scale.  In my opinion, if you are under 100 engineers, you are committing an error by having a separate ops team. You need your product teams to own their products. Second of all – I don’t want to make an enemy of all the lovely Google engineers out there, but is your experience with Google services that they evolve quickly and get better once they go to wide release?  It’s not mine.  They rot.  Have you used Google Hangouts lately without it ending up with cursing and moving off to someone’s Zoom? That kind of specialization still has its downsides in terms of hindering your feedback loops that let you improve (the Second Way). Is SRE just Google-ese for “sustaining?”

I get that the Google folks say they still get feedback and innovation using the SRE model, I’m sure they do and they work hard at that, but that doesn’t change the fact that running a separate ops team is making a deliberate tradeoff between innovation and efficiency. There is no way in that you get as much feedback or improve as quickly with a separate team, you can compensate for it, but you’re still saying “look… Not as important.” Which is fine if that’s your situation, I worked at many companies with 200 abandoned apps in production and you had to do something.  But “not getting there in the first place” is better.

Some of the draw of the model, and why Google is highly aligned with it, is Kubernetes itself. k8s is very complex to run drives people back a little bit to the old priest-in-the-tabernacle model of “someone maintains the infrastructure and you write the app and then you have them deploy it,” but now there’s some standards (like deploying as a container) that make that OK – I guess? But if you think reliability, and observability, are the primary responsibility of an ops team that is not involved in constructing the application, you either have deep and profound company standards that allow seamless plugging of the one into the other or you’re fooling yourself. 90% of you are fooling yourself.

At this conference I heard “Service meshes!  They get you observability so your devs don’t have to think about it.” Do you not see how dangerous that mindset is?

SRE, as interpreted as “a separate newfangled ops team,” may work for some but you need to be realistic about the issues and tradeoffs you’re making.  Consider whether product teams supporting their product, maybe with aid from a platform team making tooling and an enabling/consulting/center of excellence team that can give expert advice?  DevOps helped us see how the “throw it over the wall from dev to ops” model was profoundly harming our industry.  Throwing over the wall from dev to SRE doesn’t improve that, it’s profoundly regressive. Doing SRE “right” to compensate for this, like doing Kanban right, requires more skill and discipline, not less – be realistic about whether you have Google levels of skill and discipline in your org, eh?

Conclusion

SRE (and Kanban) aren’t bad, they have their pros and cons, but they are easy to “pretend to do” in some minimal, cargo cult-ey way that gets you little of the benefits. And if you think spinning up an ops team and calling it SRE is “an implementation of DevOps” you’ve swallowed the worst poison pill the DevOps talk circuit can deal to you.

11 Comments

Filed under DevOps

DevOps Foundations: Lean and Agile

Well you’re in for a treat – we’re getting all of the Agile Admins in on making DevOps courses, and Karthik and I did a course that’s just released today – DevOps Foundations: Lean and Agile.

It’s available both on LinkedIn Learning:
https://www.linkedin.com/learning/devops-foundations-lean-and-agile

and on Lynda.com:
https://www.lynda.com/JIRA-tutorials/DevOps-Foundations-Lean-Agile/622078-2.html

After James and I did DevOps Foundations, the “101” course, we were focused on building out courses for the three major practice areas of DevOps – Continuous Deployment, Infrastructure Automation, and Site Reliability Engineering (in progress now). But our lynda.com content manager said there was interest in us also expanding on the use of Agile and Lean especially as it relates to DevOps.

Karthik is our agile admin Agile expert; he’s presented at several Agile conferences and the like, so he and I decided to take it on.  But how would we bring a DevOps specific take to it?  We started outlining a course and realized it could turn into a giant boring encyclopedia of every Lean and Agile term ever. Most of what we have to add isn’t reading definitions, it’s sharing our experiences actually doing this (my Scrum for Operations series on this blog is perennially popular).

So we decided to take a tip from both Eliahu Goldratt’s The Goal and Gene Kim’s The Phoenix Project by framing the course as a fictional story!  By stitching a narrative together of a Lean, Agile, DevOps transformation of a hypothetical company out of our real world stories from a variety of implementations, we figured we could explain the concepts in context and make them more interesting.  Let us know what you think!

Lynda Course Description:

By applying lean and agile principles, engineering teams can deliver better systems and better business outcomes—both of which are crucial to the success of DevOps. In this course, instructors Ernest Mueller and Karthik Gaekwad discuss the theories, techniques, and benefits of agile and lean. Learn how they can be applied to operations teams to create a more effective flow from development into operations and accelerate your path of “concept to cash.” In addition to key concepts, you can hear in-the-trenches examples of implementing lean and agile in real-world software organizations.

Topics Include:
What is agile?
What is lean?
Measuring success
Learning and adapting
Building a culture of metrics
Continuous learning
Advanced concepts

Duration:
1h 26m

Leave a comment

Filed under Agile

But How Can IT Do DevOps?

img_2455

That’s how!

While James and I were filming our lynda.com course, DevOps Fundamentals, I was impressed by this LinkedIn tech vending machine in their offices.  Swipe badge, get doodad.  I had just come off two different jobs where it was more like “pester IT for the thing you need, wait a couple weeks, maybe someone will drop by with one.”

It’s not “Chef or Puppet,” but it is self service automation!  Think outside the box.  Or in this case, I guess, inside a box. Kudos to LinkedIn IT for a user friendly solution.

2 Comments

Filed under DevOps

Loose Lips Sink Ships: Precision in Language

loose_lips_might_sink_shipsWe talk a lot about communication in the DevOps space.  Often the discussion is about how to “talk nice” and be collaborative, but I wanted to take a minute to talk about something even more profound – the precision of your communication in the first place.

I was reminded of this because over the holidays I’ve been doing some nonfiction reading – first, Thomas J. Cutler’s The Battle of Leyte Gulf, and after that, Cornelius Ryan’s classic A Bridge Too Far.

Both books are rife with examples of how imprecise language – often at the highest levels – caused terrible errors and loss of life.

The example that I’ll unpack is the infamous (to WWII wonks, at least) example of Admiral “Bull” Halsey’s naval charge northward leaving the landing forces at Leyte Gulf completely unprotected. He had previously sent a message indicating his intent to split his forces from three into four task forces. When he later sent a message that he was proceeding north with “three task forces” to pursue Japanese aircraft carriers, Admiral Nimitz and other commanders assumed that the fourth was being left behind to defend the landing forces.  But this was not the case – he had meant “I plan to reform into four groups, you know, sometime in the future, but not now.” By the time the others began to suss out that something was wrong, it was too late – Halsey was too far north to return in time when a large Japanese fleet came upon the much lighter landing forces and forced them into a lengthy fighting retreat. Many American ships sacrificed themselves against an overwhelming opponent to buy time for the doomed task force, until for reasons still not made clear to posterity, Japanese Admiral Kurita broke off the attack for no good reason. (Extensive historical analysis hasn’t clearly determined whether he had bad intel, suffered from battle fatigue, or simply lost his nerve. The after action report of Admiral Sprague of the landing forces simply stated that “The failure of the enemy… to wipe out all vessels of this task unit can be attributed to our successful smoke-screen, our torpedo counterattack, continuous harassment of the enemy by bomb, torpedo, and strafing air attacks, timely maneuvers, and the definite partiality of Almighty God.”)  A happier ending than it could have been, but still, ships sank and men died due to a lack of clear and precise communication.

The same kind of miscommunication happens at work all the time.  A recent real example – I was incident commander on a production incident where a customer-facing server couldn’t be connected to by a support server used to perform interactive customer debugging sessions. It was being coordinated through HipChat. One engineer was looking at the problem from the the customer-facing server.  One of the engineers who administers the support server came on to chat and a discussion ensued. The support engineer indicated that there were hundreds of interactive customer support sessions in progress on that support server. Another senior engineer, an expert on the customer servers, declared, “Well, let’s try rebooting the server.  It’s the only thing to do.” “Really?” says the support engineer?  “Yes, let’s do it,” says the senior engineer.

Of course he meant the customer-facing server, which we knew didn’t have any traffic at the moment, and not the support server.  But the support engineer assumed he meant the support server, and proceeded to begin to reboot it – at this point I intervened and had to say “STOP, HE IS NOT TALKING ABOUT YOUR SERVER.” But it was a reasonable mistake – if you talk about “the server,” then everyone is going to interpret that as “the server most important to me at this time.” Anyway, we stopped the support engineer from disrupting 100 customer support sessions and we restarted the other server instead. (Which didn’t help of course, “reboot the server” is something that went out with Windows NT, it’s a desperate attempt to not perform quality troubleshooting – but that’s material for another post.)

If at any time you find yourself talking to someone else about something there’s clearly more than one of:

  • “the” server – or even “the” support server or “the” bastion server
  • “the” incident/problem
  • “that” ticket
  • “that” email I sent you
  • “the” bug
  • “the” build

You are being a dumbass.  I can’t be blunt enough about this.  You are deliberately courting miscommunication that in the best case adds time and friction and in the worst case causes major errors to be made.  I have seen costly errors happen because of real world exchanges just like this:

“Are you working on the problem?”  “Yes.”  (And by this they mean the other problem the asker doesn’t know about.)

“It’s critical we fix that big bug from this morning!” “Yes, someone’s on it!” (And by this they mean the bug that got ranked as high criticality in the backlog, but the asker means their own special favorite bug.)

“Is the server back up?”  “Yes.” “OK I will now begin a lengthy operation… Hey what the hell?!?”  (Oh, you meant another one of the 5 servers being restarted this hour.)

Everyone in an environment should know, in a sober moment, that there are often multiple incidents going on in a day (and even at the same time), that there are multiple servers and multiple tickets (and apps and environments and builds and developers and customers…).

If you hit me in chat and say “the build broke what could be wrong” – and our build server has somewhere around 100 builds in it, all of which are breaking and being fixed continuously by a large group of developers, at best you are being super inconsiderate and forcing me to do diagnostic work that could be forestalled by you cut-and-pasting an URL or whatnot into your browser, and potentially I’m going to go look at some other build and you get help much later than you wanted it.

And that, if you’ve sent me 10 emails that morning, asking me “if you got that last email I sent you” is pointless verbal masturbation of an excessively time-wasting variety.  Either I’m going to assume and potentially give you the wrong answer, or I have to spend time trying to elicit identifying information from you.

Be Specific

I expect crisp and exact communication out of my engineers, and it’s as simple as hearkening back to grade school and using proper nouns.

You are working on bug BUG-123, “Subject line of the bug.”   You are logged into server i-123456 as ec2-user. You are working on incident I-23, “Customer X’s system is down.” You sent me an email around 10 AM with the subject line “Emergency audit response needed.” SAY THAT.

Being in chat vs email vs in person/phone/video call is not an excuse.  This is not a “tone formality” issue, it is a precision and correctness issue.

Having people with different primary languages is not an excuse. Unless you’re from El-Adrel, your language has proper nouns and you know how to use them, and you know which server you’re logged into.  (Well, OK, you might not, but that’s a different problem.) Slang is a language problem, other things are a language problem, but being able to clearly state the proper noun at hand is NOT a language problem.  Proper nouns are the one thing that never need translating!

Detect Fuzzy Language And Make It Specific

But what if you’re not the one initiating the discussion?  What if someone else is already giving you unclear direction or questions?  Well, I’m going to let you in on a little secret – managers are often terrible at this.  Like with Admiral Halsey, sometimes being “hard charging” gets confused with “lack of clarity.” Sometimes it seems like the higher level of management someone’s in, the more they think that the general rules of discourse and process don’t apply to them. You must not be “in tune with the business needs” if you aren’t sure exactly which thing they are vaguely referring to. Frustrating, but you just need to manage up by starting to be precise yourself. Don’t compound the possible error by making an assumption; if there’s not a proper noun in play you need to state one yourself to ensure precision.

“Customer X is angry are you working on that bug?!?”  Obviously “yes” or “no” is probably an answer that, if you’re not on the same page already, will continue to cause confusion. Reply “I am working on bug BUG-123, ‘Subject line of the bug.’  Is that the one  you’re talking about?”

“Did you get that last email I expect a response!” says the call or text.  Well, just saying “yes” because I saw an email from that guy an hour ago I just responded to isn’t right –  it might not be the right one. Get clarification.

Whatever happens, try to verify information you’re being given.  “Reboot the server” – “To confirm, you want me to reboot server i-123436, the customer support server, right?”

Ask fast.  In the example above, Nimitz and the other commanders just let the imprecision slide, thinking “Well…  I’m sure he meant the fourth task group is staying…  I don’t want to look dumb by asking or make him look bad by having to clarify.  It’ll all be fine.”  and by the time they got nervous enough to ask him directly it was too late for him to return in time and people died as a result. Now sometimes issues are timely and the person sending the request/demand doesn’t respond to your request for clarification in a timely manner, even if you respond immediately.  What do you do?  Well, you have to do what you think is best/safest, but try to get things back on the right path ASAP by using all the communication paths at hand to clarify. If the request was unclear, do not let your response be unclear.  “Restart the server!”  “Done.”  You’re both culpable at this point.”I restarted server i-123456 at 10:50 AM” puts you in the right.

Then afterwards, send the person this article so they can understand how important precision is.  Luckily, in IT it is usually less likely to cause deaths than in the military, but poor communication can and has lost many dollars, hours, and jobs.

6 Comments

Filed under DevOps

DevOps Fundamentals Going Strong!

Our LinkedIn Learning content manager, Jeff Kellum, tells James and I that our DevOps Fundamentals course is “the third most popular IT course in our library right now”!  You can start a free trial period at Lynda by going to http://www.lynda.com/trial/ErnestMueller.  We’ll post more about the experience we had making the course, it was a lot of fun and we learned a lot about going in front of the camera!

 

Leave a comment

Filed under DevOps

A New Way To Learn DevOps!

IMG_0635.JPG

James Wickett and Ernest Mueller on the set for DevOps Fundamentals

We’re really excited to announce that James Wickett and I (Ernest Mueller) from The Agile Admin have put together a comprehensive DevOps Fundamentals course for lynda.com – a 3 hour long course covering everything from DevOps’ Agile and Lean roots, to DevOps culture to book recommendations and we even cover future .

As you know we here at The Agile Admin have spent a lot of time trying to help people learn DevOps – for a variety of reasons, many of the original DevOps practitioners were reluctant to even define the term, and were against a lot of the “DevOps” training/certification programs that sprung up because they weren’t really a good reflection of the real scope of the movement. While we understand those factors and agree with some of the specific critiques, we think it’s frankly been criminally difficult to learn DevOps with the available resources to date (best answer: go to a variety of events, crawl some random blogs and twitters, try to piece it together yourself over time, read some kinda-related books…).  The unicorns don’t need any more than that, but all four of the Agile Admins have worked in corporate IT before and have a lot of sympathy for all the folks out there that *don’t* work for Etsy or Netflix and are trying to figure out how this new world can make their work and life better.

In the course, we go into what we consider to be the three primary practice areas of DevOps – continuous delivery, infrastructure automation, and reliability engineering. lynda has a free trial period so feel free and go give it a look to see if it could help you!

To give you an idea of what is included in the course, here’s the course outline. Even in a three hour class there’s no way to comprehensively cover these topics, so we tried to extensively point you out at other resources as we go and have a whole section on great DevOps learning resources.

  • Introduction
  • DevOps Basics
    • What Is DevOps?
    • DevOps Core Values: CAMS
    • DevOps Principles: The Three Ways
    • Your DevOps Playbook
    • 10 Practices For DevOps Success (Part 1)
    • 10 Practices for DevOps Success (Part 2)
    • DevOps Tools: The Cart Or The Horse?
  • DevOps: A Culture Problem
    • The IT Crowd and the Coming Storm
    • Use Your Words
    • Do Unto Others
    • Throwing Things Over Walls
    • Kaizen: Continuous Improvement
  • The Building Blocks of DevOps
    • DevOps Building Block: Agile
    • DevOps Building Block: Lean
    • ITIL, ITSM, and the SDLC
  • Infrastructure Automation
    • Infrastructure As Code
    • Golden Image to Foil Ball
    • Immutable Deployment
    • Your Infrastructure Toolchain
  • Continuous Deployment
    • Small + Fast = Better
    • Continuous Integration Practices
    • The Continuous Delivery Pipeline
    • The Role of QA
    • Your CI Toolchain
  • Reliability Engineering
    • Engineering Doesn’t End With Deployment
    • Design for Operation: Theory
    • Design for Operation: Practice
    • Operate for Design: Metrics and Monitoring
    • Operate for Design: Logging
    • Your SRE Toolchain
  • Additional DevOps Resources
    • Unicorns, Horses, and Donkeys, Oh My
    • The 10 Best DevOps Books You Need To Read
    • Navigating the Series of Tubes
  • The Future of DevOps
    • Cloud to Containers to Serverless
    • The Rugged Frontier of DevOps: Security
  • Conclusion
    • Next Steps: Am I a DevOp Now?

We worked long and hard on the course and we think it represents all the must-know aspects of DevOps and can get you started down the path of implementation with a good foundation.  Check it out and let us know what you think!

3 Comments

Filed under DevOps

Three Upcoming DevOps Events You Should Attend

I wanted to mention a couple Austin area events folks should be aware of – and one international one!  November is full of DevOps goodness, so come to some or all of these…

The international one is called All Day DevOps, Tuesday November 15 2016, and is a one long day, AMER and EMEA hours, 3-track, free online conference.  It has all the heavy hitter presenters you’d expect from going to Velocity or a DevOpsDays or whatnot, but streaming free to all.  Sign up and figure out what you want to watch in what slot now!   James, Karthik, and I are curating and hosting the Infrastructure track so, you know, err on that side 🙂  There’s nearly 5000 people signed up already, so it should be lively!

Then there’s CD Summit Austin 2016.  There’s a regional IT conference called Innotech, and devops.com came up with the great idea of running a DevOps event alongside it. It’s Wednesday November 16 (workshops) and Thursday November 17 (conference) in the Austin Convention Center. All four of the Agile Admins will be doing a panel on “The Evolution of Agility” at 11:20 on Thursday so come on out!  It’s cheap, even both days together are like $179.

But before all that – the best little application security convention in Texas (or frankly anywhere for my money) – LASCON is next week!   Tues and Wed Nov 1-2 are workshop days and then Thu-Fri Nov 3-4 are the conference days. I’m doing my Lean Security talk I did at RSA last fall on Friday, and James is speaking on Serverless on Thursday. $299 for the two conference days.

Loads of great stuff for all this month!

 

Leave a comment

Filed under Conferences, DevOps