Author Archives: Ernest Mueller

About Ernest Mueller

Ernest is Engineering Manager at Verica, in Austin, TX. More...

Incident Management Course Coming!

2019-06-13 10.51.48I know we’ve been quiet on the blog, all four agile admins have been busy – several of us moved to new jobs, everyone has a lot going on.

But we’re still doing stuff!  I just went out to Carpenteria to film a LinkedIn Learning course on Incident Management.  The agile admins have a full DevOps curriculum on LinkedIn Learning (which was lynda.com); most of them are in the “Become a DevOps Engineer” learning path!  You can view them as a LIL member or they can be bought individually nowadays too.

We’ve done the 101 level (DevOps Foundation), the 201 level (CI/CD, CM/Infrastructure as Code, SRE, Monitoring and Observability, Lean and Agile) and now we’re hitting more details – Karthik’s done a bunch of Kubernetes and Cloud Native courses, Peco is doing more monitoring courses, James is doing DevSecOps courses…

2019-06-13 12.28.02And I just went and filmed an Incident Management course.  Incident Response, really, I’m hoping for a subsequent course that focuses on retrospectives (each class is only like an hour long and retros are a huge fun topic so I wanted to give them enough time on their own).

Pictured are my producers Adam and Lori and my live action director Julia (who’s also done some of my other courses!) This was a slides course (my first), but they have a program where they can add in a little live action, and since I’ve done it a bunch and Julia’s great we burned through a bunch of scripts in a short time on camera! Thanks to all of them (and my content manager Brian Anderson, not pictured).

The Course

I’ve been creating IM processes and training and leading organizations in them for a while now. A good incident response program removes friction and lets your smart technical staff focus on one thing, solving the problem, without having to worry about what to do otherwise. When I left AlienVault, the #1 thing people came and said to me in my 2 week notice period was “Hey, that incident management process, that’s really made a huge difference,” which is great to hear.

And it was a good opportunity to refresh on the newer developments in the field.  I first got into modern IM, which I defines as “derived from the Incident Command System”, in 2008 after I heard Brent Chapman speak at Velocity on Incident Command for IT: What We Can Learn from the Fire Department.  But (aside from retros) while that concept spread, for 5-6 years there wasn’t really a lot more in terms of new developments. Luckily that’s changed, and there’s been a lot lately. John Allspaw and J. Paul Reed have both done masters’ theses with Lund University’s Division of Risk Management and Societal Safety; there’s a new O’Reilly book Incident Management for Operations as well as IM being a hot topic in the Google SRE books, and so on. The REdeploy conference and Thai Wood’s Resilience Roundup weekly email newsletter and the Oncall Nightmares podcast re full of late breaking developments. (These sources and more are listed in the course handout!)

Special thanks to J. Paul for giving me guidance on the course content and giving me permission to use his and Kevina Finn-Braun’s Incident Lifecycle Model in it.

Expect video topics like:

  • Why Do I Need Incident Management?
  • The Incident Command System
  • Scoping the Problem
  • Your Incident Toolchain
  • Incident Toolchain Example
  • Detecting and Reporting Incidents
  • First Response and Escalation
  • Incident Communication With Your Users
  • Communicating Inside Your Organization
  • Best Practices for Diagnosis and Repair
  • Cleaning Up After
  • Continuously Improving
  • Training and Game Days
  • Implementation Challenges

Oh, and I got to use props for the first time (like that fire extinguisher in the lead pic), we threw some in for kicks. Fun!

The Experience

Speaking of that, I just wanted to give the LinkedIn Learning team a shout-out.  Making courses with them is a great experience, class all the way.  They are all super skilled at what they do and super friendly. Going to their campus/studio in Carpenteria, CA is always an exceedingly pleasant experience. Everything’s top notch, sound booths, live action studios… It’s not the average webcam tech course when you’re looking down the barrel of a camera with a director, a producer, and a sound/teleprompter person fussing over the fine details! If you are an expert in something (not just tech) and are interested in doing courses, I’m happy to introduce you to someone there; it’s all top quality.

And they treat their people well there!  As best as I can tell they always have, from when they were Lynda to when they were LinkedIn to now being owned by Microsoft. Lori confided in me, “I was a documentary filmmaker with a non-profit for years and I didn’t know jobs like this existed; I’ve never been treated so well.”

While I was there they were doing their monthly “InDay”, and apparently this is the most anticipated one of a year as it’s game themed. They had inflatable human foozball, arcade games, did up the cafeteria with a Stranger Things theme, even had a D&D training session.

 

2019-06-13 17.33.21And of course Carpinteria is beautiful, right on the beach, extremely temperate. It’s between Ventura and Santa Barbara, just north of LA. If you go out there, my hot tips are the nearby Shoals restaurant (a little down the 101) where you can get a table right on the water, and Chocolats du CaliBressan, a French chocolatier down in the far north end the beach side of Carpinteria. Oh and the booze is super cheap in the supermarket, so we always make some gin and juice and hang out in the Holiday Inn’s hot tub while we’re there…

 

2 Comments

Filed under DevOps

DevOpsDays Austin 2019 Retrospective

2019-05-02 12.49.54As mentioned, DevOpsDays Austin 2019 went off great!  And after the event, we sent out extensive surveys to attendees, sponsors, volunteers, speakers, and even the organizers to learn and improve. (Thanks to everyone who gave their feedback, we appreciate it!)

Last year we also did an extensive retrospective to figure out how we wanted this year to go, and this year’s event was driven by that feedback and our vision to make DoD Austin the place for practitioners to come, learn from each other, and build the local community.

Let me share this year’s retro with you – some of the numbers and sentiments are below with my thoughts. If you want the full details, sure, here you go!

Full DevOpsDays Austin 2019 Retrospective (pdf)

If you’re not familiar with a NPS score, it’s used to measure sentiment on a scale from -100 to +100.  When you get asked “would you recommend” something on a 1-10 scale, generally they’re taking that number and bucketing it into 1-6 being detractors (counted as negative), 7-8 being neutral, and 9-10 being promoters (counted as positive). Above 0 is “good”, above 50 is “excellent.”  See more about NPS scores here.

Sorry about the quality of the pics, these are basically ones I snapped myself on my iPhone. But hopefully they show some of what happened at the event!

Attendee Feedback (62 NPS, 50 responses)

2019-05-02 09.43.28

Damon Edwards

“Informative, laid back, friendly, humorous event. My favorite conference for a couple of years now.” 84% of attendees said they were likely to return.

The things people liked the most as measured by the freeform comments were the openspaces (9 comments), the speakers/talks, especially their diversity (8 votes), the culture/atmosphere of the event (5 votes), and the community and people (5 votes).

This makes me happy. DevOpsDays isn’t just “a conference,” it really focuses on building community – people meeting each other in a friendly and collaborative environment. The content is nice but it’s not the primary value of the event.

2019-05-02 09.48.15

Mandy Whaley

Concerns people had the most were “Nothing/great job” (10 votes), difficulty with travel and parking at the venue, including handicap access (6 votes), talks (6 votes), we want better lunches (4 votes).

Read on for more but we’re probably changing venues next year and will keep access in mind.  Now on the lunches – we used to have fancy lunches and they were a significant time and effort sink, with long lines, lots of time spent, and so on.  We moved to box lunches and now lunch goes fast and easy and leaves everyone more time to interact with each other.  We do not plan to ever change back from that, but we will see if we can get a BBQ place or something to do a nice lunch box.

(There were more likes and dislikes and we are evaluating action on all of them, but dang this post is going to be long already so I’m focusing on the top line items.)

Speaker Feedback (90 NPS, 10 responses)

2019-05-02 11.10.39

Pete Cheslock

  • “Everyone was really positive; welcoming, low-pressure environment.”
  • Experience – 50% excellent, 50% very good
  • Organization – 40% extremely, 50% very organized
  • Friendliness – 90% extremely, 10% very friendly

Likes: No tech problems/helpful techs/setup organized (x4), Supportive/welcoming (x3), Engaged audience (x3).  Dislikes: Chromebook support problem, schedule slippage, openspaces competing with Conversations talks.

Great overall, some things for us to tweak!  After several years in the same venue and buying a lot of gear, our crack AV team have the tech end of it pretty much down pat.

2019-05-03 15.20.05

Jon Loyens

Organizer Feedback (88 NPS, 8 responses)

  • “Just [wanted] to say how much I enjoy working with the crew and watching it all come together to put on a great event for the community. I get a lot out of doing it each year and see my contribution as an important way to give back.”
  • Time spent – 62.5% just right, 12.5% little long, 12.5% little short, 12.5% way too short
  • 93% likely to return (the one that isn’t pleaded a heavy year at work coming up)

Major likes included working together (x3), inclusion (x2), and the opportunity to give back (x2). Dislikes included some stressing out and looking for problems, and speaker notification happening late. There was good discussion about explaining openspaces more especially for the newer folks.

It’s important to me that our organizers have a good time too – my assigned domain on the organizer team is “Organizers” – besides working the master budget and schedule for folks, I facilitate and try to ensure that this volunteer gig is not onerous, and I’m happy we seem to be there.

2019-05-02 13.33.45

Deborah Hawkins

Volunteer Feedback (94 NPS, 17 responses)

  • Experience: 72.7% excellent, 27.78% very good
  • How much time you spend – 83% about right, 11% too much, 6% too little
  • 93% likely to return

We have a lot of volunteers from the community that come to slave away working the event for a free ticket and a couple meals, basically.  It’s very important to all of us that they have a good experience – these are the future organizers, and community members going above and beyond to give back to the community.  Boyd and Daria and the other organizers did a great job both organizing the work and making sure the volunteers had time to participate in the event and have a good experience – even given the storm-nightmare loadout at the end of the event. Thanks to all our great volunteers!

Sponsor Feedback (60 NPS, 10 responses)

  • “A++ highly recommend, etc. Y’all did a bang-up job putting this together, and the community is certainly a testament to your hard work and continuous efforts. I’ve told everyone at HQ that we need to learn from you.”
  • Experience – 70% excellent, 20% very good, 10% good
  • Liked: “Always a great event – excellent sessions, great opportunities to meet with customers and prospects.” Vendor area good. Friendly people and networking.
  • Disliked: Platinum sponsors were upstairs. Water bottles ran out. We want badge scanners. No day before setup. Only 1 minute blurb. Schedule off track. When will courtesy shipping be picked up.

2019-05-03 09.49.41So… Sponsors. For a number of years we kept expanding our sponsor offerings.  Then we realized the event had become too much of a traditional conference and we were spending lots of space, time, and effort on sponsors, when to be honest we don’t really need all that much money to put on the event.  Two years ago after a bunch of sponsor problems and everyone working themselves to the bone to provide professional conference services I did away with sponsor tables altogether. We let them back this year but really wanted to make the event not about that.  We also warn the sponsors up front this isn’t a “churn the leads” event, we want sponsors who are going to send technical people to engage with the community.

Did it work out that way?  Kinda. There’s too much expectation set up about what “conferences are like” and “DevOpsDays are like” and between the person purchasing the sponsorship and the people actually sent on site there’s a lot of room for expectations to drift.

2019-05-03 14.52.36

Tristan Slominski

I feel like there’s plenty of big conferences for that kind of sponsor engagement.  DevOpsDayses didn’t used to be like that, but as time goes on and they all grow it’s tempting to “improve” by making it more sponsor focused. We love sponsors who engage with the community but we consciously balance their participation in the event.

Funny story… Like I said we only let sponsor tables back on a limited basis this year. But there was a run on them, and we sold out of the ones we needed to fund the event quickly and had a bunch of sponsors still wanting to participate, including ones who had participated for  years. So we extended the sponsor room, just to let them participate, because we felt bad about excluding them. So we always sell out, so that’s probably a sign that we’re doing fine there.

And we got to sponsor a house for the homeless with the spare money, so that’s spiffy.

Recruiter Feedback (-50 NPS, 2 responses)

This is a new addition that didn’t work out so well. We had imagined a big recruiter speed dating thing. But few recruiters and attendees signed up for it so we pivoted into a recruiter fair.  It was during happy hour, but half the attendees leave before that. We had them by the bar, but the DevOps Trivia during the happy hour was also a big draw.

While all the recruiters rated their experience “good” they had low traffic.

So, sorry that didn’t work out. But I stressed to the organizers that this wasn’t a failure – if we don’t try new things that don’t work out sometimes, we’re not trying hard enough.

We’re one of the great grand-daddy DevOps events. We have years of experience, ample funding, and a big community.  Smaller DoDs, especially ones getting off the ground, often need to hew close to the “standard format” for a safe launch and to pay their bills.  We can afford to experiment, so I strongly urge the team every year to try different things.  It’s OK if we appeal to different sets of the community each year.  It’s OK to not do something again (even if it went well) and it’s OK to try new things as stretch goals. I kinda like putting how we run our event where our DevOps mouth is, so to speak.

This lets us try things out first. We were the first DoD with a multi-content track. We created the new “Conversations” talk format this year. We keep innovating, and sometimes there’s just not a fit given the constraints of venue, time, people, and so on. So this one didn’t go off great, but to me that just means we’re legitimately experimenting hard enough.

Ernest’s Retrospective Thoughts

Overall it went great!  Smooth, excellent execution by everyone involved. I feel like the Austin tech community is stronger for our event existing and that’s what I want out of it.

My main challenge personally this year was with the talks.

We really went into this year with an intent to curate the talks to a pretty specific practitioner format. DoD Austin has a bunch of years behind it so we don’t necessarily need the DevOps “talk circuit” talks to fill slots.  We feel like we can be very specific about the experience we want to curate – no repeat talks from other events (go watch them on the Internet, everyone posts videos!), some preference to local speakers, encourage diversity both in speakers and in content…  But we didn’t execute on that well.  We started using Papercall this year and it makes it easy for people to mass submit to multiple events – a great feature but somewhat antithetical to our needs. We had 200 submissions for 20 slots and had a lot of weeding to do and had to turn away a lot of folks. And while we had good talks, they didn’t fit our proposed theme necessarily.

We also just selected talks late, to where it risked people whose talks were declined not being able to attend because we sold out our attendee cap.

The second challenge was with openspaces.  In general the larger the event, the harder it is to make openspaces work. Once there’s more than 25 people in an openspace the format collapses and it’s just “2-3 people talking to each other and everyone else straining to hear,” basically a super crap panel talk. Putting them in the luxury boxes in the stadium worked really well there, because only so many people can fit into one, so it was a forcing function to keep them small enough to work. So they went well overall.

But some folks didn’t like them. Each year we get some feedback from folks more used to traditional content.  “Maybe we should get the openspace topics submitted before the conference so they’re already on the schedule!” No offense, but over my dead body. That’s not what openspaces are about and openspaces are the heart of DevOpsDays. They are for what the actual attendees want to talk about right then; the entire point is that they’re not programmed content. Early DevOpsDays were a couple talks and then pretty much all openspaces.  My general attitude is “if you don’t want to participate in openspaces, this is not the event for you.” We need to explain openspaces more ahead of time though, to seed ideas and get new people to understand the format.  Our experiment with mini-talks and then linked openspaces worked out great, I went to two of them and got high value out of them.

Next Year

A couple big changes are coming next year.

First of all, we’re probably changing venue.  We’ve enjoyed the stadium a lot, and love the staff there, but we’ve probably done as much as we can with the event in that particular form factor.

We’re considering going entirely to the new 20 minute talk format.  They were well received – if you really have more content than 20 minutes, a linked openspace is probably the best venue to explore it with highly engaged attendees!  And it’ll prevent people just submitting their “same talk” as much. We can also get more speakers in!

Also, we know it’s a bummer that we’ve been capping attendance and sponsors and that people who want to attend get turned away. So far we’ve felt like we have had to, both because of venue capacity but also to keep openspaces good and keep the great atmosphere and community and opportunities for engagement that make our event distinct.

Now that we have enough experience, we think we might be able to go bigger and still keep the small group and one-on-one interaction. We’ve all been to a bunch of conferences and seen other things – 1-1 mentoring table signups, for example, and other formats that facilitate it.  We’re also thinking about adding some “working groups” – opportunities to do something, produce position papers, whatnot, give the experts a really neat thing to do at the event.

And maybe even add on a third day, with all unstructured content. On a Saturday so people could bring their kids and stuff.

I wanted to just blaze big next year; the rest of the team loved the vision but reminded me how much burn-in there is on a new venue – getting A/V figured out, all the rough spots of a year one… So we may iterate into it, with getting a new venue and going slightly larger and trying out new engagement ideas next year, and then the year after saying “Big tent!  All are welcome!  Fly in for this one, no attendee or sponsor caps!” and making it a heroically sized event.

There’s no one right format for DevOpsDays – I encourage other organizers to keep experimenting as well.  Your event doesn’t have to be the same year to year; you can target different goals and audiences and sizes and such each time.

If anyone read this far, feel free and comment with your thoughts below! (Obligatory disclaimer, don’t tell me “well this isn’t right for my DevOpsDays” – that’s fine, none of this is to declare the “right” way to do an event, it’s just what is working for us in our community with our particular goals.)

Leave a comment

Filed under Conferences, DevOps

DevOpsDays Austin 2019 Highlights

devops_mascot_texas_color_swapWe held our eighth DevOpsDays Austin last month! DevOpsDays Austin 2019 was held at the UT Austin stadium for two days full of talks, openspaces, and so on. All the videos of the sessions are up on YouTube in the DevOps Austin channel that holds other years’ videos as well.

Here’s my top 10 countdown list of great things about this year’s DevOpsDays Austin!

2019-05-03 09.49.41

Platinum Sponsor Suite

10. We brought the sponsor room back, and added platinum suites in the stadium luxury boxes so sponsors that wanted to hold sessions could do so. There were very well attended sessions in these suites!

9. We had two content tracks and a new “Conversations” talk format – a short 20 minute talk followed by a linked openspace for interactive demos and discussions and command line stuff that doesn’t do well in a talk session. We only had space for a handful of them but they were very highly rated and we’re considering shifting significantly towards them next year.

8. We made the happy hour more modest and onsite, but with DevOps Trivia from Patrick Debois!  We had a bunch of teams compete and it was a wild and woolly time. We even used Patrick’s zender.tv online trivia thing to let people outside the venue compete.

2019-05-03 17.58.58

The remnants of the cupcakes

7. Our fine venue, food, and drink team and vendors… We ripped into some mini cupcakes at snack time!!!

6. The openspaces.  I actually got to attend some this year instead of just running around working.  And they were all brilliant.

5. Our organizers! We bestowed the title of MVP organizer on two organizers this year – Daria Ilic for her great job with communication and Dan Zentgraf for doing a yeoman job with the sponsors.

Special thanks to all the DevOpsDays Austin 2019 organizers: James Wickett (Speakers), Peco Karayanev (Speakers), Karthik Gaekwad (Swag), Daria Ilic (Marketing, Volunteers), Dan Zentgraf (Sponsors), Tom Hall (Sponsors), Boyd Hemphill (Volunteers), Scott Baldwin (Web site), Lee Thompson (AV), Carl Perry (AV), Ian Richardson (Attendees), Chris Casey (Signage and Slides), Richard Boyd (Venue, Food, Happy Hour), Asif Ahmad (Venue, Food, Happy Hour), Bailey Moore (Venue, Food, Happy Hour), and thanks to Laura from ConferenceOps for doing all our finances.

4. I let the other organizers talk me into buying the Jumbotron!  I am naturally thrifty so had resisted given the significant price tag in previous years, but we had a glut of sponsors and everyone really wanted it so I finally gave in. Karthik even changed his Slack name to JUMBOTRON to petition for it. It remains so until this very day. You  have to respect the dedication. So behold – the DevOpsDays Austin Jumbotron! (Yes, that’s real, not Photoshopped.)

2019-05-02 09.46.00

3. Check out our cool organizer swag I got each organizer this year as a thank you gift – custom Vans with the DevOpsDays Austin mascot on them!  (They’re only $80, if a little work intensive to design on their site, feel free and steal the idea!) People always love our DevOpsDays Austin shirts so I wanted to give the organizers a really distinctive way to show their pride in the event.

vans

2019-05-02 09.48.152. A very special thank you to DevOpsDays Austin from Mandy Whaley and the Cisco DevNet crew, who have been sponsors and speakers and attendees for many years.  I wasn’t expecting this – they actually used their sponsor shout-out time to present us onstage with a heartfelt card that they read to the audience.

We appreciate everything that Mandy and the team bring to the event and the card was super touching.2019-05-02-09.49.56.jpg

2019-05-02 09.50.08-1

1. What could be better than that, though, you ask? How can such a kind shout-out be number 2 on the list?

Well, we had a little problem, and that problem was a spare $25,000 from letting in the gold sponsors above our initial sponsor room cap because they really, really wanted in and we felt bad for them. DevOpsDays Austin (like all DoDs) is a non-profit, so while we keep a war chest to pay for next year’s venue and stuff, the rest has to go. Previous years we did some modest donations to the Capitol Area Food Bank; last year we actually had enough spare money so that we let each organizer do a $1000 donation to a charity of their choice. But this was quite a larger chunk, so what to do?

Some of the organizers brought up a great opportunity they knew about and had given to themselves. Here in Austin there’s a really unique program going on, the Community First! Village – a planned community that provides affordable, permanent housing and a supportive community for men and women coming out of chronic homelessness.

mobile-loaves-fishes-community-first-village-microhome-300x200

Community First! Village Micro-Home

And it turns out $25,000 is how much is needed to build a micro-home in their next phase of expansion, to house a formerly homeless person in their community. These are little 180-200 square foot homes with electricity but no plumbing that are the foundation of their village. The whole organizer team got super excited about this opportunity.

So that’s what we did – we sponsored one of these homes to be built. We’re pleased to have the ability to help Austin in a permanent way out of the conference!

I’m going to do a separate blog post on this because it’s an awesome program that many companies in Austin have been getting behind, and it’s remarkably successful in helping our large homeless population. But thanks so much to all the sponsors and attendees that made this possible.

2019-06-08 10.21.02

DoD Austin Organizer (and Family) Tour of the Community First! Village

We had a great time at DevOpsDays Austin this year and hope many of you did too. Next, we’ll publish a full retrospective that we hope some of you and other DevOpsDays organizers will find interesting.

1 Comment

Filed under Conferences, DevOps

Want to be part of the DevSecOps Handbook?

The word is out, at RSA this week Shannon Lietz (@devsecops), James Wickett (@wickett), John Willis (@botchagalupe), and myself (Ernest Mueller, @ernestmueller) did a panel on our upcoming book, the DevSecOps Handbook.  We’re still writing it, and we want to make you a part of it!

Like the DevOps Handbook, also from IT Revolution Press, the heart of the book is case studies from practitioners like you.  Have you done something DevSecOpsey – adapted the culture of infosec/appsec to work better with your product teams, added security testing to your CI pipeline, added instrumentation and feedback loops for your security work, or other security-as-code kind of work?  Well, we want to hear from you!

We are interested in successes and failures, in both advanced implementation and people taking their first step – others will benefit from your experience in any of these cases.  You can be hardcore security dipping your toes into devops, hardcore dev or ops dipping into security, or someone getting started on the whole ball of wax. Don’t worry, we’re not asking you to write anything, we can interview you and do all the heavy lifting. Not sure if your company will sign off?  We can anonymize it, or if it’s been published publicly as conference proceedings or whatnot then journalism rules apply, we’ll just cite prior work.

To contact us, email book@devsecops.org or go to devsecops.org and fill out the form there. Or if you already know one of us, ping your favorite!

werereadytobelieveyou

We’re ready to believe you!

Leave a comment

Filed under DevOps, Security

SRE: The Biggest Lie Since Kanban

There is a lot of discussion lately about how SRE fits into or competes with or whatever-s with DevOps.  I’m scheduled to speak on a “SRE vs DevOps Smackdown” panel today here at Innotech Austin, and at the exact same time I see Bridget tweeting Liz Fong-Jones’ slides from Velocity on using SRE to implement DevOps. And the more I think about it, and see what people are doing, the more I’m getting worried.

The Big Lie

Just to get the easily provoked to put up their pitchforks, I don’t dislike SRE and I don’t dislike Kanban.  The reason I call Kanban a “Big Lie” is because really doing Kanban correctly and getting the value out of it requires even more discipline that doing something like Scrum.  But it looks so close to doing nothing new that many lazy teams out there say “they’re doing Kanban” and by that they mean they’re doing nothing, but they’ve turned on the Kanban view in JIRA for your convenience.  They have no predictability, they’re not managing WIP, they’re not identifying bottlenecks – they just have a visible board now and that’s it. I strongly believe from my experience that most teams “doing Kanban” are really doing mostly nothing.  There’s articles on this blog about how I make my teams I’m teaching Agile do Scrum first if they want to get to Kanban to build up the required discipline.  And I’m not just a crank, David Hawks from Agile Velocity just told our management team the same thing yesterday, which brought this back to mind for me and spurred this article.

Because I’m starting to see the same thing with SRE.  It’s not surprising – there was and is plenty of “DevOps-washing” of existing teams out there.  Rename your ops team DevOps, done. Well, at least DevOps was able to say “it’s a methodology not a job description or group name stop it” to force deeper thought – it’s why my team at work is the “Engineering Operations” team not “DevOps”, Lee Thompson insisted on that when he set it up! But SRE – yeah, it’s a team just like your own ops team, from an “org chart” viewpoint it looks the same. So doing SRE can – and in many shops does – mean doing nothing new. You just call your existing ops team SRE and figure you’re done.

A brief personal history lesson – my last job before DevOps hit was running the Web Systems team at National Instruments, an ops team.  That’s where we agile admins met, Peco and James were both ops engineers on that team! (Karthik was a dev we worked with.) We had smart people and did ops all right.  We had automation, monitoring, we had “definition of done” standards for new services. You wouldn’t have to squint too hard to just call that team a SRE team and call it a day. But, I wouldn’t wish that job on my worst enemy. It was brutal trying to do ops for just 4-5 dev teams, and that’s with business support, some shared goals, and so on. Our quality of life was terrible, we weren’t empowered, and no matter how hard we tried, success was always right out of our grasp. When we actually started a team using DevOps thinking at NI after that, the difference was night and day, and we actually began to enjoy our jobs as ops engineers. I would hate for anyone to deceive themselves into thinking they’re getting the goodness they should be able to get from a DevOps/”real” SRE approach while still just doing it the way we were doing it.

I have a friend at a local legal software firm, who told me they’re going through and just renaming all the QA folks to SWET (Software Engineer In Test), whether they can code or not, and all the Ops folks to SREs in this manner. One might be charitable and say they’re leaning forward and they intend to loop around and back that up with retraining or something, but… will they? Probably not, it’s just a rename to the hot new term without any of the changes to help those engineers succeed more in their jobs!

SRE isn’t “an implementation of DevOps” if you just apply it as a name for a hopped-up ops team.  Properly understood, it can be an implementation of one of the three parts of DevOps, Infrastructure As Code, Continuous Integration/Deployment, and Site Reliability Engineering. But note that reliability engineering doesn’t start with deploy to production; so much of it is Michael Nygard-esque techniques to write your app reliably in the first place; reliability engineering, in usual DevOps fashion, requires dev and ops work both way back in the dev cycle and out in production to work right. It doesn’t need to be a different team.  If it is, and that team doesn’t get to decide if it takes over ops for a given app, and it’s not allowed to spend 50% of its time on reducing toil and you’re not comping SREs like you do dev engineers – it’s not SRE and you’re a liar for calling it SRE. If you don’t keep DevOps principles in mind, you’re just going to get your old ops team with its old problems again.

That’s why SRE is a Big Lie – because it enables people to say they’re doing a thing that could help their organization succeed, and their dev and ops engineers to have a better career and life while doing so – but not really do it.  Yes, there have been Big Lies before, which is why I cite Kanban as another example – but even if the new criminal is pretty much like the old criminal, you still put their picture up on the post office wall.

Frankly, anyone pushing SRE that doesn’t put warning labels on it is contributing to the problem.  “Well but it mentions in chapter 20 of the second book,” said someone responding to the first version of this article on Twitter.  Not good enough. If something you’re selling is profoundly misused it’s your responsibility to be more up front about the issues.

The Little Problems

Now there are legitimate issues to have even with the “real SRE” model, at least the way that it’s usually being described.  The Google books kinda try to have it both ways, describing it as an engineering practice (how I describe it above and in the SRE course I did for LinkedIn) and describing it as “a team that works this way.”  Even among those not SRE-washing classical ops, the generally understood model is that SRE is a org/job title for a production operations team.

There’s an issue here, the problem of specialization.  If you are Google scale, well, then you’re going to have to specialize and a separate ops team makes sense.  But – first of all, you are not Google scale.  In my opinion, if you are under 100 engineers, you are committing an error by having a separate ops team. You need your product teams to own their products. Second of all – I don’t want to make an enemy of all the lovely Google engineers out there, but is your experience with Google services that they evolve quickly and get better once they go to wide release?  It’s not mine.  They rot.  Have you used Google Hangouts lately without it ending up with cursing and moving off to someone’s Zoom? That kind of specialization still has its downsides in terms of hindering your feedback loops that let you improve (the Second Way). Is SRE just Google-ese for “sustaining?”

I get that the Google folks say they still get feedback and innovation using the SRE model, I’m sure they do and they work hard at that, but that doesn’t change the fact that running a separate ops team is making a deliberate tradeoff between innovation and efficiency. There is no way in that you get as much feedback or improve as quickly with a separate team, you can compensate for it, but you’re still saying “look… Not as important.” Which is fine if that’s your situation, I worked at many companies with 200 abandoned apps in production and you had to do something.  But “not getting there in the first place” is better.

Some of the draw of the model, and why Google is highly aligned with it, is Kubernetes itself. k8s is very complex to run drives people back a little bit to the old priest-in-the-tabernacle model of “someone maintains the infrastructure and you write the app and then you have them deploy it,” but now there’s some standards (like deploying as a container) that make that OK – I guess? But if you think reliability, and observability, are the primary responsibility of an ops team that is not involved in constructing the application, you either have deep and profound company standards that allow seamless plugging of the one into the other or you’re fooling yourself. 90% of you are fooling yourself.

At this conference I heard “Service meshes!  They get you observability so your devs don’t have to think about it.” Do you not see how dangerous that mindset is?

SRE, as interpreted as “a separate newfangled ops team,” may work for some but you need to be realistic about the issues and tradeoffs you’re making.  Consider whether product teams supporting their product, maybe with aid from a platform team making tooling and an enabling/consulting/center of excellence team that can give expert advice?  DevOps helped us see how the “throw it over the wall from dev to ops” model was profoundly harming our industry.  Throwing over the wall from dev to SRE doesn’t improve that, it’s profoundly regressive. Doing SRE “right” to compensate for this, like doing Kanban right, requires more skill and discipline, not less – be realistic about whether you have Google levels of skill and discipline in your org, eh?

Conclusion

SRE (and Kanban) aren’t bad, they have their pros and cons, but they are easy to “pretend to do” in some minimal, cargo cult-ey way that gets you little of the benefits. And if you think spinning up an ops team and calling it SRE is “an implementation of DevOps” you’ve swallowed the worst poison pill the DevOps talk circuit can deal to you.

15 Comments

Filed under DevOps

DevOpsDays Austin 2018 Videos Posted

Well, we were “unplugged,” but we managed to smuggle videos out anyway for your pleasure… Watch ’em, like ’em, comment to the speakers that you appreciate them giving to the global tech community!  Especially since this year they weren’t pre-selected, voting on talks was done at the event, so these folks prepared a talk but weren’t for sure to give it, which takes guts!

Leave a comment

Filed under Conferences, DevOps

DevOpsDays Austin 2018 Retrospective and 2019 Prospectus

logoAll right, DevOpsDays Austin 2018 went great and the organizers (thanks be unto them – James Wickett, Dan Zentgraf, Boyd Hemphill, Richard Boyd, Scott Baldwin, Lee Thompson, Karthik Gaekwad, Marisa Sawatphadungkij, Ian Richardson, Bill Hackett, Chris Casey, Carl Perry, and our ConferenceOps finance handler Laura Wickett) have had the time to do a retrospective and both share what we’ve learned and set a course for next year’s event! This is long and I assume mostly of interest to other DevOpsDays organizers, so buckle in.

DoD Austin this year was another experimental year. Austin was the third DevOpsDays city in the US and the eleventh globally, and has been going every year since 2012.  Because our community has such a long history with DevOpsDays, we experiment with our format to find what works the best for us.

This year, we tried a couple daring things (more details in DevOpsDays Summit Austin 2018 – “DevOps Unplugged”):

  1. Voting on talks onsite instead of ahead of time (saw this at ProductCamp Austin)
  2. No sponsor booths (like the early DevOpsDays, Silicon Valley was like this for several years)
  3. Boxed lunches (like the early DevOpsDays, Silicon Valley was like this for several years)
  4. Capped headcount low at 400 (despite having sold 650 tickets last year)
  5. No streaming the talks (video is coming though)

Read the linked article for why, but the TL;DR is that we’re a nonprofit conference that exists to drive community engagement, and the “DevOps Talk Circuit,” the increased sponsor lead-churn demands, the time we spent on fancy lunches and such, and just the sheer number of attendees and weight of extras we were adding on were choking out the actual goal of the conference.  Despite having a huge slate of great keynoters at 2017 and everything being the biggest and best DoDA ever – we the organizers didn’t have a good time. We didn’t learn anything or make new friends. And we heard from other experts in town that said the same thing. So a dramatic change was implemented to pare the event back down to basics.  But how’d it work out?

We did a bunch of retrospective activities to find the answer!

  1. SurveyMonkey survey of all attendees
  2. Survey of all sponsors
  3. Community retrospective at the Austin DevOps user group
  4. Organizer retrospective

Attendee Survey Feedback

Of 400 attendees, we got 51 respondents (12.5%). Our overall NPS was 25 (“pretty good”). We don’t have a last year NPS to compare to, we didn’t do a great job of post event surveying last year mostly due to burnout (once you’ve spent most of your time prepping a conference, it’s time to get back to your real work, family, etc.).

Food Quality Talk Quality Openspace Quality Venue Quality Happy Hour Quality
Very high – 9 (18%) Very high – 6 (12%) Very high – 7 (14%) Very high – 12 (24%) Very high – 12 (25%)
High – 20 (39%) High – 27 (53%) High – 12 (47%) High – 29 (57%) High – 12 (25%)
Neither – 17 (33%) Neither – 9 (18%) Neither – 12 (24%) Neither – 7 (14%) Neither – 22 (46%)
Low – 4 (8%) Low – 8 (16%) Low – 8 (4%) Low – 3 (6%) Low – 2 (4%)
Very low – 1 (2%) Very low – 1 (2%) Very low – 3 (6%) Very low – 0 (0%) Very low – 0 (0%)

So everything was 50% or better “very high or high,” which seems good. We asked about favorite sponsors – ones mentioned by multiple participants include Cisco, Red Hat, NS1, VictorOps, Sumo Logic, xMatters, and Praecipio.

The comments were enlightening.  This year’s format was pretty divisive – there were lots of comments about liking voting on the talks and lots of comments about not liking it; there were lots of comments about liking e.g. “The new format with less vendor bloat” and then also lots of comments wanting sponsor booths back. And frankly, that’s what we expected – the new format was expressly designed to be attractive to some kinds of attendees and sponsors and not to others.

Overall, the positive comments predominated on the openspaces, keynotes, and ignites, and negative predominated on the talks and lack of booths.  (Several of those respondents identified as sponsors.)

Sponsor Survey Feedback

Total sponsor NPS was 7 (“good”) from 14 respondents of our 17 sponsors.  Again, there wasn’t the usual bell curve distribution – some sponsors loved it and others hated it.  The venue and the conversations people had onsite were very highly rated. The limited swag table aspect was low rated. The 30 minute suite sessions and lead quality were sharply bimodal – for example:

How did your 30 minute suite demo go?

  • Did not use 7.14%
  • Very well 7.14%
  • Well 28.57%
  • Neither poorly nor well 14.29%
  • Poorly 28.57%
  • Very poorly 14.29%

User Group Feedback

Read the board yourself!  Attendees, some organizers were in attendance.

image1

Analysis

Change is hard

People’s expectations were hard to alter. Especially in the sponsor realm where the person who books the sponsorship isn’t usually the person that comes on site.  One sponsor comment said “Without a booth, not worth our $5000!”  Well, yeah, that’s why we didn’t charge you $5k this year. People that go to multiple DevOpsDays, and especially sponsors, but even people who had just been to our event multiple years – we emailed and tweeted and blogged and put stuff on the signup forms, but the changes were still a surprise to many.  Voting on the talks was a concern not as much from speakers, but from people who “wanted their schedule set in advance!” and from people who were “afraid it makes speakers feel bad.”

Money isn’t hard

Even with the much lower sponsor cost this year ($3k), and lowering our headcount significantly (400), and providing the same great venue and lunches and breakfasts and drinks and not 1 but 2 shirts and blowing it out on the happy hour, plus being ripped off by our happy hour venue (not going back there!!!), we were still well in the black enough that we’re giving thousands of dollars to charity at the end of the event.

In fact, one of the advantages of this year’s format was that we weren’t giving 1/3 of our tickets away for free to a huge army of organizers, to speakers, etc.  Adding more sponsor stuff requires adding more volunteers that just eats back into the revenue stream again.

Specific Outcomes

Voting on talks

There was enough pushback that we won’t do that next year.  Submissions were lower this year, and a bunch of people dropped out before the event.  However, many of the people who dropped out are, to be blunt, the people we wanted to drop out. Talks “submitted on behalf of” someone. Vendor roadshow talks.

Here’s the thing – here in Austin, we’re pretty blessed.  We have a huge tech community with all the big players.  If you want to “have your secretary submit your talk, fly in, drive to the venue, give your talk, fly out” – whoever you are,  you really don’t have anything more interesting to say than the people who are already here. So if your goal being at DoDA isn’t to interact with the community, we have plenty of talk submissions already, thanks.  I get that if you’re starting up a DoD in the middle of nowhere the people on the “DevOps Talk Circuit” are key to bringing in new ideas and jumpstarting you, and I don’t devalue that.  But for us, we don’t need that and it doesn’t serve the needs of our current community.

This isn’t to say people from away aren’t welcome – John Willis is from Atlanta but he’s part of our community, because when he comes here that’s how he interacts with us.  (One of the “What did you like the most” survey comments simply said “John Willis.”)

People suggested various half-measures – “have us vote a week before!” But the additional logistics on that is very much not worth it, especially given what we think we’ve learned about our talk needs – read on for that!

Sponsor tables

OK, no sponsor tables was not universally beloved. Some sponsors – and not just the “here for the leadz” sponsors we were deliberately discouraging with the format – didn’t like it because it was harder to interact with folks about their product. But – here’s the rub – we had just as many complaints last year when we *did* have sponsor tables!  “My table was in the corner.” “There wasn’t enough foot traffic driven to me.”

The stadium format is pretty “noisy” and if we had sponsor tables back we’d have to do talks in some far-away rooms again, and removing those rooms this year saved us a lot of money and also people always hated it (like – FAR away).

Also, I’ll be honest, we had problems with sponsor misbehavior last year.  Silver sponsors claiming a table and standing behind it like a gold. Sponsors going out on the field (forbidden by UT). Sponsors trying to have food trucks park outside (also forbidden by UT police). Disruptive activity of a number of different sorts, requiring lots of work by organizers and volunteers and venue staff to deal with. I am sure many of them thought they were being “scrappy” etc. but in the end, we don’t get paid for this conference so we don’t need to put up with crap for it either. Discussion about “firing” certain sponsors was had.

We aren’t going back to the usual sponsor tables, but we are going to try something even more different – read on for that!

Boxed lunches

In early DoDA, we kept having super-deluxe Austin fare – BBQ, tex-mex – not from a caterer but from the real good places. This was for all the folks from away we were bringing in and wanted to show an Austin good time to!

Unfortunately, last year food lines for 650 people were a problem. Vendors weren’t adequately prepared with people or food.  We had to have many volunteers assigned. Food lines were super long and slow and a source of frustration.

This year we did have some comments about “I wanted the deluxe foods.” But they were far overwhelmed by those who appreciated being able to grab sustenance and get back to why they are here, learning and discussion. So with enough money we may try to get some kind of super-deluxe box lunch, but the box lunches will stay.

Lower headcount

The lower headcount was universally beloved except by lead generators and those who couldn’t get a ticket. More and better interaction, many positive comments noted the more intimate communication in openspaces and hallway track.  Keep.

No streaming

Worked out great.  No one complained, and the cost and org/volunteer time and schedule and stage compromises we have to make for live streaming are immensely negative.  Not going back.

2019 Planning

First of all, a disclaimer.  I am sharing this in the interests of transparency and helping other organizers learn from what we’ve done.  I don’t claim Austin is doing things the “one true way” and I know our community’s needs are different from many others. None of this is intended to denigrate any other events and their decisions. You don’t need to justify why you do things differently or why any of this isn’t right for your community.

Every year I start our planning with some basic questions.

  1. Do we want to have a DevOpsDays Austin next year?
  2. If so, why?  What is the goal of this year’s event?

“Inertia” is a bad reason to do anything.  We don’t have “money” as a reason because we have to spend what we get, we don’t pocket anything except some gifts. (My kid has already appropriated the bluetooth speaker I got this year…)

The group of organizers (over a tasty dinner at Chez Zee) decided “yes”, and after a good bit of discussion they decided that to us, this year, the goal of DevOpsDays Austin is to “Promote collaboration and sharing and networking specifically for the Austin technical community.” Now, that’s a pretty non-controversial statement on its face – but then as we plan stuff, we really test it against our goal and see if it supports it, is neutral, or takes away from it.  If it’s neutral or takes away, it goes.

This decision and clear statement (I think Marisa is who put it together for us) pricked my memory and I pulled out our attendee survey comments.  What did you like the most about DevOpsDays Austin 2018?  “Ability to collaborate with others.” “Enjoyed hearing what others were doing.” “Focus on the community.” “It’s a well-run, intimate conference.  I always see people I know.” “The community involvement.”  Her sentence crystallized what people were telling us was their favorite part of the event – super!

OK, so what does that mean for each area?

Content

People love the lightning talks more than anything.  Then the keynotes. Then the talks. It’s why we tried the attendee voting. The discussion covered how many of the talks seem too long and boring even at 35 minutes, and people trying to get too technical in them suffer from people not being able to follow along well due to screen size and large group.  People say they want themed tracks and stuff, but we rely on volunteers giving talks, we aren’t buying these off the shelf somewhere (“Give me 6 Kubernetes talks, 6 DevOps culture talks, 6 DevOps manager talks, and 6 intermediate level technical talks…”)  We are still committed to multiple technical tracks (DoDA was the first DoD to do this, many are still uni-track) because we’re 7 years in and we have a great diversity of experience in our community, and people don’t want to sit through the same messaging again.

Some talks are beloved and others aren’t.  As we sifted through the details, one comment from “What can we do better” on the attendee survey came to me.  “Talks focused on ‘I am a _____, here’s the problem we had and how we solved it.’ I say that because one of the coolest, most useful talks I saw was the Coinbase engineer who described how he used EBS volumes creatively to solve their scaling problem.”

So we decided to retire the voting but heavily curate the talks.  We don’t want “whatever talk you’re giving nowadays on the DevOps talk circuit” – we want talks in that format, the problem you had and how you solved it.

We’re working out the details, but we’re thinking about having these talks be more like 15 minutes long, with then linked openspaces that afternoon for the truly interested to get together and go ‘command line level’ with them.  This also allows for more breaks and collaboration time.

We also decided that idiosyncratic is better.  A couple of the organizers got excited about a sports/fitness theme to align with the stadium; one wants to set up a 5K, one has a wife that does yoga classes and we could have one, we can give fitbits as speaker gifts… While I and the other Agile Admins have been filming lynda.com courses and doing other creative things, the advice we keep getting from producers and directors and content managers is “Use *your* voice.  Do what *you* find interesting and other people will find it interesting.” Andrew Shafer loves running Werewolf games at openspaces at conferences, and people really respond to it! So we’re not going to hesitate to put stuff in we find interesting and we figure that enthusiasm will draw others. Trying to give attendees a “standard conference experience” is severely counterproductive because there’s plenty of regular conferences for people to go to, they get sick of it, and that doesn’t fit the devopsdays ethos in the first place.

Sponsors

I challenged the group.  “Tell me why we should have sponsors at all?  Half our revenue was ticket sales and half was from sponsors.  If we double ticket prices to $400 – still very low for any 2-day conference in the world – we can just not take sponsors at all, done and done. If we needed their money it’d be one thing, but we don’t. Let them spend their ‘limited marketing budget’ on the DoD events that do need it. How do the sponsors contribute to our goal other than with funding?”

The immediate response was that there are a bunch of sponsors who *are* part of the community and interacting with them is important; we have loads of Amazon/Google/Atlassian/Oracle/etc hiring going on here for example, and folks who work for Chef and Salt and Puppet and so on in town… We want those folks to be part of the conversation.  Just not disrupt that conversation.  And, some people pay for those tickets out of pocket so having some money to defray attendee costs is good.

We decided to try something different – we are using the luxury boxes at the stadium more and more; they’re relatively inexpensive and we used them for all the openspaces and such this year.   What if, we said, we intersperse sponsor suites with openspace suites, maybe even have them host some of the openspaces, do their own presentations in there too for whoever’s interested?  This means a limited number of sponsor slots (no more than 10, possibly fewer), but a more premium experience right there where the action is happening. And target Austin-presence companies to let them know about it. They can also then get food/drink catered into their suites to bring people in even more.

Attendees

Keep the headcount low – at least our limit of 400 from this year, if not lower. Consider a ‘two-tier’ ticket price with one price if your company is paying and another if you are; Data Day Austin has used this format to good effect.  Lets the non-backed solo folks in without breaking their bank but lets companies that do send attendees pay a reasonable amount.

Venue

UT Stadium is great, we don’t really see a reason to do all the work to change if we’re not doing booths and we’re going with a suite strategy for sponsors. Plus we have developed great relationships with the venue staff.

Keep refining the AV experience but doing it ourselves – we bought equipment and have a large set of “A/V geeks” so we don’t need to have outside people do it.

Food

Keep with boxed lunches. Austinites have had enough BBQ and tex-mex and this event is primarily for them per our goal. The benefit of fast lunch and snacks was tremendous this year. Could spend more on boxes from premium vendors but keep it boxed.  Maybe do drink service ourselves because we got truly rooked by the UT caterers on it this year.  Though Rich said he found the place the athletes eat and we might be able to get in on that… Keeping it fast, though, one way or the other.

Happy Hour

We put a lot of work into this and spend double what the happy hour sponsor gives us each year, and then only half the people come and only half of those say they like it.  This year we had unlimited food and booze at a venue with video games in it for Pete’s sake, I think we’re done chasing the idea of the ultimate happy hour. Probably we’ll do more of an onsite short sponsor room crawl at the venue, and then an “after party” we don’t put as much money/work into. “A couple free rounds at Scholtz’, get your own ass there.”

Conclusion

All right, that’s all the plan one dinner could get us.  But in the end, we’re happy with how the event went this year.  We’ll change a couple of the things that didn’t work out – talk voting, no booths – but not back to the old way because we already know that was suboptimal, instead we’ll try more options!  If you don’t have experiments not work out, you’re not being experimental enough, so we embrace that with DevOpsDays Austin.

Let us know your thoughts too!  Who are you, and what do you get or want to get out of DevOpsDays Austin?

Leave a comment

Filed under Cloud, DevOps

Keep Austin Agile 2018 Trip Report

This Thursday, both myself and my boss (the SVP of Engineering at Alienvault) went to Keep Austin Agile, the annual conference that Agile Austin, the local Austin agile user group network, puts on!  I used to run the Agile Austin DevOps SIG till I just ran out of time to do all the community stuff I was doing and had to cut it out.

Logo-Tagline.2376.v2017.08.16

It’s super professional for a practitioner conference, and was at the JW Marriott in downtown Austin one day only.  It was sold out at 750 people. I figured I’d share my notes in case anyone’s interested.  All the presentations are online here and video is coming soon.

DevOps Archaeology

My first session was DevOps Archaeology by Lee Fox (@foxinatx), the cloud architect for Infor. The premise is that it’s an unfortunately common task in the industry to have to “go find out how that old thing works,” whether it’s code or systems or, of course, the hybrid of the two.  So he has tips and tools to help with that process.  Super practical.  Several of my engineers at work are working on projects that are exactly this. “Hey that critical old system someone pooped out 3 years ago and then moved on – go figure it out and operationalize it.”

I basically wrote down the list of cool tools that help with this process…
  • Codecity – visualizes your code as a city
  • Gource  – visualizes the evolution of your codebase over time
  • Signaturesurvey – scan for patterns in code
  • Logstalgia – visualizes historical traffic to a Web endpoint
  • Proxies – setting up proxies helps understand what’s going on, at an even deeper level than flow logs.
  • Monitoring – you know, all the usual monitoring tools.
  • Logs – you know, all the usual log aggregation tools.
Then he had a bunch of AWS-specific tools too.  All our stuff is in AWS, so super useful.
  • Cloudtrail – AWS API logs, yeah.  We pump our cloudtrail into our own USM Anywhere instance to report on weirdness.
  • Config – new service, have it report on things not tagged right, if volumes are encrypted, whatever kind of rules you want to set up.  Nice!
  • Trusted Advisor – well, don’t trust it too much, I’ve learned the hard way there’s lots of limits and stuff it doesn’t know about.  But useful.
  • Macie – “machine learning” (I always put that in scare quotes nowadays because of its overuse) to identify weirdness in your environment. Detect high risk cloudtrail events, unusual locations of activity, and so on.

And, some discussion of testing, config management, and so on.  Great talk, I will look into some of these tools!

Brewing Great Agile Team Dynamics

This talk, by Allison Pollard (@allison_pollard) and Barry Forrest (@bforrest30), wasn’t really my cup of tea. It did a basic 4-quadrant personality survey to break us up into 4 categories of Compliance, Dominance, Steadiness, or Influencer.  Then we spent most of the time wandering the room in a giant circle doing activities that each took 10 minutes longer than they needed to.

So I’m fine with the 4 quadrant thing – but I got taught a similar thing back when starting my first job at FedEx back in 1993, so it wasn’t exactly late breaking news.  (Driver, Analytical, Amiable, and Expressive were the four, IIRC.)  As a new person it was illuminating and made me realize you have to think about different personalities’ approaches and not consider other approaches automatically “bad.” So yay for the concept.

But I’m not big on the time consuming agile game thing that is at lots of these conferences. “What might turn you off about a Dominant person?  That they can be rude?” Ok, good mini-wisdom, should it take 10 minutes to get it? Maybe it’s just because I’m a Driver, but I get extremely restless in formats like this. A lot of people must like them because agile conferences have them a lot, but they’re not for me.

Modern Lean Leadership

Next up was Modern Lean Leadership by Mark Spitzer (@mspitzer), an agile coach. I love me some Deming and also am always looking to improve my leadership, so this drew me in this time slot.

First, he quoted Deming’s 14 points for total quality management.  For the record (quoted from asq.org:

  1. Create constancy of purpose for improving products and services.
  2. Adopt the new philosophy.
  3. Cease dependence on inspection to achieve quality.
  4. End the practice of awarding business on price alone; instead, minimize total cost by working with a single supplier.
  5. Improve constantly and forever every process for planning, production and service.
  6. Institute training on the job.
  7. Adopt and institute leadership.
  8. Drive out fear.
  9. Break down barriers between staff areas.
  10. Eliminate slogans, exhortations and targets for the workforce.
  11. Eliminate numerical quotas for the workforce and numerical goals for management.
  12. Remove barriers that rob people of pride of workmanship, and eliminate the annual rating or merit system.
  13. Institute a vigorous program of education and self-improvement for everyone.
  14. Put everybody in the company to work accomplishing the transformation.

His talk focused on #7 and #8 – instituting leadership and driving out fear.

Many organizations are fear driven. Even if it’s more subtle than the fear of being fired, the fear of being proven wrong, losing face, etc. is a very real inhibitor.  Moving the organization from fear to safety to awesome is the desired trajectory.

He uses “Modern Agile” (Modernagile.org) which I hadn’t heard of before, but its principles are aligned with this:

  • Make People Awesome
  • Make Safety a Prerequisite
  • Experiment & Learn Rapidly
  • Deliver Value Continuously

So how do we create safety? There’s a lot to that, but he presented a quality tool to analyze fear and its sources – who cares and why – to help.

Then the next step is to determine mitigations, and how to measure their success and timebox them. I’m a big fan of timeboxing, it is critical to making deeper improvement without being stuck down the rabbit hole.  I tell my engineers all the time when asked “well but how much do I go improve this code/process” to pick a reasonable time box and then do what you can in that window.

OK, but once you have safety, how do you make people awesome? Well, what is awesome about a job?  Focus on those things.  You can use the usual Lean techniques, like stop-work authority, making progress visible (e.g. days without an incident), using the Toyota kata for continuous improvement, using Plan-Do-Check-Act…

In terms of tangible places to start, he focused on things that disrupt people’s sleep at night, doing retros for fear/safety, and establishing metric indicators as targets for improvement.

Another good talk!

How The Marine Corps Creates High-Performing Teams

Andy McKnight gave this interesting talk – explaining how the Marines build a culture and teamwork, so that we might adapt their approach to our organizations.  I do like yelling at people, so I am all in!

Marine boot camp is partially about technical excellence, but also about steeping recruits in their organizational culture. (In business, new hire orientations have been shown to give strong benefits… And mentoring after the fact.)

What is culture?  It is the shared values, beliefs, assumptions that govern how people behave.

Most organizations have microcultures at the team level.  But how do you make a macroculture?  Culture comes first, teambuilding second.

  1. shift your org structure to align with the value stream instead of functional silos
  2. measure as a team

The 11 Marine Corps Leadership Principles:

  1. Know yourself and seek self-improvement.
  2. Be technically and tactically proficient.
  3. Develop a sense of responsibility among your subordinates.
  4. Make sound and timely decisions.
  5. Set an example.
  6. Know your people and look out for their welfare.
  7. Keep your people informed.
  8. Seek responsibility and take responsibility for your actions.
  9. Ensure assigned tasks are understood, supervised, and accomplished.
  10. Train your people as a team.
  11. Employ your team in accordance with its capabilities.

On the scrum team – those necessary to get the work done

The two Leadership Objectives – mission accomplishment and team welfare, a balance.

Discussion of Commanders Intent and delegating decisions down to the lowest effective level.

Good discussion, loads of takeaways. At my work I would say we are working on developing a macroculture but don’t currently have one, so I’ll be interested to put some of this into practice.

Agile for Distributed Teams

And finally, Agile for Distributed Teams by Paul Brownell (@paulbaustin). At my work we have distributed teams and it’s a challenge. Lots of stuff in the slides, my takeaways are:

  • People’s biggest concern – not understanding enough context, not sharing values
  • Use multiple communication channels – video, chat, email.
  • Get F2F time.  Quarterly.  Make it happen. Use ambassadors.
  • Expose the team to Other parts of the org, get users involved
  • Establish rules of engagement – hours, channels, etc. for clarity.
  • Teams will have local subcultures – make a space for shared learning, encourage lateral communication, emphasize early progress.
  • Use icebreakers in standups etc – something about your week
  • Teambuilding- slack channels, scavenger hunts
  • Sprint planning – one or two meetings? Involve the team.
  • Standups – try all on headsets to level the playing field for in room/out of room.
  • Online whiteboards
  • Retros – be creative, get written feedback ahead of time

All right!  4 of 5 sessions made me happy, which is a good ratio. Check out these talks and more on the Keep Austin Agile 2018 Web site!  It’s a large and well run conference; consider attending it even if you’re not an “agile coach”!

 

2 Comments

Filed under Agile, Conferences, Security

Released! SRE, Monitoring and Observability!

Well we haven’t had a lot of spare blogging time but we’ve been busy – the agile admins have been hard at work on some more LinkedIn Learning/lynda.com video courses to help make DevOps more accessible for the common man and woman.

lyndalinkedinJames and I did DevOps Foundations: Site Reliability Engineering, which explains our view of what SRE is, and its position in the DevOps arena. This rounds out our “DevOps 201” series, where we delve down into the three practice areas of DevOpscontinuous delivery, infrastructure as code, and SRE.

Monitoring is a big part of SRE, but too much to do in the scope of all one course – so at the same time, Peco and I filmed a companion course, DevOps Foundations: Monitoring and Observability!  Like the CD and IaC courses, this alternates theory with demos.

Along with the recent DevOps Foundations: Lean and Agile I did with Karthik, we feel like we’ve now completed a curriculum that can introduce you to all the major areas of DevOps and prepare you to grow from there. (Link to lynda.com playlist.) The guys are doing other courses with lynda as well, we’ve kinda gotten addicted to doing them!  You can check them all out by your favorite Agile Admin:

I love talking to the folks I meet at conferences and whatnot who have done the courses; let us know what other areas you’d like to get the Agile Admin learning treatment!

We’re not a consultancy or anything – just four practitioners here in Austin who love giving back to the community when we’re not doing our day jobs. We get some royalties per click from the classes, but other than that we don’t have anything to sell you. We got into this to help people make sense of the confusing DevOps landscape and we’ll keep doing it as long as it seems like it’s meeting that goal, so your feedback is needed to let us know if we should keep going and if so on what.

Leave a comment

Filed under DevOps

Monitoring and Observability

Ah, observability, the new buzzword of the day. Monitoring vendors aplenty are using the word, to basically mean “better monitoring!” You know, #monitoringlove not #monitoringsucks. Because monitoring doesn’t help with debugging and doesn’t have app instrumentation right?

Well, I have to say “bah” to that.  So here’s the thing.  I’m an electrical engineer by education, and I spent a lot of time working at National Instruments, an engineering test and measurement company.  You may be surprised to know these terms have actual definitions that don’t require Twitter arguments to discover.

Monitoring is an activity you perform. It’s simply observing the state of a system over a period of time.

Why do we monitor? For three reasons, in general.

  • Problem Detection – you know, alerting, or seeing issues on dashboards.
  • Problem Resolution – root cause and troubleshooting.
  • Continuous Improvement – capacity planning, financial planning, trending, performance engineering, reporting.

How do we monitor?  Well, that’s called instrumentation. You can instrument your systems and get CPU and stuff, you can use synthetic probes, you can use JavaScript bugs to get end user monitoring, you can emit metrics from applications, you can introspect services and apps via whatever parts are exposed (from JMX to nginx stats to sysdig traces), you can take network traces… (Some folks are similarly trying to redefine “instrumentation” to just mean application instrumentation, which is lame, and in defiance of the fact that application performance management tools that do app instrumentation have existed for decades.)

You can instrument metrics or events; metrics have certain sampling frequency and resolution…

So what is observability?  This isn’t a new term. It comes from system control theory. You know, the stuff that makes your A/C system and electrical plants and your car work.

Observability is a measure of how well the internal states of a system can be inferred from knowledge of its external outputs.

Observability is a property of a system. You can monitor a system using various instrumentation, but if the system doesn’t externalize its state well enough that you can figure out what’s actually going on in there, then you’re stuck.

So is observability hippy bullcrap?  No, of course not. In a DevOps world, it’s very important that the apps and systems concentrate on making themselves both observable and controllable (I leave it to the reader to research controllability, unless I get agitated enough to post about that too). Do you make yourself “easy to monitor”?

Externalizing custom metrics contributes to observability (you know, like with dropwizard metrics).  So does good logging.  So does proper architecture!  Take a system that sticks all kinds of messages into one message queue rather than using separate queues for separate types – the latter is more observable; you can more readily see how many of what is flowing through.  (It’s more controllable too, as you can shut off one queue or another.)

Making your system observable is therefore important, so that if you monitor it with appropriate instrumentation, you understand the state of the system and can make short or long term plans to change it.

While a monitoring tool can definitely contribute to this via its innovation in instrumentation, analysis, and visualization, in large part observability is a battle won or lost before you start sticking tools on top of the system. It’s very important to take it into account when designing and implementing services. No tool is going to “give you” observability and that’s the usual silver bullet fallacy heard from someone who wants to sell you something.

I’m not saying every vendor is using the term wrongly (in fact I just came across this New Relic post that is very well done), but I have to say I am less than impressed when common engineering terms are so widely misused and misunderstood widely in our industry.

Would you like to know more?  Peco and I are working on a new lynda.com course on monitoring and observability!  There’ll be real engineering, a broad canvas of the different kinds of monitoring instrumentation, tips on implementation and use… We’ve both been using and/or building monitoring tools for decades now so we hope to have some useful info for you.

1 Comment

Filed under DevOps, Monitoring