Tag Archives: devopsdays

by | May 29, 2018 · 10:00 am

DevOpsDays Austin 2018 Retrospective and 2019 Prospectus

logoAll right, DevOpsDays Austin 2018 went great and the organizers (thanks be unto them – James Wickett, Dan Zentgraf, Boyd Hemphill, Richard Boyd, Scott Baldwin, Lee Thompson, Karthik Gaekwad, Marisa Sawatphadungkij, Ian Richardson, Bill Hackett, Chris Casey, Carl Perry, and our ConferenceOps finance handler Laura Wickett) have had the time to do a retrospective and both share what we’ve learned and set a course for next year’s event! This is long and I assume mostly of interest to other DevOpsDays organizers, so buckle in.

DoD Austin this year was another experimental year. Austin was the third DevOpsDays city in the US and the eleventh globally, and has been going every year since 2012.  Because our community has such a long history with DevOpsDays, we experiment with our format to find what works the best for us.

This year, we tried a couple daring things (more details in DevOpsDays Summit Austin 2018 – “DevOps Unplugged”):

  1. Voting on talks onsite instead of ahead of time (saw this at ProductCamp Austin)
  2. No sponsor booths (like the early DevOpsDays, Silicon Valley was like this for several years)
  3. Boxed lunches (like the early DevOpsDays, Silicon Valley was like this for several years)
  4. Capped headcount low at 400 (despite having sold 650 tickets last year)
  5. No streaming the talks (video is coming though)

Read the linked article for why, but the TL;DR is that we’re a nonprofit conference that exists to drive community engagement, and the “DevOps Talk Circuit,” the increased sponsor lead-churn demands, the time we spent on fancy lunches and such, and just the sheer number of attendees and weight of extras we were adding on were choking out the actual goal of the conference.  Despite having a huge slate of great keynoters at 2017 and everything being the biggest and best DoDA ever – we the organizers didn’t have a good time. We didn’t learn anything or make new friends. And we heard from other experts in town that said the same thing. So a dramatic change was implemented to pare the event back down to basics.  But how’d it work out?

We did a bunch of retrospective activities to find the answer!

  1. SurveyMonkey survey of all attendees
  2. Survey of all sponsors
  3. Community retrospective at the Austin DevOps user group
  4. Organizer retrospective

Attendee Survey Feedback

Of 400 attendees, we got 51 respondents (12.5%). Our overall NPS was 25 (“pretty good”). We don’t have a last year NPS to compare to, we didn’t do a great job of post event surveying last year mostly due to burnout (once you’ve spent most of your time prepping a conference, it’s time to get back to your real work, family, etc.).

Food Quality Talk Quality Openspace Quality Venue Quality Happy Hour Quality
Very high – 9 (18%) Very high – 6 (12%) Very high – 7 (14%) Very high – 12 (24%) Very high – 12 (25%)
High – 20 (39%) High – 27 (53%) High – 12 (47%) High – 29 (57%) High – 12 (25%)
Neither – 17 (33%) Neither – 9 (18%) Neither – 12 (24%) Neither – 7 (14%) Neither – 22 (46%)
Low – 4 (8%) Low – 8 (16%) Low – 8 (4%) Low – 3 (6%) Low – 2 (4%)
Very low – 1 (2%) Very low – 1 (2%) Very low – 3 (6%) Very low – 0 (0%) Very low – 0 (0%)

So everything was 50% or better “very high or high,” which seems good. We asked about favorite sponsors – ones mentioned by multiple participants include Cisco, Red Hat, NS1, VictorOps, Sumo Logic, xMatters, and Praecipio.

The comments were enlightening.  This year’s format was pretty divisive – there were lots of comments about liking voting on the talks and lots of comments about not liking it; there were lots of comments about liking e.g. “The new format with less vendor bloat” and then also lots of comments wanting sponsor booths back. And frankly, that’s what we expected – the new format was expressly designed to be attractive to some kinds of attendees and sponsors and not to others.

Overall, the positive comments predominated on the openspaces, keynotes, and ignites, and negative predominated on the talks and lack of booths.  (Several of those respondents identified as sponsors.)

Sponsor Survey Feedback

Total sponsor NPS was 7 (“good”) from 14 respondents of our 17 sponsors.  Again, there wasn’t the usual bell curve distribution – some sponsors loved it and others hated it.  The venue and the conversations people had onsite were very highly rated. The limited swag table aspect was low rated. The 30 minute suite sessions and lead quality were sharply bimodal – for example:

How did your 30 minute suite demo go?

  • Did not use 7.14%
  • Very well 7.14%
  • Well 28.57%
  • Neither poorly nor well 14.29%
  • Poorly 28.57%
  • Very poorly 14.29%

User Group Feedback

Read the board yourself!  Attendees, some organizers were in attendance.

image1

Analysis

Change is hard

People’s expectations were hard to alter. Especially in the sponsor realm where the person who books the sponsorship isn’t usually the person that comes on site.  One sponsor comment said “Without a booth, not worth our $5000!”  Well, yeah, that’s why we didn’t charge you $5k this year. People that go to multiple DevOpsDays, and especially sponsors, but even people who had just been to our event multiple years – we emailed and tweeted and blogged and put stuff on the signup forms, but the changes were still a surprise to many.  Voting on the talks was a concern not as much from speakers, but from people who “wanted their schedule set in advance!” and from people who were “afraid it makes speakers feel bad.”

Money isn’t hard

Even with the much lower sponsor cost this year ($3k), and lowering our headcount significantly (400), and providing the same great venue and lunches and breakfasts and drinks and not 1 but 2 shirts and blowing it out on the happy hour, plus being ripped off by our happy hour venue (not going back there!!!), we were still well in the black enough that we’re giving thousands of dollars to charity at the end of the event.

In fact, one of the advantages of this year’s format was that we weren’t giving 1/3 of our tickets away for free to a huge army of organizers, to speakers, etc.  Adding more sponsor stuff requires adding more volunteers that just eats back into the revenue stream again.

Specific Outcomes

Voting on talks

There was enough pushback that we won’t do that next year.  Submissions were lower this year, and a bunch of people dropped out before the event.  However, many of the people who dropped out are, to be blunt, the people we wanted to drop out. Talks “submitted on behalf of” someone. Vendor roadshow talks.

Here’s the thing – here in Austin, we’re pretty blessed.  We have a huge tech community with all the big players.  If you want to “have your secretary submit your talk, fly in, drive to the venue, give your talk, fly out” – whoever you are,  you really don’t have anything more interesting to say than the people who are already here. So if your goal being at DoDA isn’t to interact with the community, we have plenty of talk submissions already, thanks.  I get that if you’re starting up a DoD in the middle of nowhere the people on the “DevOps Talk Circuit” are key to bringing in new ideas and jumpstarting you, and I don’t devalue that.  But for us, we don’t need that and it doesn’t serve the needs of our current community.

This isn’t to say people from away aren’t welcome – John Willis is from Atlanta but he’s part of our community, because when he comes here that’s how he interacts with us.  (One of the “What did you like the most” survey comments simply said “John Willis.”)

People suggested various half-measures – “have us vote a week before!” But the additional logistics on that is very much not worth it, especially given what we think we’ve learned about our talk needs – read on for that!

Sponsor tables

OK, no sponsor tables was not universally beloved. Some sponsors – and not just the “here for the leadz” sponsors we were deliberately discouraging with the format – didn’t like it because it was harder to interact with folks about their product. But – here’s the rub – we had just as many complaints last year when we *did* have sponsor tables!  “My table was in the corner.” “There wasn’t enough foot traffic driven to me.”

The stadium format is pretty “noisy” and if we had sponsor tables back we’d have to do talks in some far-away rooms again, and removing those rooms this year saved us a lot of money and also people always hated it (like – FAR away).

Also, I’ll be honest, we had problems with sponsor misbehavior last year.  Silver sponsors claiming a table and standing behind it like a gold. Sponsors going out on the field (forbidden by UT). Sponsors trying to have food trucks park outside (also forbidden by UT police). Disruptive activity of a number of different sorts, requiring lots of work by organizers and volunteers and venue staff to deal with. I am sure many of them thought they were being “scrappy” etc. but in the end, we don’t get paid for this conference so we don’t need to put up with crap for it either. Discussion about “firing” certain sponsors was had.

We aren’t going back to the usual sponsor tables, but we are going to try something even more different – read on for that!

Boxed lunches

In early DoDA, we kept having super-deluxe Austin fare – BBQ, tex-mex – not from a caterer but from the real good places. This was for all the folks from away we were bringing in and wanted to show an Austin good time to!

Unfortunately, last year food lines for 650 people were a problem. Vendors weren’t adequately prepared with people or food.  We had to have many volunteers assigned. Food lines were super long and slow and a source of frustration.

This year we did have some comments about “I wanted the deluxe foods.” But they were far overwhelmed by those who appreciated being able to grab sustenance and get back to why they are here, learning and discussion. So with enough money we may try to get some kind of super-deluxe box lunch, but the box lunches will stay.

Lower headcount

The lower headcount was universally beloved except by lead generators and those who couldn’t get a ticket. More and better interaction, many positive comments noted the more intimate communication in openspaces and hallway track.  Keep.

No streaming

Worked out great.  No one complained, and the cost and org/volunteer time and schedule and stage compromises we have to make for live streaming are immensely negative.  Not going back.

2019 Planning

First of all, a disclaimer.  I am sharing this in the interests of transparency and helping other organizers learn from what we’ve done.  I don’t claim Austin is doing things the “one true way” and I know our community’s needs are different from many others. None of this is intended to denigrate any other events and their decisions. You don’t need to justify why you do things differently or why any of this isn’t right for your community.

Every year I start our planning with some basic questions.

  1. Do we want to have a DevOpsDays Austin next year?
  2. If so, why?  What is the goal of this year’s event?

“Inertia” is a bad reason to do anything.  We don’t have “money” as a reason because we have to spend what we get, we don’t pocket anything except some gifts. (My kid has already appropriated the bluetooth speaker I got this year…)

The group of organizers (over a tasty dinner at Chez Zee) decided “yes”, and after a good bit of discussion they decided that to us, this year, the goal of DevOpsDays Austin is to “Promote collaboration and sharing and networking specifically for the Austin technical community.” Now, that’s a pretty non-controversial statement on its face – but then as we plan stuff, we really test it against our goal and see if it supports it, is neutral, or takes away from it.  If it’s neutral or takes away, it goes.

This decision and clear statement (I think Marisa is who put it together for us) pricked my memory and I pulled out our attendee survey comments.  What did you like the most about DevOpsDays Austin 2018?  “Ability to collaborate with others.” “Enjoyed hearing what others were doing.” “Focus on the community.” “It’s a well-run, intimate conference.  I always see people I know.” “The community involvement.”  Her sentence crystallized what people were telling us was their favorite part of the event – super!

OK, so what does that mean for each area?

Content

People love the lightning talks more than anything.  Then the keynotes. Then the talks. It’s why we tried the attendee voting. The discussion covered how many of the talks seem too long and boring even at 35 minutes, and people trying to get too technical in them suffer from people not being able to follow along well due to screen size and large group.  People say they want themed tracks and stuff, but we rely on volunteers giving talks, we aren’t buying these off the shelf somewhere (“Give me 6 Kubernetes talks, 6 DevOps culture talks, 6 DevOps manager talks, and 6 intermediate level technical talks…”)  We are still committed to multiple technical tracks (DoDA was the first DoD to do this, many are still uni-track) because we’re 7 years in and we have a great diversity of experience in our community, and people don’t want to sit through the same messaging again.

Some talks are beloved and others aren’t.  As we sifted through the details, one comment from “What can we do better” on the attendee survey came to me.  “Talks focused on ‘I am a _____, here’s the problem we had and how we solved it.’ I say that because one of the coolest, most useful talks I saw was the Coinbase engineer who described how he used EBS volumes creatively to solve their scaling problem.”

So we decided to retire the voting but heavily curate the talks.  We don’t want “whatever talk you’re giving nowadays on the DevOps talk circuit” – we want talks in that format, the problem you had and how you solved it.

We’re working out the details, but we’re thinking about having these talks be more like 15 minutes long, with then linked openspaces that afternoon for the truly interested to get together and go ‘command line level’ with them.  This also allows for more breaks and collaboration time.

We also decided that idiosyncratic is better.  A couple of the organizers got excited about a sports/fitness theme to align with the stadium; one wants to set up a 5K, one has a wife that does yoga classes and we could have one, we can give fitbits as speaker gifts… While I and the other Agile Admins have been filming lynda.com courses and doing other creative things, the advice we keep getting from producers and directors and content managers is “Use *your* voice.  Do what *you* find interesting and other people will find it interesting.” Andrew Shafer loves running Werewolf games at openspaces at conferences, and people really respond to it! So we’re not going to hesitate to put stuff in we find interesting and we figure that enthusiasm will draw others. Trying to give attendees a “standard conference experience” is severely counterproductive because there’s plenty of regular conferences for people to go to, they get sick of it, and that doesn’t fit the devopsdays ethos in the first place.

Sponsors

I challenged the group.  “Tell me why we should have sponsors at all?  Half our revenue was ticket sales and half was from sponsors.  If we double ticket prices to $400 – still very low for any 2-day conference in the world – we can just not take sponsors at all, done and done. If we needed their money it’d be one thing, but we don’t. Let them spend their ‘limited marketing budget’ on the DoD events that do need it. How do the sponsors contribute to our goal other than with funding?”

The immediate response was that there are a bunch of sponsors who *are* part of the community and interacting with them is important; we have loads of Amazon/Google/Atlassian/Oracle/etc hiring going on here for example, and folks who work for Chef and Salt and Puppet and so on in town… We want those folks to be part of the conversation.  Just not disrupt that conversation.  And, some people pay for those tickets out of pocket so having some money to defray attendee costs is good.

We decided to try something different – we are using the luxury boxes at the stadium more and more; they’re relatively inexpensive and we used them for all the openspaces and such this year.   What if, we said, we intersperse sponsor suites with openspace suites, maybe even have them host some of the openspaces, do their own presentations in there too for whoever’s interested?  This means a limited number of sponsor slots (no more than 10, possibly fewer), but a more premium experience right there where the action is happening. And target Austin-presence companies to let them know about it. They can also then get food/drink catered into their suites to bring people in even more.

Attendees

Keep the headcount low – at least our limit of 400 from this year, if not lower. Consider a ‘two-tier’ ticket price with one price if your company is paying and another if you are; Data Day Austin has used this format to good effect.  Lets the non-backed solo folks in without breaking their bank but lets companies that do send attendees pay a reasonable amount.

Venue

UT Stadium is great, we don’t really see a reason to do all the work to change if we’re not doing booths and we’re going with a suite strategy for sponsors. Plus we have developed great relationships with the venue staff.

Keep refining the AV experience but doing it ourselves – we bought equipment and have a large set of “A/V geeks” so we don’t need to have outside people do it.

Food

Keep with boxed lunches. Austinites have had enough BBQ and tex-mex and this event is primarily for them per our goal. The benefit of fast lunch and snacks was tremendous this year. Could spend more on boxes from premium vendors but keep it boxed.  Maybe do drink service ourselves because we got truly rooked by the UT caterers on it this year.  Though Rich said he found the place the athletes eat and we might be able to get in on that… Keeping it fast, though, one way or the other.

Happy Hour

We put a lot of work into this and spend double what the happy hour sponsor gives us each year, and then only half the people come and only half of those say they like it.  This year we had unlimited food and booze at a venue with video games in it for Pete’s sake, I think we’re done chasing the idea of the ultimate happy hour. Probably we’ll do more of an onsite short sponsor room crawl at the venue, and then an “after party” we don’t put as much money/work into. “A couple free rounds at Scholtz’, get your own ass there.”

Conclusion

All right, that’s all the plan one dinner could get us.  But in the end, we’re happy with how the event went this year.  We’ll change a couple of the things that didn’t work out – talk voting, no booths – but not back to the old way because we already know that was suboptimal, instead we’ll try more options!  If you don’t have experiments not work out, you’re not being experimental enough, so we embrace that with DevOpsDays Austin.

Let us know your thoughts too!  Who are you, and what do you get or want to get out of DevOpsDays Austin?

Leave a comment

Filed under Cloud, DevOps

Tagged as ,

by | December 22, 2017 · 10:16 am

DevOpsDays Summit Austin 2018 – “DevOps Unplugged”

Hey all!  We’re starting work on next year’s DevOpsDays Austin – our seventh here in the ATX.  Many of you have come out to the event (or another of the great DevOpsDays around the world). Well, we have some changes in store this year!

Last year’s DevOpsDays Austin, “Monsters of DevOps” was bigger than ever and had a stadium rock theme – we had a huge venue,  all the DevOps VIPs we could pull down (including the first time all 4 authors of the DevOps Handbook managed to get together at an event), multiple content tracks, killer swag, great food, a hackathon, the best Happy Hour I’ve attended at a conference, we invited in and comped local user groups to give talks…  Part of our continuing trajectory to make DoDA more all encompassing and awesome.

But – every year we sit down and discuss vision before we launch into the conference.  What do we want to accomplish and why?  Who are we serving and why?  Why are we, personally, putting in huge amounts of unpaid work to serve the community? “Because it’s there and we did it last year” isn’t a good answer, so we like to really put some thought into it.

This time when we talked about it, first in our core group and then with the rest of the 2017 organizers, we realized that we’ve been concentrating on “bigger” but we’ve been putting more and more money and effort into the parts of the event that aren’t really of high DevOps value. Here in Texas, it’s easy to conflate bigger with better, since we’re both the biggest and the best!  But we’re not sure that’s right. Many of the more expert people we know here in Austin don’t really come out to the event any more, unless they are giving a talk or recruiting for their current gig.  Talks and openspaces have kept focused on introducing new people to DevOps, enterprise folks, “horses and donkeys,” and so on.

And as we talked, we said “Well – what do we personally get out of the conference nowadays as attendees?”  The answer was “not much.” Openspaces are huge and end up being a couple people talking.  Talks are either pretty familiar from the conference circuit or also designed for new folks.  We have more content but it’s more passive content, sit and watch.  It’s good for the newbies but not as much for the experienced folks.

We contrasted this to the first couple DevOpsDays we went to in Silicon Valley.  The first couple were just in a big auditorium at LinkedIn.  There weren’t any sponsor booths. More of the event was focused on the openspaces and interaction between the highly driven participants. We ate box lunches wherever we could perch in the parking lot outside – and swag was just a t-shirt.  Heck, the third one was in a weird abandoned building Dave Nielsen had access to, we had to carry our own chairs around to talks and the food and stuff was in a concrete-and-cage loading dock. But it’s those events we got the most out of.

Therefore, this year DevOpsDays Austin is going to go to what we call a “Summit” format.  We’re reducing the size of the event, and focusing more on local, motivated practitioners.  What does this mean?

  1. No sponsor tables.  We’d love sponsors to participate, but in recent years we’ve gotten more folks who have either just sent aggressive marketers, or sent people we enjoy and then locked then down behind tables. So we’ve come up with a sponsorship package that gets them exposure and value but lets them actually participate in the event.  Folks that just want to churn leads will self-select out.  The sponsorships are less expensive, and we’ll just have venue food etc. instead of premium.
  2. No preselected talks.  Well, OK, maybe we’ll have one keynote a day.  But I went to a ProductCamp here in Austin and they did something brilliant – they had a RFC but don’t do a final selection – finalists show up and the audience votes on what talks they want to hear (kinda like openspaces but more prepared).  This means people who say ‘well… I’ll come to your event if I can talk (or sponsor if I can talk, or…)’ will self-select out. You come because you want to be here, and you can give a talk!
  3. Smaller headcount.  We’re lowering the cap (including sponsors and organizers and volunteers) to 400. We’re going to get openspaces to be the kind of highly engaged discussions that make the so valuable.  We’re going to be up front with people that attendees are expected to engage.  DoD used to be the only thing around to learn from.  But now, if you’re an enterprise person that wants to have some DevOps talked at them – you have  variety of options now, like you can go to DevOps Enterprise Summit (also a great event), or to another DevOpsDays like the one in Dallas using the conference format, or one of a dozen events either completely DevOps or DevOps-tracked.  But for here in Austin this year, we need something where the unicorns can also have an event meaningful to them, so they can gather and refresh on what’s going on. Not to say only “unicorns” are welcome, but frankly we’d prefer people only come out if they intend to discuss, share, and engage; this will not be a passive-learning friendly event.
  4. No streaming.  Every year we put a lot of work and money into live-streaming and/or recording the event.  But it’s often problematic, and doesn’t get viewed a lot – there’s so much content out there now.  But even worse, we end up having to degrade the experience of real attendees around the requirements of broadcast – space, money, schedule, the presenter has to stay in a little box… So we’re not going to do it.  You want to participate – come out and participate.

But How Can This Work???

That was everyone’s initial reaction to this plan.  But that’s silly – it has worked.  We’re just doing things that DevOpsDays has already done, that ProductCamp has already done, and so on. It’s just not what’s become customary.  After the organizers had a little time for it to sink in, they all rallied behind it with a vengeance.

We’ve run the numbers and just the basic $200/head attendee fees can pay for the venue, basic food, and a shirt, even if we get zero sponsors.  (We won’t have zero sponsors, we just put our sponsor page up and someone bought in the first hour it was live.) As we get more funding we’ll pump up the event, but deliberately focus on the core experience of highly skilled techies learning from each other, instead of adding distractions.

How Dare You Dis My Format???

This is the format we’d like to try this year.  Other events will use other formats and that’s fine. Here at DoDA we try something different every year!  We were the first to have multiple content tracks (over the complaints of some purists).  We added a hackathon, we added a local user group track… Last year we went big with a vengeance, and it was cool.  Now we’re going to do more small and exclusive, and that’ll be cool.  Next year, it’ll be different. Whatever your event is doing, more power to you, don’t confuse us having a vision we believe in with us thinking you’re “wrong.”

Come on down!

We’d love to see everyone out at DevOpsDays Austin 2018!  Come ready to interact and share.  Come ready to give a talk, with the risk it won’t make.  Come sponsor your company, just you won’t have a table to lounge at. This change has gotten us excited about running our seventh DevOpsDays, and we bet you’ll love it!

11 Comments

Filed under Conferences, DevOps

Tagged as ,

by | February 27, 2015 · 7:25 am

Awesome Upcoming Austin Techie Events

We’re entering cool event season…  I thought I’d mention a bunch of the upcoming major events you may want to know about!

In terms of repeating meetings you should be going to,

  • CloudAustin – Evening meeting every 3rd Tuesday at Rackspace for cloud and related stuff aficionados! Large group, usually presentations with some discussion.
  • Agile Austin DevOps SIG – Lunchtime discussion, Lean Coffee style, at BancVue about DevOps. Sometimes fourth Wednesdays, sometimes not. There are a lot of other Agile Austin SIGs and meetings as well.
  • Austin DevOps – Evening meetup all about DevOps.  Day and location vary.
  • Docker Austin – First Thursday evenings at Rackspace, all about docker.
  • Product Austin – Usually early in the month at Capital Factory. Product management!

3 Comments

Filed under DevOps

Tagged as , , , , , , ,

by | July 1, 2014 · 9:51 am

DevOpsDays Silicon Valley 2014 Day Two Notes

Day Two

spam First, a public shaming. Some goober from Merantis left this spam on everyone’s car. Bah.

Presentations

@bridgetkromhout spoke on “how I learned to stop worrying and love devops” and @benzobot spoke on onboarding and mentoring apprentices. DoD SV certainly made a strong effort to get more female speakers this year!  We tried in Austin (I personally wrote like every local techie woman group I could find) but we only had like one.

Then there were two super bad ass presentations back to back. I can’t find the slides online yet.

The Future of Configuration Management

Mark Burgess (@markburgess_osl), aka “The CFEngine Guy” and noted Promise Theory advocate, spoke.  Chef and Puppet had eclipsed CFEngine for a while but it turns out as the Internet of Things and containers and stuff are arriving that maybe many of his design decisions were actually prescient and not retro. Here it is broken down into wise sayings.

  • Why do we not have CAD for IT systems?
  • Orchestration is not bricklaying.
  • We need the equivalent of style sheets for servers.
  • We are entering a world of decentralized smart infrastructure.
  • Scale, complexity, and knowledge increase as our desire for flexibility increases.
  • Separation of concerns adds complexity and fragility.
  • To handle complexity – atomize and untether.
  • 3D printed datacenters are coming.

DevOps as Relationship Management

James Urquhart (@jamesurquhart) spoke about the interconnectedness of our systems. The SEC, post flash crash, added circuit breakers, defined rollback protocols, inserted agents into the flow of the stock exchange trading systems to prevent uncontrolled cascading.

One simple rule – visualize the whole system (monitor your outside relationships) but take action at the agent level. “How are you doing today?” “Good.” Monitoring is going well, new approaches in the space look at policies and interactions and performance and business medtrics – but need to differentiate reductionist vs expansionist approaches.

Michal Nygard’s book Release It! is full of great patterns, and Netflix’ open sourced Hystrix is an example of the kind of relational system safeguards you can build off it.

Ignite Block

  • Tips for Introverts (at Conventions) by Tom Duffield – They include find a role, don’t fear failure, attend preconference activities, go to lunch early and sit, engage, share interests, find a comfortable setting, take time to recharge. As someone initially introverted myself (no one believes that now) I like that this has actual tips to get past it; in some circles “introversion” has become the new “Asperger’s” as a blanket excuse for not wanting to bother to relate to people.
  • Mike Place on scalable container management – Google kubernetes is an example. Don’t just provision your systems, you need to manage them too. Images came and went and came back now, but you also can’t ignore what’s onboard the image. It’s time to join image and config management.
    This was really good and the world should listen. On the one hand, conducting CM operations on 1000 servers in parallel is contributing unnecessarily to the heat death of the universe.  On the other hand, you need to build those images in a non-manual way in the first place! And too many systems worry about the configuration but not the runtime operation. Amen brother!
  • Finally (well, there were two more, but I didn’t care for them so took no notes), John Willis (@botchagalupe) did [Darwin to] Deming to DevOps, a burst-fire reading list of nondeterminism tracing from Darwin through various scientists to the Deming/TPS stuff through into the DevOps world with Gene Kim and Patrick Debois.  It was pimp. Here it is when he gave it at another venue:

Conclusions

Here’s some big themes from the week.

  • Deterministic, reductionist, and centralized are for suckers.
  • Complexity is the enemy.  Systems thinking is necessary.
  • We love continuous deployment.  But DevOps is not just about delivering code to production.
  • Women exist in DevOps and are cool.  More would be great.
  • Most vendors have figured out to just relax and talk to techies in a way they might listen to.  Some haven’t.

It was a great event, kudos to Marius and the other organizers who put in a lot of work to wrangle 500 people, nearly 30 sponsors, food, venue, and the like.  If you haven’t been to a DevOpsDays, look around, there may be one near you!  I help organize DevOpsDays Austin (just had our third annual) and there’s ones coming this year from Tel Aviv to Minneapolis.

If you went to DoD SV, feel free and comment below with your thoughts (linking any posts you’ve made, slides, etc. is welcome too)!

Leave a comment

Filed under Conferences, DevOps

Tagged as , ,

by | June 30, 2014 · 5:05 pm

DevOpsDays Silicon Valley 2014 Day One Notes

I have more notes from Velocity, but thought I’d do DevOpsDays first while it’s freshest in my brain. This isn’t a complete report, it’s just my thoughts on the parts I felt moved to actually write down or gave me a notable thought. More notes when I was learning, less when I wasn’t (not a reflection on the quality of the talk, just some things I already knew a bit about).

DevOpsDays Silicon Valley 2014 was June 27-28 at the San Jose Computer Museum. 500 people registered; not sure how many showed but I’d guess definitely in excess of 400.

Day One

State of the Union

First we had John Willis (@botchagalupe) giving the DevOps State of the Union. Here’s the slides (I know it says Amsterdam, he gave it there too.) This consisted of two parts – the first was a review of Gene Kim et al’s 2014 State of DevOps Report – go download it if you haven’t read it, it’s great stuff.

The second part is about how we are moving towards software defined everything – robust API driven abstractions decoupled from the underlying infrastructure. John’s really into software defined networking right now as it’s one of the remaining strongholds of static-suckiness in most infrastructures. A shout out to the blog at networkstatic.net and tools like mesos and Google’s kubernetes that are making computing even more fluid (see this article for some basics). “Consumable, composable infrastructure.”

Saying No

Next, our favorite Kanban expert Dominica DeGrandis (@dominicad) spoke on “Why Don’t We Just Say No?”  Here’s the slides. As a new product manager, and as a former engineering manager who had engineers that would just take on work till they burst even with me standing there yelling “No!  Don’t do it!”, it’s an interesting topic.

Why do you take on more work than you have capacity to do? She cites The Book Of No by Susan Newman, Ph. D and a very recent Psychology Today “Caveman Logic” post called Why So Many People Just Can’t Say No. She proposes that it is easier for devs to say no; ops have more pressing demands and are forced into too much yes. Some devs took exception to this on twitter – “our product people make us do all kinds of stuff we don’t like to” – but I think that’s different from the main point here. It’s not that “you have to do something you don’t like and are overruled when you say no” but that “you become severely over-committed due to requests from many quarters and being unwilling to say no.”

She goes through a great case study of changing over a big ops shop to a more modern “SRE” model and handle both interrupt and project work by getting metrics, having a lower WIP limit, closing out >90 day old tickets, and saying no to non-emergency last minute requests. In fact, the latter is why I prefer scrum over kanban for operations so far – she contended that devs have an easier time saying no to interrupt work because of the sprint cadence.  OK, so adopt a sprint cadence! Anyway, by having some clear definitions of done for workflow stages they managed to improve the state of things considerably. Use kaizen.  The book about the Pixar story, Creativity Inc., talks about how the Pixar folks were running themselves ragged to try to finish Toy Story, till someone left their baby in a car because they were too frazzled. “Asking this much of people, even when they wanted to give it, was not acceptable.” What should your WIP level be? The level of “personal safety” would be a great start!

It’s interesting – I did some of these things at Bazaarvoice and tried to do some other ones too. But often times the resistance would be from the engineers that the current process was working to death.  “We can’t close those old tickets!  They have valuable info and analysis and it’s something that needs to happen!” “Yes, but our rate of work done and rate of work intake proves mathematically that they’ll never get done. Keeping them open is therefore us making a false promise to whoever logged those tickets.” Not everyone is able to ruthlessly apply logic to problems – you’d think that would be an engineer attribute but in my experience, not really any more than the general population. But given that “not acceptable” quote above, I really struggled with how to get engineers who were burning themselves out to quit it.  It’s harder than you’d think.

Agile at Scale

Next was a fascinating case study from Capital One’s transformation to an agile, BDD, devops-driven environment given by Adam Auerbach (@bugman31). The slides are available on Slideshare.  They used the Scaled Agile Framework (SAFe) and BDD/acceptance test driven development with cucumber as well as continuous integration. In a later openspace there were people from Amex, city/state/federal governments, etc. trying to do the same thing – Agile and DevOps aren’t just for the little startups any more! He reported that it really improved their quality. Hmm, from the Googles it looks like the consulting firm LitheSpeed was involved, I met one of their principals at Agile Austin and he really impressed me.

Sales and Marketing Too

Sarah Goff-DuPont (@devtoolsuperfan) spoke about having sales and marketing join the agile teams as well.  Some tips included cross-pollinating metrics and joining forces on customer outreach.

Ignites

Just some quick thoughts from the day one Ignites.

  • @eriksowa on OODA and front end ops and screaming at your team in German (I am in favor of it)
  • Aater (futurechips) on data acquisition and multitenancy with docker
  • Jason Walker on LegoOps
  • Ho Ming Li on Introducing DevOps
  • @seemaj from Enstratius on classic to continuous delivery – slides. Pretty meaty with lots of tool shout-outs – grunt, bowler, angular, yo, bootstrap, grails, chef, rundeck, hubot, etc. I don’t mind a good laundry list of things to go find out more about!
  • Matt Ho on Docker+serf – with Docker there is a service lookup challenge. AWS tagging is a nice solution to that. Serf does that with docker like a peer-to-peer zookeeper. Then he used moustache to generate configs. This is worth looking at – I am a big fan of this approach (we did it ourselves at National Instruments years ago) and I frankly think it’s a crime that the rest of the industry hasn’t woken up to it yet.

Openspaces

If you haven’t done openspaces before, it’s where attendees pitch topics and the group self-organizes into a schedule around them. Here’s some pics of part of the resulting schedule:

openspaces3 openspaces1 openspaces2

I went to two.  The first was a combination of two openspace pitches, “Enterprise DevOps” and “ITIL, what should it be?”  This was unfortunately a bad combination.  Most folks wanted to talk about the former, and the Capital One guy was there and people from Amex etc. were starting to share with the group. But the ITIL question was mostly driven by a guy from the company that “bought ITIL” from the UK government and he had a bit of a vendory agenda to push.  So most of the good discussion there happened between smaller groups after it broke up.

The second was a CI/CD pipeline one, and I got this great pic of what people consider to be “the new standard” pipeline.

Generally Accepted Continuous Integration and Delivery Pipeline

Generally Accepted Continuous Integration and Delivery Pipeline

Next, Day 2 and wrap-up!

Leave a comment

Filed under Conferences, DevOps

Tagged as , ,

by | June 23, 2014 · 12:04 pm

Meet The Agile Admins At Velocity/DevOpsDays Silicon Valley!

Three of the four agile admins (James, Karthik, and myself) will be out at Velocity and DevOpsDays this week. Say hi if you see us!

James will be doing a workshop with Gareth Rushgrove on Tuesday 9-10:30 AM, “Battle-tested Code without the Battle – Security Testing and Continuous Integration.” Get hands on with gauntlt and other tools! [Conference site] [Lanyrd]

Ernest is doing a 5 minute sponsor keynote on Thursday, “A 5 Minute Checklist for Application Monitoring.” OK, so it’s during the USA vs Germany game – come see me anyway!  I hate keynote sales pitches so I’m not doing one, I’ll be talking about a Lean approach to monitoring and stuff to cover in your MVP. There’s a free white paper too since what can you really say in 5 minutes? And so you know what to expect, the hashtag you’ll want to use is #getprobed! [Conference site] [Lanyrd]

Leave a comment

Filed under Conferences

Tagged as , , , , , ,

by | March 17, 2014 · 6:50 pm

DevOpsDays Austin Is Coming!

The third annual DevOpsDays conference in Austin will be May 5-6 (Cinco de Mayo!) at the Marchesa, where it was held last year! As many of you know, the DevOpsDays conferences are a super popular format – half talks from practitioners, half openspaces, all fun – held in many cities around the world since the first one in Ghent launched the DevOps movement proper.

  • You can register – all the early bird tickets are sold out but the regular ones are only half gone.
  • You can also propose a talk!  There’s 35-minute full talk slots but we’re even more in need of 5-minute Ignite! style lightning talks! RFP ends 3/26 sp
  • You can sponsor! The Gold sponsorships are half gone already. And we have some special options this year…

DevOpsDays Austin has been bigger and better every year since its inception and should have something good for everyone this year. Come out and join your comrades from the trenches who are trying to forge a new way of delivering and maintaining software!

1 Comment

Filed under Conferences, DevOps

Tagged as , , ,

by | June 22, 2013 · 12:17 pm

DevOpsDays Silicon Valley 2013 Day 2 Liveblog

Woooo!  Last day of a week of conferencing.  DevOpsDays Day 1 was good and I have even more openspace topics I plan to propose next time.  As usual this is being livestreamed and will be viewable later as well at bmc.com/devops.

Sponsor Watch… Got to talk to our friends at PagerDuty (alert management) and Datadog (monitoring/dashboarding), we use them and love them. And I got to see Stormpath again, they first showed up at last DevOpsDays with a SaaS hosted auth solution (not like PingIdentity and Okta, they actually store the usernames/passwords for you, Les Hazlewood the Apache Shiro guy started it) and they’re growing quickly. Also talked to SaltStack which does salt, a remote command execution framework. 10gen was here with a MongoDB SaaS backup solution (nice!) and monitoring solution.

Leading the Horses to Drink

By Damon Edwards (@damonedwards) from DTO and now #SimplifyOps.

How to spread DevOps in enterprises.  There’s silos you know.  The term DevOps may work against you – it’s evangelical and being overused/washed already.

There is no ‘why’ other than the why of the business. Read your Deming/Collins/Four Steps to the Epiphany/etc.

Go ask people… Something.

Develop a common DevOps vision. Not a process because they’ll get blinders on. [Ed: I believe this is a false dichotomy – you should teach both. Vision without process lacks focus and process without vision lacks direction.  It’s like accuracy and precision.]

  1. See the system
  2. Focus on flow
  3. Recognize feedback loops

Do a value stream mapping – read Learning to See.  OK, this is the meat of the preso – very hard to read though.

Take your information flow and turn it into an artifact flow

Do a timeline analysis, find waste

Metrics.  Establish the metric chain of what matters to the business, driven down to a capability which influences what matters to the business, and driven down to an activity over which an infividual can cause/influence outcomes.

Doesn’t require saying “devops.”

  1. Teach concepts
  2. Analysis
  3. Metrics chains
  4. Do something
  5. Iterate

Only takes like 3 days to bootcamp it. Then put in continuous improvement loops.

You can only break silos by brute-force being the boss, but misalignment will reassert itself. Have to change the alignment.

Q&A: Do it with everyone in the same room with whiteboards/postits, it works better than getting fancy

Beyond the Pretty Charts

Toufic Boubez from Metafor.  Cofounded Layer 7 and escaped when CA acquired them.

Came from a popular DOD Austin Openspace – see the blog post!

  1. We’ve moved beyond static thresholds – or, at least, everyone thinks they suck. Need more dynamic analytics.
  2. Context is important – planned and known (or should be known) events cause deviation. Correlate events with metric gathering.
  3. Don’t just look at timelines. Check the thinking round Etsy’s Kale and Skyline, many eval methods assume normal metric distribution and that’s uncommon. Look at a histogram of any given data – like latency is usually gamma not gaussian.
  4. Is all data important to collect? There’s argument over that.  Get it all and analyze vs figure out what’s important to not waste time.
  5. We all want to automate. Need detection before it’s critical. Can’t always have a human in the loop. Whipping out the control theory – open loop control systems, closed loop – to get self healing systems we need current state/desired state diffing from our monitoring systems and taking action. [Ed. We experimented with this back at NI, we had Sitescope going to a homegrown system called “monolith” that would take actions. Hard to account for all factors though and eventually was discontinued.] Also supervised vs unsupervised loops [Ed: – we might have kept monolith around if it SMSed us and said “memory is high on this server I believe I should restart the java process, is that OK” and we could PagerDuty-like say yea or nay.]

How much data do you need?  No more res that twice your highest frequency (Nyquist-Shanon). Most algorithms will smooth/average/etc.

Q&A: Are control systems more appropriate for small not large systems?  No – just like in industry, as long as you design for that then it’s not just for toys.

And now I step in for the vendor pitch for Riverbed.  Agile Admin Peco left yesterday and the other Riverbed booth guys made themselves scarce, so I did their shout-out for them. They have Zeus EC2 LBs, Aptimize web front end optimizer, and Opnet Appinternals Xpert APM tool!  Very cool.

Identifying Waste in your Build Pipeline

Scott Turnquest from Thoughtworks

Tools: Value stream mappings, fishbone analysis, “5 Whys”

So how do we do that value stream mapping? Here we go!  [Ed: Oh, this is nice, I was sad that in the DTO presentation they mentioned them and threw some up but didn’t really dive down into one.]

A day of analysis of one small feature –  a day of wait, 4 days of dev, 2 mins of wait, 1 hour of acceptance tests, 4 hours of deploy, 1 day in staging, 4 hours to deploy to prod. Note the waste areas – “4 days in dev?  Really?” and the long ass deploy windows [Ed: Our value stream looks depressingly like this.] Process cycle efficiency of 75% (value creation time/total time)

So to determine the source of those waste areas, use the fishbone diagram. Had long feedback cycles from structure of code and build/deploy pipelines. Couldn’t test w/o AWS and can’t test individual components, provisioning was serial and repos were flaky.

Fix underlying cause (most impact first) – deploy pipelines. Reduce failure rate of deployments. Half were failing, and failing slow. Moved to AMI baking for reliability. [Ed: They said I was crazy a couple years ago when I said this, “no it’s a foil ball…” Bake when you can!] So this got them from 4 hours to 2 hours, and then parallelized and got down to 25 minutes. This cut down the staging and prod deploys but also the dev time. Process cycle efficiency up to 83%.

5 Whys root cause analysis method. Figured out manual hard to automate deployments were at the root, automated them – don’t be afraid to restructure/redesign when complexity gets in the ways.

Analysis techniques are not just for analysts!

Read Jez Humble’s “Continuous Delivery”, Poppendieck’s “Lean Soft Dev”/”Implementing Lean Soft Dev” , Derby/Larsen “Agile Retrospectives”

Clusters, developers, and the complexity in Infrastructure Automation

Antoni Batchelli of PalletOps. Complexity, essential and accidental. Building a system is simple but the systems are complex at runtime, and “complexity of a system is the degree of difficulty in predicting the properties of the system given the properties of the system’s parts.”

In DevOps we see infrastructure-aware software and concepts moving up into dev processes.

Devs want to run “their own” cluster with all the setups they need – productionlike, but with specific versions/timings/data/code/etc. Don’t care about infra details but want consistent envs/code.

Software has to be infrastructure aware now to autoscale, self-heal, etc. The app is the best informed actor to make/orchestrate infra decisions.

[Ed: This late into a conference week, I get a little irritated about presentations that are not really clear *why* they are telling you what they’re telling you.]

He hates incidental complexity. Me too.

OK, maybe we’re getting to a thesis. Let people solve problems where they are less complex: at the right level of abstraction. Build layers of abstraction – infrastructure, OS, services, actions. Make them into modules, make them functional and polymorphic.

Ignites!

James Wickett (@wickett) on Rugged DevOps and gauntlt for security + DevOps. gauntlt is a gem for continuous security testing as part of your build cycle. BDD your app’s security! Knock Out!!! go to gauntlt.org to get started.

Karthik Gaekwad (@iteration1) on DevOps Culture in the CIA. Devops is culture/automation/measurement/sharing. Seen Zero Dark Thirty? Well, the true story behind that details the COA’s transformation from a split between analysts and operatives especially using Sisterhood, a group of female analysis tracking Bin Laden since 1980. Post 9/11 there was a mass reorg to become more tactical – analysts became Targeters and worked with Operatives hand in hand. Same kind of silo busting. The Phoenix Project is Zero Dark Thirty for DevOps!

Dave Mangot (@davemengot) for DevOps Do’s and Don’ts from Salesforce.  Do give everyone the tools they need to do their jobs. Don’t make ops the constraint, Do lots of communicating. Don’t forget to include everyone. Do get ops involved early. Don’t create a front door (loaded) process. Do have integration environments, Don’t forget config management. Do have blameless post-mortems. Don’t use the Phoenix Project as a bludgeon. Do use Agile as a cultural tool. Don’t rely on tools to change culture. Do get executive sponsorship. Don’t do shadow IT. Do use Damon Edward’s levers. Don’t just lecture, it’s a participation sport. Do structure the org around delivery. Don’t make separate DevOps teams or jackets. Do get the whole company involved, DevOps is for everyone.

Jonathan Thorpe – Preventing DevOps success. Not planning for scale. Not having unit tests. Not designing automated tests to scale. Not managing your capacity. Not using your resources effectively. Not using same deployment process for all environments. Not knowing what/where/when/who (activity tracking). Getting covered in ants.

DevOps is the future – John Esser from ancestry.com. What keeps CIOs up at night? Besides ants? IT. Need time to value. Transform mindset/processes/tools/etc. Strangler pattern.

DevOps productivity survey by Oliver White from ZeroTurnaround. DevOps oriented teams spend more time on infrastructure improvements and less on firefighting and support. Problem recoveries are shorter. Release software faster. use more custom tools. Make love for longer time. @rebel_labs

Nathan Harvey on leveling up your skills. Quit!  Go to a conference. Try new things. Do a project somewhere. Always be interviewing.

Leave a comment

Filed under Conferences, DevOps

Tagged as , , ,

by | June 21, 2013 · 12:07 pm

DevOpsDays Silicon Valley Day 1 Presentations

All right, the corporatey part of the week (Velocity) is over, and the tech Illuminati have stayed for DevOpsDays Silicon Valley (used to be Mountain View) – with like 500 people!

The hashtag is #devopsdays and all the presentations was live streamed at the usual place for DevOpsDays live streaming, www.bmc.com/devops.  The videos are now all up on Vimeo.

To open, a funny Point Break DevOps parody!  Ah, makes me want to watch that movie again.

DevOps + Agile = Business Transformation

The first talk is from Jesse Robbins (@jesserobbins)!  Ex-Velocity co-host and co-founder of Opscode, he has now found a home in the TECH UNDERGROUND which is DevOpsDays. He started Velocity because he couldn’t share all the secret stuff they were doing at Amazon but knew it was so important and crucial to the Web and thus the world… Sometimes frighteningly so.

DevOps, he says, is the ability to consistently create and deploy reliable software to an unreliable platform that scales horizontally. The right tools and culture are critical to doing this successfully.

The Internet is becoming pervasive.  Applications became customer service vehicles. Walmart and Amazon both understand this. Email killed the post office. These rips in the social fabric reveal something better. These changes are coming faster and faster and the technology that does this is ours – we build and run it.

Misaligned incentives cause conflict. “Operant conditioning.” People know what the good guys are doing, but they just can’t change themselves to do it – elephants can’t fly just by flapping their ears harder.

You can do your keep-it-small DevOps effort, but eventually you have to say “if we don’t do this everywhere we will fail” – that’s not a business or technology problem, it’s a culture problem.  He’s given this speech inside a bunch of organizations and knows how much resistance there is to change because they all wriggle around like itchy bear cubs when he says it.

Circuit City’s downfall and Blockbuster’s downfall due to Netflix are examples of cultures making agility impossible.  You can’t “agile out” of that. You can provide tools and culture but the overall foundation has to spread. True story, Blockbuster decided the brilliant way to get out of its death spiral was to buy Circuit City, which was also in a death spiral.  And “make it up in volume” I guess.  Is “being a meathead” a culture problem? I reckon.

Conway’s Law – you make things that are copies of your org structure.

Fundamental attributes of successful cultures:

  1. shared mission and incentives
  2. infrastructure as code
  3. application as services
  4. dev+ops+all as teams

Successful practices:

Full stack automation, commodity hw or cloud, reliability in the software, infrastructure APIs, code infra services – infra as product, app as customer.

Service orientation. versioned APIs, resiliency (design for failure), storage abstraction, push complexity up the stack, deep instrumentation

Agile, trust basis, shared metrics and monitoring, incident management, service owners on call, tight integration (maybe you end up with dedicated network or sec oncall, like SREs,  but at the core still collaborative), continuous integration, SRE/SRO to spread concepts, game days.

It takes time – amazon.com didn’t switch to EC2 till Nov 10, 2010.

Changing culture:

  • Start small, build trust & safety
  • Create champions
  • Use metrics to build confidence
  • Celebrate successes
  • Exploit compelling events – cause moments of openness

Continuous Quality: What DevOps Means for QA

By Jeff Sussna, @jeffsussna.

Old definition of quality – “does the software meet the spec?” But agile is about delivering value and cloud is about turning software into services. New definition of quality is “does the service help customers accomplish their jobs-to-be-done.”

A restaurant isn’t just about food delivery, there’s a lot of value creation in the whole chain. A service needs functionality, operability, deliverability, coherency (does it engage me throughout my journey).

New approaches include user-centered design, test-driven development, continuous delivery, MTTR over MTBF; build in testing and learn from failure.

So QA changes. Boundaries blur and automation takes over the manual activities. New and more valuable role: represent the “service not software” perspective, watchdog those 4 attributes.

QA engineers need to lift their gaze above the mechanics of testing, treat tests as code, focus on building quality into the system (quality advocate). New skills include understanding and thinking about service (e.g. outage comms), ops (sec, monitoring), process/automation

[Ed: If we add all this devopsy stuff into the definition of done, QA should look at all of it.]

Good testers see systems and their prts (and gaps), ask probing questions, design good tests, engage that proficiency in design and test plan critiques.

So we need this new kind of testing as well as the old kinds so with continuous delivery how do we catch up?  And people give a lot of “buts” about automated functional testing, But there are frameworks and DSLs that allow you to make changeable, encapsulated testing.  And it’s a process problem – no one asks what it’ll take to test the system. Write code and tests together, commit them together.

Operability and deliverability need testing. Design for internal users too…

You still want QA (instead of obsoleting them) as attached to the customer and as an antidote to confirmation bias.

Continuous Quality – everyone is  testing all the time, quality infused, QA is a mirror for the organization. There is still specialization.

Shout out to @guidostompff of designinteams.com.

Is your team instrument rated?

By J. Paul Reed (@SoberBuildEng) – also see the podcast theshipshow.com!

Culture. Is it hugs and beer? No, it’s incentives + human factors.

Why aviation as a DevOps analogy? It progressed from craft to trade to science to industry. DevOps is in th e”late trade” phase of that development.

Incident response is good but the house is already on fire by then. In aviation there’s a lot of scale and you want to avoid the incidents in the first place.

Learning to fly – first, you learn visual flight rules. Use your eyeballs. Then you move to instrument flight rules – flying in the system.

Flying by instrument relies on standardization, communication (precision), expectations (responsibilities in a situation), remediation. It is not static, blindly relying on automation or process, or fun-verboten.

How to get there?  Define your current process even if it’s weird, focusing on operational requirements, derive primitives, define operational dictionary, and make sure the nonfunctional requirements) are owned.

Formalize roles, responsibilities. There should be clear transfer of control on who “has the ball.” Drill/train and delegate. Priority classes. Fly|navigate|communicate.

Understand your org’s limitations.

Holding patterns/WIP are bad because it adds chaos to the system.

Investigate outcomes. Should you have an external team investigate? “No blame” postmortems aren’t about not being a jerk and making people sad, but  because it’s very unlikely a failure is “one guy’s fault” and it’s a red herring to think so. [I made a lovely “Root cause is a myth” custom t-shirt at the con! -Ed.]

You should have a day-to-day operational model that accounts for incentives and the human factors that make people able to deliver on them.

Leveling Up a New Engineer in a Devops Culture; Healthy Sustainability

By Gary Foster and Mercedes Coyle from Scripps

You want to hire a new engineer, teach them “our way,” inculcate a devops mindset from the beginning, add good practices and training to the local labor pool, and pay it forward.

Identify needs and outcomes desired and get a mentor.  THEN go hire! They go to hackathons and stuff to hire. Incubators, boot camps (e.g. Hackbright).

And now the new engineer! She was looking for a place where she could get up to speed quickly, support and challenge but no hand holding, senior engineers to help a new engineer grow. She had basics skills in coding/testing/deploying and willingness to learn.

What to do on the job as a new person? Question what you don’t understand, avoid perfectionism, and speak up.

The mentor’s responsibility is patience, giving them responsibility like seasoned engineers, ask them for ideas, teach problem solving not syntax and don’t give the answer.

So train ’em, listen to ’em, form a cult around ’em. Take responsibility for bad habits.

Ignite Time!!!!

Adrian Cockroft of Netflix (@adrianco) on beer pineapples and bottlenecks. “Cockroft headroom plot” helps you see when there’s serialization due to a bottleneck.

Peco!!! @bproverb on how we have an incident driven culture and effectively reward failure.  “Actionable alerts” are reactive and often we have sparse bad data. Analyze and track close calls, reward for prevention. Spend time with your data. No need to theorize if you have data, you can track close calls and pursue root cause. Find analyst ninjas. Close call focused analysis.

David Hatten from UrbanCode/IBM. The positive powers of negative thinking. And nihilism. And criticism.  Somebody needs a nap. Read Be Nice To Programmers.

Chantell Smith from ITSM Academy (subbing in for Jayne Groll) – what is devops culture? a multicultural society of frameworks and tools and standards and whatnot. There’s evangelists and detractors. Need communication to get over the cultural divide. Use the “git r done” scrum, not just for devs. Pair with a kanban board. ITSM is still a good thing and not lame! Let’s get a common dictionary/vocabulary. [Ed: So Gene and co., stop slacking on the devops cookbook!]

Systems theory for enjoyment of AWS – read John Gall’s Systemantics/The Systems Bible. Systems in general work poorly or not at all. Complex systems are always broken somewhere. Some simple services don’t even really work… Start simple and working and grow to complex and working. @whirlycott from Stackdriver (Philip Jacob)

Openspaces

I attended three openspaces on Day 1 afternoon.

Women in DevOps

The first was on getting more women into DevOps/related tech jobs. There were a lot of people and so we didn’t get too deep into any specific area of that. It was noted that benefits and especially maternity leave were super important and a good thing to stress in your job postings. Also that women are likely to not apply to”you must know all these 20 things” job descriptions. Though I’ve known guy engineers that fall into the same trap.  “I only meet 19 of those, I’d best not apply.” Hint as a hiring manager – if you meet like half of the things, you’re well advised to put a resume in!

We churned a bit over the fact that largely, people are hiring by exercising their known-people networks, and since historically more engineers are male, that tends to be self-reinforcing. You can go deliberately look into female-tech boot camps and the like.

In the end, I think the core problem is that we’re working at a fast pace. We put out a job posting (or a call for DevOpsDays presenters, as was brought up as an example) and we look through the responses we get.  If there’s no responses from women, then we can’t include them. But to break through that, we have to take the time to deliberately reach out (and figure out where to reach out to).

There was some talk about “wiping identifying information off” but I think that’s a blind alley.  I’m personally more likely to interview/etc. a woman or minority for an engineering position to try to level the playing field, if all the resumes are “Candidate 26” then so much for that.  I mean, maybe it’s true there’s a lot of old school tech companies out there who are like “wimmen on the front lines with us? Never!” but I have to say I’ve never seen that.

My main takeaway was that we should probably get the female engineers we have on staff, ask them to super-plumb their social networks, and get their views on what aspects of job descriptions/interviews/work environments are or are not attractive to them and double down on it.

Where the Hell are the Product Managers?

The second was one I proposed, entitled “Where the Hell are the Product Managers?” DevOps is nominally about bringing Ops into the agile team that is already a mashup of Product, Development, and QA. But unfortunately, despite 10 years post-agile manifesto, I find that healthy PM embedding into the agile team is honored more in the breach than in the observance.  Furthermore, in terms of owning “nonfunctional” requirements, or God forbid, an entire platform-type product, they tend to not want to do that.

We had a good discussion; some people had good PM engagement and others didn’t. Few had success with PMs doing effective prioritization of nonfunctional requirements and most “platform teams” didn’t have a PM, though some did and reported that it was super awesome. In fact, Bryan Dove from here at Bazaarvoice talked about one team he worked where the designer and marketing person came to colocate with the team as well and it was very effective.

The main takeaway was to continue to try to push the agile practice of crossfunctional, embedded and ideally colocated teams, because the results are so much better. And if one needs to hire more X (PMs, Ops, whatever) so that there can be one per product team, do it.

Running a DevOpsDays

The third was about running a DevOpsDays event.  Since I helped run DevOpsDays Austin I went to that to share the love.  If you’re looking to run one, we’ve made our budget and planning docs and everything available for others to crib from. My short playbook is:

  1. Get around 8 people as organizers, from a mix of companies.  2 will punk out and the other 6 will be able to share the load.
  2. Find a venue, that’s the most important thing.  It’ll give you a capacity and whether you’re planning on charging. We did a free DevOpsDays and had a large (>30%+) no show rate, and then did a $120 DevOpsDays and had a small (<10%) no show rate.
  3. Don’t get fancy.  Nail the hard requirements and then if you get excess sponsor money, add on other goodies.  For DOD Austin we added a band and a movie and more swag and more snacks later as our bank account swelled, but we could have cut off after venue/internet/some food and been done with it.
  4. BMC loves to do the A/V and stream the event! But beware, once they leave after the morning events you’ll be without mikes and stuff.
  5. Patrick Debois has the usual schedule/format for you to use.
  6. Don’t worry about sponsor money, they’re lining up to pay you. It’s more important to set expectations – this isn’t a “high traffic sales leads” event and you don’t get the attendees’ emails – it’s better to send engineers than salespeople, you’re trying to affect influencers.

And that’s Day 1, expanded!

Leave a comment

Filed under Conferences, DevOps

Tagged as

by | January 14, 2013 · 10:38 pm

DevOpsDays Austin 2013 Is Coming!

devopsdaysaustinIt’s been quiet here on the blog but we’ve been busy… And one of those items of busy-ness is setting up DevOpsDays Austin 2013!

Many of you came to DevOpsDays Austin 2012, the largest super-awesome Austin DevOps event ever!  Well, it’s even bigger this year.  Registration is open for DevOpsDays Austin 2013! Sign up quick to be assured a spot.

It’ll be April 30th and March 1st at the Marchesa in the middle of Austin. Because we had to pay for a venue this time to let more people attend, and to prevent no-shows from taking up the limited slots, we’ve instituted a $120 early bird fee for the event, which covers the food/venue/shirts for both days.

Proposals are open too, as are opportunities for companies to sponsor – the more sponsorships, the more cool activities and goodies for everyone involved!

I had a blast at DevOpsDays Austin in 2012 and this year stands to be huge.  Come on out and share expert tips with other elite DevOps practitioners from around the world! Patrick Debois and a growing list of other respected DevOps ninjas will be in attendance.

Email organizers-austin-2013@devopsdays.org with questions!

Leave a comment

Filed under Conferences, DevOps

Tagged as , ,