Tag Archives: austin

DevOpsDays Austin 2018 Retrospective and 2019 Prospectus

logoAll right, DevOpsDays Austin 2018 went great and the organizers (thanks be unto them – James Wickett, Dan Zentgraf, Boyd Hemphill, Richard Boyd, Scott Baldwin, Lee Thompson, Karthik Gaekwad, Marisa Sawatphadungkij, Ian Richardson, Bill Hackett, Chris Casey, Carl Perry, and our ConferenceOps finance handler Laura Wickett) have had the time to do a retrospective and both share what we’ve learned and set a course for next year’s event! This is long and I assume mostly of interest to other DevOpsDays organizers, so buckle in.

DoD Austin this year was another experimental year. Austin was the third DevOpsDays city in the US and the eleventh globally, and has been going every year since 2012.  Because our community has such a long history with DevOpsDays, we experiment with our format to find what works the best for us.

This year, we tried a couple daring things (more details in DevOpsDays Summit Austin 2018 – “DevOps Unplugged”):

  1. Voting on talks onsite instead of ahead of time (saw this at ProductCamp Austin)
  2. No sponsor booths (like the early DevOpsDays, Silicon Valley was like this for several years)
  3. Boxed lunches (like the early DevOpsDays, Silicon Valley was like this for several years)
  4. Capped headcount low at 400 (despite having sold 650 tickets last year)
  5. No streaming the talks (video is coming though)

Read the linked article for why, but the TL;DR is that we’re a nonprofit conference that exists to drive community engagement, and the “DevOps Talk Circuit,” the increased sponsor lead-churn demands, the time we spent on fancy lunches and such, and just the sheer number of attendees and weight of extras we were adding on were choking out the actual goal of the conference.  Despite having a huge slate of great keynoters at 2017 and everything being the biggest and best DoDA ever – we the organizers didn’t have a good time. We didn’t learn anything or make new friends. And we heard from other experts in town that said the same thing. So a dramatic change was implemented to pare the event back down to basics.  But how’d it work out?

We did a bunch of retrospective activities to find the answer!

  1. SurveyMonkey survey of all attendees
  2. Survey of all sponsors
  3. Community retrospective at the Austin DevOps user group
  4. Organizer retrospective

Attendee Survey Feedback

Of 400 attendees, we got 51 respondents (12.5%). Our overall NPS was 25 (“pretty good”). We don’t have a last year NPS to compare to, we didn’t do a great job of post event surveying last year mostly due to burnout (once you’ve spent most of your time prepping a conference, it’s time to get back to your real work, family, etc.).

Food Quality Talk Quality Openspace Quality Venue Quality Happy Hour Quality
Very high – 9 (18%) Very high – 6 (12%) Very high – 7 (14%) Very high – 12 (24%) Very high – 12 (25%)
High – 20 (39%) High – 27 (53%) High – 12 (47%) High – 29 (57%) High – 12 (25%)
Neither – 17 (33%) Neither – 9 (18%) Neither – 12 (24%) Neither – 7 (14%) Neither – 22 (46%)
Low – 4 (8%) Low – 8 (16%) Low – 8 (4%) Low – 3 (6%) Low – 2 (4%)
Very low – 1 (2%) Very low – 1 (2%) Very low – 3 (6%) Very low – 0 (0%) Very low – 0 (0%)

So everything was 50% or better “very high or high,” which seems good. We asked about favorite sponsors – ones mentioned by multiple participants include Cisco, Red Hat, NS1, VictorOps, Sumo Logic, xMatters, and Praecipio.

The comments were enlightening.  This year’s format was pretty divisive – there were lots of comments about liking voting on the talks and lots of comments about not liking it; there were lots of comments about liking e.g. “The new format with less vendor bloat” and then also lots of comments wanting sponsor booths back. And frankly, that’s what we expected – the new format was expressly designed to be attractive to some kinds of attendees and sponsors and not to others.

Overall, the positive comments predominated on the openspaces, keynotes, and ignites, and negative predominated on the talks and lack of booths.  (Several of those respondents identified as sponsors.)

Sponsor Survey Feedback

Total sponsor NPS was 7 (“good”) from 14 respondents of our 17 sponsors.  Again, there wasn’t the usual bell curve distribution – some sponsors loved it and others hated it.  The venue and the conversations people had onsite were very highly rated. The limited swag table aspect was low rated. The 30 minute suite sessions and lead quality were sharply bimodal – for example:

How did your 30 minute suite demo go?

  • Did not use 7.14%
  • Very well 7.14%
  • Well 28.57%
  • Neither poorly nor well 14.29%
  • Poorly 28.57%
  • Very poorly 14.29%

User Group Feedback

Read the board yourself!  Attendees, some organizers were in attendance.

image1

Analysis

Change is hard

People’s expectations were hard to alter. Especially in the sponsor realm where the person who books the sponsorship isn’t usually the person that comes on site.  One sponsor comment said “Without a booth, not worth our $5000!”  Well, yeah, that’s why we didn’t charge you $5k this year. People that go to multiple DevOpsDays, and especially sponsors, but even people who had just been to our event multiple years – we emailed and tweeted and blogged and put stuff on the signup forms, but the changes were still a surprise to many.  Voting on the talks was a concern not as much from speakers, but from people who “wanted their schedule set in advance!” and from people who were “afraid it makes speakers feel bad.”

Money isn’t hard

Even with the much lower sponsor cost this year ($3k), and lowering our headcount significantly (400), and providing the same great venue and lunches and breakfasts and drinks and not 1 but 2 shirts and blowing it out on the happy hour, plus being ripped off by our happy hour venue (not going back there!!!), we were still well in the black enough that we’re giving thousands of dollars to charity at the end of the event.

In fact, one of the advantages of this year’s format was that we weren’t giving 1/3 of our tickets away for free to a huge army of organizers, to speakers, etc.  Adding more sponsor stuff requires adding more volunteers that just eats back into the revenue stream again.

Specific Outcomes

Voting on talks

There was enough pushback that we won’t do that next year.  Submissions were lower this year, and a bunch of people dropped out before the event.  However, many of the people who dropped out are, to be blunt, the people we wanted to drop out. Talks “submitted on behalf of” someone. Vendor roadshow talks.

Here’s the thing – here in Austin, we’re pretty blessed.  We have a huge tech community with all the big players.  If you want to “have your secretary submit your talk, fly in, drive to the venue, give your talk, fly out” – whoever you are,  you really don’t have anything more interesting to say than the people who are already here. So if your goal being at DoDA isn’t to interact with the community, we have plenty of talk submissions already, thanks.  I get that if you’re starting up a DoD in the middle of nowhere the people on the “DevOps Talk Circuit” are key to bringing in new ideas and jumpstarting you, and I don’t devalue that.  But for us, we don’t need that and it doesn’t serve the needs of our current community.

This isn’t to say people from away aren’t welcome – John Willis is from Atlanta but he’s part of our community, because when he comes here that’s how he interacts with us.  (One of the “What did you like the most” survey comments simply said “John Willis.”)

People suggested various half-measures – “have us vote a week before!” But the additional logistics on that is very much not worth it, especially given what we think we’ve learned about our talk needs – read on for that!

Sponsor tables

OK, no sponsor tables was not universally beloved. Some sponsors – and not just the “here for the leadz” sponsors we were deliberately discouraging with the format – didn’t like it because it was harder to interact with folks about their product. But – here’s the rub – we had just as many complaints last year when we *did* have sponsor tables!  “My table was in the corner.” “There wasn’t enough foot traffic driven to me.”

The stadium format is pretty “noisy” and if we had sponsor tables back we’d have to do talks in some far-away rooms again, and removing those rooms this year saved us a lot of money and also people always hated it (like – FAR away).

Also, I’ll be honest, we had problems with sponsor misbehavior last year.  Silver sponsors claiming a table and standing behind it like a gold. Sponsors going out on the field (forbidden by UT). Sponsors trying to have food trucks park outside (also forbidden by UT police). Disruptive activity of a number of different sorts, requiring lots of work by organizers and volunteers and venue staff to deal with. I am sure many of them thought they were being “scrappy” etc. but in the end, we don’t get paid for this conference so we don’t need to put up with crap for it either. Discussion about “firing” certain sponsors was had.

We aren’t going back to the usual sponsor tables, but we are going to try something even more different – read on for that!

Boxed lunches

In early DoDA, we kept having super-deluxe Austin fare – BBQ, tex-mex – not from a caterer but from the real good places. This was for all the folks from away we were bringing in and wanted to show an Austin good time to!

Unfortunately, last year food lines for 650 people were a problem. Vendors weren’t adequately prepared with people or food.  We had to have many volunteers assigned. Food lines were super long and slow and a source of frustration.

This year we did have some comments about “I wanted the deluxe foods.” But they were far overwhelmed by those who appreciated being able to grab sustenance and get back to why they are here, learning and discussion. So with enough money we may try to get some kind of super-deluxe box lunch, but the box lunches will stay.

Lower headcount

The lower headcount was universally beloved except by lead generators and those who couldn’t get a ticket. More and better interaction, many positive comments noted the more intimate communication in openspaces and hallway track.  Keep.

No streaming

Worked out great.  No one complained, and the cost and org/volunteer time and schedule and stage compromises we have to make for live streaming are immensely negative.  Not going back.

2019 Planning

First of all, a disclaimer.  I am sharing this in the interests of transparency and helping other organizers learn from what we’ve done.  I don’t claim Austin is doing things the “one true way” and I know our community’s needs are different from many others. None of this is intended to denigrate any other events and their decisions. You don’t need to justify why you do things differently or why any of this isn’t right for your community.

Every year I start our planning with some basic questions.

  1. Do we want to have a DevOpsDays Austin next year?
  2. If so, why?  What is the goal of this year’s event?

“Inertia” is a bad reason to do anything.  We don’t have “money” as a reason because we have to spend what we get, we don’t pocket anything except some gifts. (My kid has already appropriated the bluetooth speaker I got this year…)

The group of organizers (over a tasty dinner at Chez Zee) decided “yes”, and after a good bit of discussion they decided that to us, this year, the goal of DevOpsDays Austin is to “Promote collaboration and sharing and networking specifically for the Austin technical community.” Now, that’s a pretty non-controversial statement on its face – but then as we plan stuff, we really test it against our goal and see if it supports it, is neutral, or takes away from it.  If it’s neutral or takes away, it goes.

This decision and clear statement (I think Marisa is who put it together for us) pricked my memory and I pulled out our attendee survey comments.  What did you like the most about DevOpsDays Austin 2018?  “Ability to collaborate with others.” “Enjoyed hearing what others were doing.” “Focus on the community.” “It’s a well-run, intimate conference.  I always see people I know.” “The community involvement.”  Her sentence crystallized what people were telling us was their favorite part of the event – super!

OK, so what does that mean for each area?

Content

People love the lightning talks more than anything.  Then the keynotes. Then the talks. It’s why we tried the attendee voting. The discussion covered how many of the talks seem too long and boring even at 35 minutes, and people trying to get too technical in them suffer from people not being able to follow along well due to screen size and large group.  People say they want themed tracks and stuff, but we rely on volunteers giving talks, we aren’t buying these off the shelf somewhere (“Give me 6 Kubernetes talks, 6 DevOps culture talks, 6 DevOps manager talks, and 6 intermediate level technical talks…”)  We are still committed to multiple technical tracks (DoDA was the first DoD to do this, many are still uni-track) because we’re 7 years in and we have a great diversity of experience in our community, and people don’t want to sit through the same messaging again.

Some talks are beloved and others aren’t.  As we sifted through the details, one comment from “What can we do better” on the attendee survey came to me.  “Talks focused on ‘I am a _____, here’s the problem we had and how we solved it.’ I say that because one of the coolest, most useful talks I saw was the Coinbase engineer who described how he used EBS volumes creatively to solve their scaling problem.”

So we decided to retire the voting but heavily curate the talks.  We don’t want “whatever talk you’re giving nowadays on the DevOps talk circuit” – we want talks in that format, the problem you had and how you solved it.

We’re working out the details, but we’re thinking about having these talks be more like 15 minutes long, with then linked openspaces that afternoon for the truly interested to get together and go ‘command line level’ with them.  This also allows for more breaks and collaboration time.

We also decided that idiosyncratic is better.  A couple of the organizers got excited about a sports/fitness theme to align with the stadium; one wants to set up a 5K, one has a wife that does yoga classes and we could have one, we can give fitbits as speaker gifts… While I and the other Agile Admins have been filming lynda.com courses and doing other creative things, the advice we keep getting from producers and directors and content managers is “Use *your* voice.  Do what *you* find interesting and other people will find it interesting.” Andrew Shafer loves running Werewolf games at openspaces at conferences, and people really respond to it! So we’re not going to hesitate to put stuff in we find interesting and we figure that enthusiasm will draw others. Trying to give attendees a “standard conference experience” is severely counterproductive because there’s plenty of regular conferences for people to go to, they get sick of it, and that doesn’t fit the devopsdays ethos in the first place.

Sponsors

I challenged the group.  “Tell me why we should have sponsors at all?  Half our revenue was ticket sales and half was from sponsors.  If we double ticket prices to $400 – still very low for any 2-day conference in the world – we can just not take sponsors at all, done and done. If we needed their money it’d be one thing, but we don’t. Let them spend their ‘limited marketing budget’ on the DoD events that do need it. How do the sponsors contribute to our goal other than with funding?”

The immediate response was that there are a bunch of sponsors who *are* part of the community and interacting with them is important; we have loads of Amazon/Google/Atlassian/Oracle/etc hiring going on here for example, and folks who work for Chef and Salt and Puppet and so on in town… We want those folks to be part of the conversation.  Just not disrupt that conversation.  And, some people pay for those tickets out of pocket so having some money to defray attendee costs is good.

We decided to try something different – we are using the luxury boxes at the stadium more and more; they’re relatively inexpensive and we used them for all the openspaces and such this year.   What if, we said, we intersperse sponsor suites with openspace suites, maybe even have them host some of the openspaces, do their own presentations in there too for whoever’s interested?  This means a limited number of sponsor slots (no more than 10, possibly fewer), but a more premium experience right there where the action is happening. And target Austin-presence companies to let them know about it. They can also then get food/drink catered into their suites to bring people in even more.

Attendees

Keep the headcount low – at least our limit of 400 from this year, if not lower. Consider a ‘two-tier’ ticket price with one price if your company is paying and another if you are; Data Day Austin has used this format to good effect.  Lets the non-backed solo folks in without breaking their bank but lets companies that do send attendees pay a reasonable amount.

Venue

UT Stadium is great, we don’t really see a reason to do all the work to change if we’re not doing booths and we’re going with a suite strategy for sponsors. Plus we have developed great relationships with the venue staff.

Keep refining the AV experience but doing it ourselves – we bought equipment and have a large set of “A/V geeks” so we don’t need to have outside people do it.

Food

Keep with boxed lunches. Austinites have had enough BBQ and tex-mex and this event is primarily for them per our goal. The benefit of fast lunch and snacks was tremendous this year. Could spend more on boxes from premium vendors but keep it boxed.  Maybe do drink service ourselves because we got truly rooked by the UT caterers on it this year.  Though Rich said he found the place the athletes eat and we might be able to get in on that… Keeping it fast, though, one way or the other.

Happy hour

We put a lot of work into this and spend double what the happy hour sponsor gives us each year, and then only half the people come and only half of those say they like it.  This year we had unlimited food and booze at a venue with video games in it for Pete’s sake, I think we’re done chasing the idea of the ultimate happy your. Probably we’ll do more of an onsite short sponsor room crawl at the venue, and then an “after party” we don’t put as much money/work into. “A couple free rounds at Scholtz’, get your own ass there.”

Conclusion

All right, that’s all the plan one dinner could get us.  But in the end, we’re happy with how the event went this year.  We’ll change a couple of the things that didn’t work out – talk voting, no booths – but not back to the old way because we already know that was suboptimal, instead we’ll try more options!  If you don’t have experiments not work out, you’re not being experimental enough, so we embrace that with DevOpsDays Austin.

Let us know your thoughts too!  Who are you, and what do you get or want to get out of DevOpsDays Austin?

Leave a comment

Filed under Cloud, DevOps

DevOpsDays Summit Austin 2018 – “DevOps Unplugged”

Hey all!  We’re starting work on next year’s DevOpsDays Austin – our seventh here in the ATX.  Many of you have come out to the event (or another of the great DevOpsDays around the world). Well, we have some changes in store this year!

Last year’s DevOpsDays Austin, “Monsters of DevOps” was bigger than ever and had a stadium rock theme – we had a huge venue,  all the DevOps VIPs we could pull down (including the first time all 4 authors of the DevOps Handbook managed to get together at an event), multiple content tracks, killer swag, great food, a hackathon, the best Happy Hour I’ve attended at a conference, we invited in and comped local user groups to give talks…  Part of our continuing trajectory to make DoDA more all encompassing and awesome.

But – every year we sit down and discuss vision before we launch into the conference.  What do we want to accomplish and why?  Who are we serving and why?  Why are we, personally, putting in huge amounts of unpaid work to serve the community? “Because it’s there and we did it last year” isn’t a good answer, so we like to really put some thought into it.

This time when we talked about it, first in our core group and then with the rest of the 2017 organizers, we realized that we’ve been concentrating on “bigger” but we’ve been putting more and more money and effort into the parts of the event that aren’t really of high DevOps value. Here in Texas, it’s easy to conflate bigger with better, since we’re both the biggest and the best!  But we’re not sure that’s right. Many of the more expert people we know here in Austin don’t really come out to the event any more, unless they are giving a talk or recruiting for their current gig.  Talks and openspaces have kept focused on introducing new people to DevOps, enterprise folks, “horses and donkeys,” and so on.

And as we talked, we said “Well – what do we personally get out of the conference nowadays as attendees?”  The answer was “not much.” Openspaces are huge and end up being a couple people talking.  Talks are either pretty familiar from the conference circuit or also designed for new folks.  We have more content but it’s more passive content, sit and watch.  It’s good for the newbies but not as much for the experienced folks.

We contrasted this to the first couple DevOpsDays we went to in Silicon Valley.  The first couple were just in a big auditorium at LinkedIn.  There weren’t any sponsor booths. More of the event was focused on the openspaces and interaction between the highly driven participants. We ate box lunches wherever we could perch in the parking lot outside – and swag was just a t-shirt.  Heck, the third one was in a weird abandoned building Dave Nielsen had access to, we had to carry our own chairs around to talks and the food and stuff was in a concrete-and-cage loading dock. But it’s those events we got the most out of.

Therefore, this year DevOpsDays Austin is going to go to what we call a “Summit” format.  We’re reducing the size of the event, and focusing more on local, motivated practitioners.  What does this mean?

  1. No sponsor tables.  We’d love sponsors to participate, but in recent years we’ve gotten more folks who have either just sent aggressive marketers, or sent people we enjoy and then locked then down behind tables. So we’ve come up with a sponsorship package that gets them exposure and value but lets them actually participate in the event.  Folks that just want to churn leads will self-select out.  The sponsorships are less expensive, and we’ll just have venue food etc. instead of premium.
  2. No preselected talks.  Well, OK, maybe we’ll have one keynote a day.  But I went to a ProductCamp here in Austin and they did something brilliant – they had a RFC but don’t do a final selection – finalists show up and the audience votes on what talks they want to hear (kinda like openspaces but more prepared).  This means people who say ‘well… I’ll come to your event if I can talk (or sponsor if I can talk, or…)’ will self-select out. You come because you want to be here, and you can give a talk!
  3. Smaller headcount.  We’re lowering the cap (including sponsors and organizers and volunteers) to 400. We’re going to get openspaces to be the kind of highly engaged discussions that make the so valuable.  We’re going to be up front with people that attendees are expected to engage.  DoD used to be the only thing around to learn from.  But now, if you’re an enterprise person that wants to have some DevOps talked at them – you have  variety of options now, like you can go to DevOps Enterprise Summit (also a great event), or to another DevOpsDays like the one in Dallas using the conference format, or one of a dozen events either completely DevOps or DevOps-tracked.  But for here in Austin this year, we need something where the unicorns can also have an event meaningful to them, so they can gather and refresh on what’s going on. Not to say only “unicorns” are welcome, but frankly we’d prefer people only come out if they intend to discuss, share, and engage; this will not be a passive-learning friendly event.
  4. No streaming.  Every year we put a lot of work and money into live-streaming and/or recording the event.  But it’s often problematic, and doesn’t get viewed a lot – there’s so much content out there now.  But even worse, we end up having to degrade the experience of real attendees around the requirements of broadcast – space, money, schedule, the presenter has to stay in a little box… So we’re not going to do it.  You want to participate – come out and participate.

But How Can This Work???

That was everyone’s initial reaction to this plan.  But that’s silly – it has worked.  We’re just doing things that DevOpsDays has already done, that ProductCamp has already done, and so on. It’s just not what’s become customary.  After the organizers had a little time for it to sink in, they all rallied behind it with a vengeance.

We’ve run the numbers and just the basic $200/head attendee fees can pay for the venue, basic food, and a shirt, even if we get zero sponsors.  (We won’t have zero sponsors, we just put our sponsor page up and someone bought in the first hour it was live.) As we get more funding we’ll pump up the event, but deliberately focus on the core experience of highly skilled techies learning from each other, instead of adding distractions.

How Dare You Dis My Format???

This is the format we’d like to try this year.  Other events will use other formats and that’s fine. Here at DoDA we try something different every year!  We were the first to have multiple content tracks (over the complaints of some purists).  We added a hackathon, we added a local user group track… Last year we went big with a vengeance, and it was cool.  Now we’re going to do more small and exclusive, and that’ll be cool.  Next year, it’ll be different. Whatever your event is doing, more power to you, don’t confuse us having a vision we believe in with us thinking you’re “wrong.”

Come on down!

We’d love to see everyone out at DevOpsDays Austin 2018!  Come ready to interact and share.  Come ready to give a talk, with the risk it won’t make.  Come sponsor your company, just you won’t have a table to lounge at. This change has gotten us excited about running our seventh DevOpsDays, and we bet you’ll love it!

11 Comments

Filed under Conferences, DevOps

Awesome Upcoming Austin Techie Events

We’re entering cool event season…  I thought I’d mention a bunch of the upcoming major events you may want to know about!

In terms of repeating meetings you should be going to,

  • CloudAustin – Evening meeting every 3rd Tuesday at Rackspace for cloud and related stuff aficionados! Large group, usually presentations with some discussion.
  • Agile Austin DevOps SIG – Lunchtime discussion, Lean Coffee style, at BancVue about DevOps. Sometimes fourth Wednesdays, sometimes not. There are a lot of other Agile Austin SIGs and meetings as well.
  • Austin DevOps – Evening meetup all about DevOps.  Day and location vary.
  • Docker Austin – First Thursday evenings at Rackspace, all about docker.
  • Product Austin – Usually early in the month at Capital Factory. Product management!

3 Comments

Filed under DevOps

DevOpsDays Austin Is Coming!

The third annual DevOpsDays conference in Austin will be May 5-6 (Cinco de Mayo!) at the Marchesa, where it was held last year! As many of you know, the DevOpsDays conferences are a super popular format – half talks from practitioners, half openspaces, all fun – held in many cities around the world since the first one in Ghent launched the DevOps movement proper.

  • You can register – all the early bird tickets are sold out but the regular ones are only half gone.
  • You can also propose a talk!  There’s 35-minute full talk slots but we’re even more in need of 5-minute Ignite! style lightning talks! RFP ends 3/26 sp
  • You can sponsor! The Gold sponsorships are half gone already. And we have some special options this year…

DevOpsDays Austin has been bigger and better every year since its inception and should have something good for everyone this year. Come out and join your comrades from the trenches who are trying to forge a new way of delivering and maintaining software!

1 Comment

Filed under Conferences, DevOps

LASCON 2013 Report – First Morning

IMG_1475Arriving at #LASCON 2013, hosted as usual at the Norris Conference Center, the first thing you see is the vintage video games throughout the lobby! As usual it’s well run and you get your metal badge and other doodads without any folderol; volunteers packed the venue ready to help folks with anything. I got a lovely media badge since I’m on the hook to blog/tweet it up while I’m there! It’s in a nice central location on Anderson Lane so getting there took a lot less time than my normal commute to work did.

IMG_1481The MCs, James Wickett and David Hughes, got us kicked off. Thanks went out to many the LASCON sponsors!

  • White Hat
  • Qualys
  • Gemalto
  • Trustwave/Spider Labs
  • Critical Start
  • Sourcefire
  • SOS Security

IMG_1482Then everyone stood and raised their right hand to say the “LASCON pledge,” which consists of “I will not hack the Wi-fi,” “I will not social engineer other attendees and the nice Norris Conference Center staff who are hosting us,” and similar.

Then, the keynote!

Keynote- Nick Galbreath, The Origins of Insecurity

IMG_1488Nick Galbreath (@ngalbreath), VP of Engineering at Iponweb. He used to work for Etsy, now he works in Tokyo for a Russia-based ad infrastructure company.  Suck that, Edward Snowden.

Slides at speakerdeck.com/ngalbreath!

If you’re in security, you should be bringing someone else from dev or ops or something here! We can’t get much done by ourselves.

Crypto

There’s a lot of consternation about crypto and SSL and PKI lately. The math is sound!  See FP’s “The NSA’s New Code Breakers” – it’s way easier to get access other ways. I don’t know of any examples of brute forcing SSL keys – it’s attacking data at rest or bypassing it altogether.

But what about the android/bitcoin break and alleged fix re: Java SecureRandom PRNG? I can’t find the fix checked in anywhere.  Let’s look at SHA1PRNG. Where’s the spec? You’re forced to use it, where’s the open implementation, tests…

Basically everything went wrong in specification, implementation, testing, review, postmortem… Then the NIST’s Dual-EC-DRBG spec – slow and with a potential backdoor – but at least it’s not required by FIPS!  It’s broken but not mandatory and we know it’s broken, so fair enough. It’s a “standard turd.” Standards aren’t a replacement for common sense. Known turdy in 2007.  Why are you just removing it now? TLS 1.2 was approved in 2008, why don’t all browsers support it and no browsers support GCM mode? Old standards need augmentation and updates.

Fixing the CA system – four great ways, certificate pinning, pruning, HTTPS Strict-Transport-Security, certificate-transparency.org.

Everything Else

  • Network Security – stuff you didn’t write
  • App Security – stuff you did write
  • Endpoint Security – stuff you run

IT internal tech is mostly Windows/Mac CM and patching, 99% C-based stuff.

Tech Ops – Routers, Linux, Core server (all C too)

Dev:

  • Input validation – not hard
  • Configuration problems
  • Logical problems – more interesting
  • Language platform problems (most patches here also in C!)

Reactive work is patching, CM, fixing apps, patching infrastructure. You can focus your patching though – Win7 at current patches, Flash, Adobe, Java will get 99% of your problems, focus there – but it’s hard to do. But either you can do it trivially or it’s really hard.

Learn from the hardest apps to deploy.  The Chrome model of self updating gets 97% of people within a version in 4-6 weeks. Android, not so good- driven more by throwing out phones than any ability to upgrade. They’re chipping stuff away from the OS and making more into apps to speed it up. Apple/iOS just figured out app auto-update. Desktop lags though. WordPress is starting background updates. BSD is automatically installing security updates at first boot.

Releasing faster and safely is a competitive advantage AND makes you more secure.

For desktop upgrades, can’t we do something with containers? Why only one version installed? How can we find out about problems from users faster? How do we make patching and deployment easy for the dumbest users?

Even info on “How do I configure Apache securely” is wide and random on the Web. Silently breaks all the time, and it’s simple compared to firewalls, ssh, VPN, DNS… Rat’s nests full of crap, while it gets easier and easier to put servers on the internet. How can we make it safe to configure a server and keep it secure?

Can we do this for application development? Ruby BrakeMan is great, it does static analysis on commit and sends you email about rookie mistakes. Why not for apache config? (Where did chkconfig go?)

PHP Crypt – great for legacy passwords and horrible for new ones. Approximately 0% chance of a dev getting its configuration right.

See @manicode’s best practices – have a business level API for that.

By default, every language has a non-crypto, insecure PRNG. So people use them. They are used for some science stuff, but seriously if you’re doing physics you’re going to link something else in. Being slightly slower for toy apps that don’t care about security isn’t a big deal. Make the default PRNG secure! And, there’s 100x more people interested in making things fast than making them secure, so make the default language PRNG secure and people will make it faster.

libinjection.client9.com to try to eliminate SQL injection! It’s C, fast, low false positives, plug in anywhere.

Products focus on blocking and offense/intrusion, but leave these areas (actual fixing) uncovered.  Think globally, act locally. Even if you’re not a dev, most open source doesn’t have a security anything – join in!
Write fuzzers, compile with different flags, etc.

So think big, get involved, bring your friends!

Malware Automation

IMG_1490By Christopher Elisan from RSA, aka @tophs.

Total discovered malware is growing geometrically year over year. There are a lot of “DIY malware creation kits” nowadays; SpyEye, Zeus… These are more oriented around online crime; the kits of yesteryear were more about pissing contests about “mine is better than yours” (VCL, PS-MPC). The variation they can create is larger as well.

Armoring tools exist now – PFE CX for example, claims to encrypt, compress, etc. your executable – but all the functions don’t always work and buyers don’t check.  Indetecitbles.net is online and will do it! It was free but now it’s “hidden.”

Use a tool like ExeBundle to bundle up your malware and then share it out via whatever route (file sharing, google play, whatever). Or hacking and overwriting good wares – even those that bother publishing a hash to verify their software often keep it on the same Web site that is already getting hacked to change the executable, so the hash just gets changed too.

So you make your malware with a kit, put it through a crypter a realtime packer, an EXE binder, other armoring tools, then run through QA in terms of on premise and cloud AV, then you’re ready to go.

Targeted vs opportunistic attacks… Delivery is a lot easier when you can target.

Anyway, many of those new malware samples are really just the same core malware run through a different variety of armoring tools. They’re counted as different malware but should get grouped into families; he’s working on that at RSA now.

Besides the variation in malware, domains serving malware can rotate in minutes. Since the malware can be created so quickly it effectively defeats AV by generating too many unique signatures. Reversing has to be done but it takes weeks/months.

Demo: Creating Malware in 2 Minutes!

ZeuS Builder – bang, bot.exe, one every couple seconds. Unique but not hash-unique at this point. They look different on disk and in memory. Then runs Saw Crypter, in seconds it creates multiple samples from one ZeuS sample. Bang, automated generation of billlllllyuns of armored samples.

There’s really just a handful of kits behind all the malware, need new solutions that go after the tools and do signature-less detection.

From Gates to Guardians: Alternate Approaches to Product Security

IMG_1493Jason Chan, Director of Engineering from Netflix, in charge of security for the streaming product. Here are his slides on Slideshare!

Agile, cloud, continuous delivery, DevOps – traditional security doesn’t adapt well to these. We want to move fast and stay safe at Netflix.

The challenges are speed (rapid change) and scale. To address these…

  • Culture – If your culture has moved towards rapid delivery, it’s innovation first. Don’t be “Doctor No” and go against your company culture, you won’t be successful.  Adapt.
  • Visibility – you need to be able to see whats going on in a big distributed system.
  • Automation – no checklists and spreadsheets

At Netflix we do ~200+ pushes to production a day, 40M subscribers, 1000+ devices supported.

Culture

We have a lot of stuff on our site about this, it’s a big differentiator.  “Freedom and responsibility” is the summary. No buck passing. Responsible disclosure program externally.

We’re moving towards “full stack engineers” that know some about appsec, online operations, monitoring and response, infrastructure/systems/cloud – that can write some kind of code. The security industry seems to be moving towards superspecialists, we don’t see that as successful.

2 week sprint model, JIRA Scrum workflow (CLDSEC project!). No standups, weekly midsprint meeting. Bullpen shared-space model.

Visibility

Use their internal security dashboard (VPC, crypto, other services plug in and display their security metrics). Alerts send emails with descriptive subjects, the alert config, instructions/links as to where to check/what to do. Chat integration.

NSA asks, how do you verify software integrity in production?  How do you know you’re not backdoored?

They have their Mimir dashboard that is a CI/CD dashboard, that tracks source code to build to deploy to JIRA ticket. Traceability!

Canary testing because code reviews don’t catch much.  Deploy a new version and test it (regression, perf, security) and see if it’s OK. Automatic Canary Analyzer gets a confidence level – “99% GO!”

Simian Army does ongoing testing. Go to prod… Then the monkeys test it.

Security Monkey shows config change timestamps of security groups and stuff.

So they have Babou (the ocelot from Archer) that does file integrity monitoring. They use the immutable server pattern so checking is kinda easy, but you still can be running multiple canary versions at the same time so there’s not one “golden master.” This allows multiple baselines.

Q: How long did it take to make this change and implement? What were the triggers?
A: This push started when he started in 2011; previously IT security handled product security. He hired his first person last year and now they’re up to 10.

Q: What do you do earlier on in the lifecycle in arch and design (threat modeling etc.)?
A: Can’t be automated, the model here is optionally come engage us (with more aggressiveness for stuff that’s clearly sensitive/SOXey).

Q: So this finds problems but how do people know what to do in the first place, share mistakes cross teams?
A: As things happen, added libraries with training and documentation. But think of it as “libraries.”

Q: Competing with Amazon while renting their hardware? (Laaaaaame, the CEO has talked about this in multiple venues.)
A: AWS is the only real choice. Our CEOs talked.

Next – Lunch!  No liveblog of lunch, you foodie voyeurs!

Leave a comment

Filed under Conferences, Security

Awesome Austin Events!

Besides the “big one,” DevOpsDays Austin 2013, there’s a bunch of great events going on in Austin for techies.

The Agile Austin DevOps SIG meets every last Wednesday over lunch at Bazaarvoice; lunch is provided.  This month’s meeting on January 30 is Breaking the Barriers.

The Austin Cloud User Group meets every third Tuesday in the evening at Pervasive; dinner is provided. This month’s meeting is January 15, sponsored by Canonical, and there is a talk on Openstack Quantum, the network virtualization platform.

South by Southwest Interactive is here of course on March 8-12.

Hip security event BSides Austin is on March 21-22.

Data Day Austin is on January 29.

Texas Linux Fest will be May 31 – June 1.

It’s never been a better time to be a techie in Austin!

3 Comments

Filed under Cloud, Conferences, DevOps

Awesome Austin Tech Meetups

Austin is such a great place to be a techie.

  • The Austin Cloud User Group (I help run it) meets every third Tuesday evening, and we’ve ben having 50+ people come in to check out some awesome stuff.  Next meeting Feb 21 on Puppet, hosted by Pervasive.
  • The Agile Austin DevOps SIG meets fourth Wednesdays, we had our meeting today and had about 20 attendees, hosted by CA/Hyperformix. I also help run that one.
  • The Austin Big Data User Group is back meeting – next one is tomorrow night! Hosted by Bazaarvoice.
  • The Austin OWASP chapter is one of the biggest and most active in the country, and also meets monthly, hosted by National Instruments. Fellow Agile Admin James Wickett helps run that group.
  • The Cloud Security Alliance, Austin chapter is just getting started but has a lot of momentum and we’re coordinating with them from the ACUG and OWASP sides. Their first meeting is tonight, come out!

There are others but those are my favorites and therefore the coolest by definition.

There’s also cool events coming up you should keep an eye out for.

  • DevOpsDays Austin, Apr 2-3, hosted by National Instruments, and this’ll be big! Patrick Debois and the whole crew of DevOps illuminati will be here. Now taking sponsors and speakers! Register now!
  • AppSec USA 2012, Oct 23-26 – Austin OWASP kicks so much ass with LASCON that the annual OWASP convention is coming here to Austin this year!
  • South by Southwest Interactive, March 9-13 – quickly becoming theWeb conference in the flyover states :-). Lots of stuff happens during it, like:
    • Austin Cloud/DevOps party courtesy GeekAustin (ACUG is a community sponsor). March 10.
    • CloudCamp – Dave Nielsen will be bringing a CloudCamp to Austin again this year during SXSWi. Details TBD, sounding like Mar 11 maybe.
  • The Cloud Security Alliance and ACUG are hoping to put together an Austin cloud conference, too. Maybe early 2013.

Leave a comment

Filed under Cloud, Conferences, DevOps