Agile Austin asked me to help re-launch their blog, so I’ve contributed a piece on “What Is DevOps?” for them!

Leave a comment

by | March 16, 2014 · 9:58 am

Steal This Teambuilder: Meat, Guns, and Booze

I thought I’d take a minute for a teambuilding tip from Management Corner and share a winning teambuilder from here in Austin that I’ve used a lot and has always been a solid success. We call it “Meat, Guns, and Booze.” Steal it and use it with your teams!

Coming up with good teambuilders for an engineering team is pretty difficult.  You have a mix of introverts and extroverts. Going to a movie doesn’t really allow for much personal interaction.  A group meal lets you talk to the 3-4 people next to you, which in large groups isn’t a large percentage, and even getting everyone seated together can be a challenge.  Not everyone drinks and may not appreciate hanging out at a bar. Messing around in the office is fine but sometimes you want to get out of there so that people aren’t continually distracted by work. Not everyone is active enough for something like paintball (especially in Texas weather!). Well, back some years ago while I was at National Instruments we were mulling over what a good field trip/teambuilder could be given all these limitations, and happened across a winning combination that never fails to deliver!


BBQ At Smitty's 4

Meat Frenzy

First, head out to Lockhart, about an hour from most places in Austin, for a lunch of barbeque. Austin BBQ is good, but for the truly top tier stuff you have to go out to other locations (we love Franklin’s/LA Barbecue but going and standing in line at 10 AM to get some isn’t really a team activity). You can have internal debates over whether you go to Black’s, Smitty’s, or Kreutz’; try them all out over time as everyone will passionately defend the one they think is best. For vegetarians, there’s good sides, but also there’s a cafe right across the street from Smitty’s Market with salads and whatnot. Smitty’s also has a “no utensils” rule; eating meat with your hands definitely switches the mood to informal and friendly quickly!

Pro Tip: Bring cash. Some of them – Smitty’s for sure IIRC – don’t take credit!

Blacks BBQ

Goodsprings from Fallout: New Vegas?

Besides the great BBQ, many folks haven’t seen old style rural Texas – even folks living in Austin, but especially people from other countries. I remember once we were headed into town, and it has a traditional town square with raised boardwalks and all – and the two guys from Ukraine in the back seat started wriggling around like itchy bear cubs and talking rapidly to each other in Ukranian.  “What’s up?” I asked from the driver’s seat.  “It looks just like Fallout!” they breathlessly exclaimed. I had to laugh. It’s worth a brief stroll around the town square and courthouse. There’s a western-stuff store for the greenhorns, too.



My .30/.30


A Ukranian Engineer Learns About Texas

If it’s raining, you can fall back to Red’s in Austin, they have several locations.  But they’re small and cramped for groups. Right near Lockhart is the Lone Star Gun Range. They have rifle and pistol ranges, outdoor but with shade.  You just need a driver’s license (or passport for the foreigners) and you can rent anything from a revolver to an AK-47 for a reasonable fee – you mainly end up paying for ammunition. (You can bring your own guns and ammo, but have to buy their ammo to use in their rental guns.) They give a quick safety briefing and off you go.  As long as you have one or two more gun-savvy people to help folks who are having trouble figuring out how to reload or whatnot, you’re good to go; I’ve been there many times with groups of 10-20 composed of mostly noobs with no problems. And don’t worry – everyone gets into it.  I’ve had people who were initially intimidated but once they get hands on for a minute they always take to it with gusto.

Even in Texas, gun owners usually don’t go out to the range all the time, so it’s a treat even for those who are more into it, and a novelty for those who aren’t.

Pro Tip: Don’t drive a super-nice car – the road to the place is not paved, it’s shell.  If you have a sweet hand-waxed Porsche you should ask someone else to drive. Also, it’s outside in Texas in (whatever season) – bring sunscreen, hats, and a cooler full of water. There is an air-conditioned store with a bathroom and all if someone gets overwhelmed by the heat.

Steve on the Barrett 1

The Barrett

Sad Note: They used to have a Barrett .50 out there at Lone Star but they sold it.  The rounds were super expensive but no one wanted to fire more than 1 or 2 anyway! But they have revolvers, automatics, rifles from .22 to ARs to AKs…


After shooting, you’re usually hot, and so stopping in at a watering hole on your way back to Austin is a good thing to do.  Anywhere that’s convenient for you is good – I’ve done the Whip In as that’s uber Austiny, has an outdoor patio with picnic tables for good social mobility, and has a lot besides just beer (food, wide non-alcoholic drink variety). You got out of the office (and out of town), had the team nourish itself on food, then have an exciting activity, and now they get to interact with the ice already well broken. People don’t drink a lot because it’s already been a long day, as opposed to a “let’s go drink” teambuilder where people are more tempted to overindulge. But they have plenty to talk about. Non-drinkers – or if your entire company doesn’t like sponsoring events with alcohol – can just drink sodas and have an appetizer or two to unwind.

And yes, to all the wags out there who inevitably respond with “but don’t mix up the order!”, you do want to do it in this order – there’s the wisdom of no drinking before gun handling, but also because this sequence of events is carefully constructed to provide a cadence of icebreaking and interaction. You “break bread” together, which is a universal signal of togetherness, and sit by some people to eat, by others in the car, interact with others while doing a stimulating activity, unwind with others after, etc. And since it’s multipart, if someone really does have to come late or leave early there are multiple natural breakpoints. We also like doing three-round K1 racing as teambuilders but that’s more disruptive if people come and go.

I’ve used this teambuilder with multiple teams, multiple companies, and it’s always a hit. It’s especially a hit with international teams – folks from Malaysia, Ukraine, etc. see all of this as a special treat; it’s very common to talk to them a year later about their trip to Austin and they’ll immediately volunteer that this was their favorite part of their visit hands down. It’s relatively inexpensive and just takes a long afternoon (we usually leave at 11 and start wrapping up at 5). It’s maybe $1000 for a full day of fun for a dozen people.

Not In Austin?

Well of course the luridly titled “Meat, Guns, and Booze” specifically evokes Texas and is tuned to our location.  But consider the same kind of event cadence and think about what’s distinctive in your area. You just want the team to share a meal, then participate in an activity that everyone can easily participate in, and then have an opportunity to relax and talk with each other afterwards.


Filed under General

Agile Organization Incorporating Various Disciplines

I started thinking about this recently because there was an Agile Austin QA SIG meeting that I sadly couldn’t attend entitled “How does a QA manager fit into an Agile organization?” which wondered about how to fit members of other disciplines (in this case QA) in with agile teams. Over the last couple years I’ve tried this, and seen it tried, in several ways with DevOps, QA, Product Management, and other disciplines, and I thought I’d elaborate on the pros and cons of some of these approaches.

Two Fundamental Discoveries

There are two things I’ve learned from this process that are pretty universal in terms of their truth.

1. Conway’s Law is true. To summarize, it states that a product will tend to reflect the structure of the organization that produces it. The corollary is that if your organization has divisions which are of no practical value to the product’s consumer, you will be creating striations within your product that impact client satisfaction. Hence basic ITSM and Agile doctrines on creating teams around owning a service/product.

2. People want to form teams and stay with them. This should be obvious from basic psychology/sociology, but if you set up an organization that is too flexible it strongly degrades the morale of your workers. In my previous role we conducted frequent engineer satisfaction surveys and the most prominent truth drawn from them is the more frequently people are asked to change roles, reorg, move to different teams, the less happy they are. Even people that want to move around to new challenges frequently are happier if they are moving to those new challenges with a team they’ve had an opportunity to move through Tuckman’s stages of group development with.

I have seen enough real-world quantified proof of both of these assertions that I will treat them as assumptions going forward.

Organizational Options

We tried out all four of these models within the same organization of high performing engineers and thus had a great opportunity to compare their results.

Separate Teams

When we started, we had the traditional model of separate teams which would hand off work to each other.  “Dev,” “Ops,” “QA,” “Product” were all under separate management up through several levels and operated as independent teams; individual affinities with specific products were emergent and simply matters of convenience (e.g. “Oh, he knows a lot about that BI stuff, let him handle that request”) and not a matter of being dedicated to specific product(s).

Embedded Crossfunctional Service Teams

Our first step away from the pure separate team model was to take those separate teams and embed specific members from them into service oriented teams, while still having them report to a manager or director representing their discipline. In some cases, the disciplinary teams would reserve some number of staff for tool development or other cross-cutting concerns. So QA, for example, had several engineers assigned to each product team, even though they were regarded as part of the permanent QA org primarily.

We (very loosely) considered our approach to be decentralized and microservice based; Martin Fowler is doing a good article in installments on Microservice Architecture if you want more on that topic.

Fully Integrated Service Teams

With our operations staff we went one step farther and simply permanently assigned them to product teams and removed the separate layer of management entirely. Dev and “DevOps” engineers reported to the same engineering manager and were a permanent part of a given product team. Any common tooling needed was created by a separate “platform” engineering team which was similarly integrated.

Project Based Organization

Due to the need to surge effort at times, we also had some organizations that were project, not product, based. Engineers would be pulled either from existing teams or entire teams would additionally be pulled into a short term (1, 2, 3 month) effort to try to make significant headway across multiple products, and then dissolve afterwards.

Hmm, this looks like it’s getting big (and I need to do some diagrams).  I’ll break it up into separate articles for each type of org and its pros and cons, and then a conclusion.

  • Separate Teams By Discipline
  • Embedded Crossfunctional Service Teams
  • Fully Integrated Service Teams
  • Project Based Organization
  • Combos and Conclusions


Filed under Agile, DevOps

Special CloudAustin SXSW Edition 3/6

There’s a special early CloudAustin user group this month on Thursday, March 6 out at Rackspace. We’re having some folks from  West Coast startup Stormpath (, API-driven user and group management for developers come and give two talks:

Cloud Marketing 101: How to Market Your Cloud Product

You pour blood, sweat and tears into your API, open source and weekend projects – let’s make sure they get the attention they deserve! We’ll go through real-world examples of tactics developers can do to attract attention to their work. Beyond growth hacking and that first post to Hacker News, we’ll look at high-value marketing maneuvers that will drive usage, but won’t make you feel like a dirty huckster.

To Infinity and Beyond! Scaling Your Stack with Service Oriented Architecture

Abstract: Service Oriented Architecture is a proven design pattern which allows you to simplify your codebase, seamlessly scale your service, reduce engineering frustrations — and even helps lessen hosting costs. Come learn what SOA is, why it’s useful, and take a look at an in-depth technical overview of SOA, and how it can help your organization. Delight your engineers (and business people!) by building your product on top of simple, REST API services.

Sign up here!

Leave a comment

Filed under Cloud, Conferences

Stupid webappsec Tricks Talk at LASCON with Zane Lackey

Zane Lackey spoke at LASCON 2013 about how they do data driven security at Etsy. At the conference Ernest took some notes and blogged them in this post: . Now that the LASCON vids are out we thought this would be a good time to revisit this stellar talk. Enjoy!

Leave a comment

Filed under DevOps

Clean up your cookbook mess with meez

Is your kitchen a mess? Meez will help you get things straightened out.

There is a new gem in town, and it’s here to clean up the mess you made out of your cookbooks.  Its called meez.  

If you are like me, maybe you started writing some chef cookbooks, and then later decided to add some testing and you followed some blog posts to set up some different tools.  Some where along the way you figured out that the cool kids don’t use Librarian (although I still am fond of it) so you decide to use Berkshelf (I am learning to like it).  You also figured out that you need a linting tool and some sort of way to do TDD for your infrastructure. Man, this cookbook is starting to get pretty crowded with a bunch of files that have nothing to do with actually installing the code you want to install.  You also start looking around and wondering why you have to learn all these esoteric frameworks/tools to write a simple chef cookbook (technically you don’t have to, but the technohipsters frown on you if you don’t).

What are you to do?

Enter meez. Meez sets up an opinionated cookbook replete with all the testing tools and frameworks a modern chef requires: chefspec, foodcritic, rubocop, berkshelf, kitchenci, …  Once you tell meez to create a cookbook for you, it sets up all the different frameworks and gets you ready to start actually writing your recipes and working on your cookbook.  No more remembering how to setup all the testing tools and frameworks.  Sweet!

gem install meez
meez --cookbook-path /tmp -C "James Wickett" -m mycookbook

What this will do is set up ‘mycookbook’ with all the testing tools you need.  By giving it my name and email, it autofills all that in the relevant spots as well.  Once meez finishes running, it tells you what to do next:

You must run `bundle install' to fetch any new gems.
Cookbook mycookbook created successfully
Next steps...
$ cd /tmp/mycookbook
$ bundle install
$ bundle exec berks install
$ bundle exec strainer test

Follow those steps and you are now ready to start working on cookbooks and stop worrying about all the testing frameworks and tools surrounding TDD and chef.

Meez was created as a gem after @pczarkowski‘s excellent sysadvent blog post “The Lazy SysAdmin’s Guide to Test Driven Chef Cookbooks.” Reading that will give you more context behind what meez is doing.

Moar Links

1 Comment

Filed under DevOps

Product Security the Netflix Way with Jason Chan

Jason Chan did a great presentation at LASCON last fall in Austin on product security at Netflix offically titled ‘From Gates to Guardrails: Alternate Approaches to Product Security.’  You may have even seen the Agile Admin’s coverage of LASCON and @ernestmueller‘s interview of Jason Chan.  The LASCON videos are now online and we thought we would share some of our favorites from the conference.

Leave a comment

by | January 18, 2014 · 8:30 am