Category Archives: DevOps

Pertaining to agile system administration concepts and techniques.

Agile Organization: Embedded Crossfunctional Service Teams

This is the second in the series of deeper-dive articles that are part of Agile Organization Incorporating Various Disciplines. Previously I wrote about the “traditional” siloed model as Agile Organization: Separate Teams By Discipline.

Basic agile doctrine, combined with ITSM thinking, strongly promulgate a model where a team owns a business service or product.  I’ll call this a “service team” for the sake of argument, but it applies to products as well.

The Embedded Crossfunctional Team Model

Simply enough, a service team has specialists from other groups – product, QA, Ops, whatever – assigned to it on an exclusive basis. They go sit with (where possible) and participate with the group in their process. The team then contains all the resources required to bring their service or product to completion (in production, for services). The mix of required skills may vary – in this diagram, product #4 is probably a pure back end service and product #3 is probably a very JavaScriptey UI product, for example.

embedded

Benefits of Embedded Crossfunctional Teams

This model has a lot of benefits.

  • Since team members are semi-permanently assigned into the team, they don’t have conflicting work and priorities. They learn to understand what the needs are of that specific product and collaborate much more effectively with all the others working on it.
  • This leads to removal of many bottlenecks and delays as the team, if it has all the components it needs to deliver its service, can organically assign work in an optimal way.
  • It is amazing how much faster this model is in practice than the separate teams by discipline model in terms of time to delivery.
  • Production uptime and performance are improved because the team is “eating its own dog food” and is aware of production issues directly, not as some ticket from some other team that gets routed, ignored, finger-pointed…

Drawbacks of Embedded Crossfunctional Teams

  • Multiple masters is always an issue for an engineer.  Are you trying to please the service team’s manager or your “real” manager, especially when their values seem to conflict? You have to do some political problem-solving then, and engineers hate doing that. It also provides some temptation to double-resource or otherwise have the embedded engineer “do something else” the ops team needs done, violating the single service focus.
  • It’s more expensive.  Yes, if you do this, you need at least one specialist per service team. You have to play man-to-man, not zone, to get the benefits of the approach. This should make you think about how you create your service teams; if you define a service as one specific microservice and you have teams with e.g. 2 devs on them, then embedding specialists is way more expensive. Consider basing things on the business concept of a service and having those devs working on more than one widget, targeting the 2-pizza team size (also those devs will be supporting/sustaining whichever services aren’t brand new). (Note that this is only “more expensive” in the sense that doesn’t bother with ROI and just takes raw costs as king – you get stuff out and start making revenue faster, so it’s really less expensive from a whole-company POV, but from the IT beancounter perspective “what’s profit?”)
  • While you get benefits of crosstraining across the disciplines, the e.g. ops folks don’t have regular all-day contact with other ops folks and so you need to take care to set up opportunities for the “ops team” to get together, share info, mentor each other, etc. as well. Folks call these “tribes” or “guilds” or similar.

Experience with Crossfunctional Teams

It can be hard at the beginning, when teams don’t understand each others’ discipline or even language yet.  I had a lengthy discussion with an application architect on one team – he felt that having ops people in the design reviews he was holding was confusing and derailing. The ops people spoke some weird moon-man language to him and it made the reviews go longer and require a lot more explanation.  I said “Yes, they do.  But we have two choices – keep doing it together, have people learn each others’ concerns and language, and start a virtuous cycle of collaboration, or split them apart and propagate the vicious cycle we know all too well where we have difficulty working together.” So we powered through that and stayed together, and it all worked out well in the end.

When we piloted this at Bazaarvoice, one of the first ops to embed got in there, worked with the devs, and put all his work into their JIRA project.  The devs got sticker shock very quickly once they saw how much work there was in delivering a reliable service – and when they dug into those tickets, they realized that they weren’t BS or busywork, but that when they thought about it they said “yeah… We certainly do need that, all this is for real!” The devs then started pulling tickets on monitoring, backups, provisioning, etc. because they realized all that workload on one person would put their delivery date behind. It was nice to see devs realize all the work that really went into doing ops – “out of sight, out of mind,” and too often devs assume ops don’t do anything except move their files to production on occasion. The embedding allowed them to rally to control their own delivery date instead of just “be blocked on the ops team.”

No one approach is “best,” but in general my experiences so far lead me to consider this one of the better models to use if you can get the organizational buy-in to the fundamental “you built it, you run it” concept.

Leave a comment

Filed under Agile, DevOps

Docker and The Future of Configuration Management – Call For Posts

Docker Docker Docker

docker-logoDocker is taking the DevOps world by storm. It has moved from “cool DevOps tool” to “lots of attention at the highest levels in the enterprise” in record time. One of the key discussion points when introducing docker, however, is how that will interact with the rest of your configuration management picture. Some people have claimed that the docker model greatly reduces the need for traditional CM; others disagree and feel that’s too simplistic an approach. (If you aren’t sure what docker is yet, check out Karthik’s recent Webcast Why To Docker? for an intro.)

What we want to do here at the Agile Admin is collect a diversity of experiences on this!  Therefore we are declaring a blog roundup on this topic where we’ll publish a bunch of articles on the topic. And we’d love for any of you out there to participate!

Come Blog With Us

Just email us (ernest dot mueller at theagileadmin.com works) or reply in the comments to this post saying “I will write something on this!” and we’ll publish your thoughts on the topic here on The Agile Admin. We’ll be starting this roundup in about a week and looking to finish up by the end of November. We’re specifically soliciting some authors to write on the topic, but we also want all of you in the DevOps community to get your say in. We can set you up as an author here in our WordPress, or you can write in Word or whatnot and mail it in, either way. Make sure and include a brief blurb about who you are at the end!

Fun and Prizes!

lego-falconStackEngine has generously agreed to sponsor this roundup, and the best community post (we’ll choose one) by Nov 30 will win a LEGO Millenium Falcon to show off their geek cred! And very soon even your kids will know what it is.

So please let us know  you’d like to participate, and then write an article – really anything touching on docker and config management is fine, even “we started to look into docker and stopped because we couldn’t figure out how to slot it in with our existing CM,” or “the simplicity of dockerfiles makes me swear off CM forever” or “golden images are the way to go” or “who needs docker anyway if you have a stellar chef implementation you get the same benefits…” No restrictions (other than those good taste and professionalism dictate). Heck, other containerization technologies than docker per se are fine too. “Containers, CM, what do you know?” No length requirements, and you can have images and videos and stuff, that’s all great.

We know keeping up a whole blog is a lot of work many technologists don’t have time for, so this is our first experiment in letting other great DevOps folks who may want to write an article every once in a while without the hassle of running their own thing use The Agile Admin to get their post out to a decent-sized audience. Come on down, we’re friendly!

StackEngineAnd thanks to StackEngine for sponsoring the roundup! They are an Austin startup that’s all about Docker tooling. Fellow Agile Admin Karthik and Boyd, who many of you know from the Austin-and-wider DevOps scene, are part of the SE team, which vouches for their awesomeness.

Leave a comment

Filed under DevOps

Innotech Austin Continuous Delivery Summit

Last week we had a DevOps track branded “CD Summit” at Innotech Austin, run by devops.com, and the agile admins were there!

I did a presentation about the various DevOps transformations I had a leadership role in at National Instruments and Bazaarvoice:

And James Wickett did a presentation on Application Security Epistemology in a Continuous Delivery World:

Jez Humble also spoke, as well as a batch of other folks including Austinite Boyd Hemphill and “our friend from Chicago” JP Morgenthal.  Once those slides are all posted I’ll pass the link on to you all!

Leave a comment

Filed under Conferences, DevOps

Why to Docker?

I recently gave a presentation on “Why to Docker” for the BrightTalk summit. Here’s a list of all the things I talked about. I had a great turnout of over 300 people, and some great questions that followed. Fortunately, I finished a bit early and was able to answer a bunch of the questions (was asked about 30). I’ll end up adding answers to all the questions this weekend!

Here’s a link to the slides:

Leave a comment

Filed under DevOps

ShirtOps: How to Make T-shirts for Tech Conferences that People Actually Wear

Over the last 6 years I have helped organize over 10 different conferences (all the LASCON conferences, all the DevOpsDays Austin conferences, AppSec USA 2012, and even a couple for my church) and for most of the events I have been in charge of swag. T-shirts, bags, shot glasses, lanyards, usb keys… You name it, I have swagged it.

From all these conferences I have learned a few things, and specifically I have learned a bit about making t-shirts. T-shirts are a funny thing. Everyone has opinions, however as an organizer you have to learn that most of those opinions are wrong. I have had lots of bad ideas recommended to me by well-meaning organizers and friends: Print the logo big! Put all the sponsors logos on the back (also known as the “the NASCAR special”). Have a big design on the back which I like to call “the restaurant shirt.” Then there is the design someone on the team knocked out with MS Paint.

Everyone has good intentions, but as the one in charge of making the shirt you have to lead them through the process. Show the team what good actually means. In this presentation I highlight the last several years of DevOpsDays Austin t-shirts and walk you through the process of how to make t-shirts people want to wear after the event is over.

Links from the presentation:

If you have any other tips, add to the comments and/or tweet with #shirtops.

7 Comments

Filed under Conferences, DevOps

Agile Austin DevOps SIG June Meeting – Change Patterns

aalogoAgile Austin has a DevOps SIG that meets monthly over lunch at BancVue, and I help Dan Zentgraf out with it. This month’s meeting is on Wednesday June 24 and is called:

Change Patterns – Don’t Get Mowed Down with the Grassroots

Come on out!  RSVP on Eventbrite; lunch is provided.

Leave a comment

Filed under Agile, DevOps

Pragmatic Security and Rugged DevOps

Turns out James (@wickett) is too shy to pimp his own stuff properly here on The Agile Admin, so I’ll do it!

As you may know James is one of the core guys behind the open source tool Gauntlt that helps you add security testing to your CI/CD pipeline.  He just gave this presentation yesterday at Austin DevOps, and it was originally a workshop at SXSW Interactive, which is certainly the big leagues.  It’s got a huge number of slides, but also has a lab where you can download Docker containers with Gauntlt and test apps installed and learn how to use it.

277 pages, 8 labs – set aside some time! Once you’re done you’re doing thorough security testing using a bunch of tools on every code deploy.

2 Comments

Filed under DevOps, Security

Devops State of the Union 2015

James, Karthik, and Ernest did a Webcast on Devops State of the Union 2015 talk for the BrightTalk Cloud Summit. It went well!  We had 187 attendees on the live feed. In this blog post we’ll add resources discussed during the talk and we will seed the comments below with all the questions we received during the webcast and answer them here – you’re all welcome to join in the discussion.

The talk was intended to be an overview of DevOps, with a bunch of blurbs on current and developing trends in DevOps – we don’t go super deep into any one of them (this was only 40 minutes long!). If you didn’t understand something, we’ve added resource links (we got some questions like “what is a container” and “what is a 12-factor app,” we didn’t have time to go into that in great detail so check some of the links below for more.

devopsstateoftheunion

Resources:

22 Comments

Filed under DevOps

Agile Admin Webcast on The State of DevOps

Tomorrow (Tuesday June 9), James, Karthik and I will be doing a DevOps State of the Union webcast live on BrightTALK, at 1 PM Central time. We’ll be taking questions and everything! You can watch it here: DevOps State of the Union 2015.

A hint on topics we might cover:

  • Containers
  • Security
  • ChatOps
  • IoT
  • Microservices
  • Donkeys

And more! Come and join us.

Leave a comment

Filed under DevOps

DevOps 101 at Innotech San Antonio

Here’s a DevOps 101 presentation based on the definition of DevOps here at The Agile Admin I’m delivering at Innotech San Antonio tomorrow as part of a devops.com attempt to spread DevOps learning to IT and the enterprise. (You probably want to go view it on slideshare.com so you can read the notes, too…)

Leave a comment

Filed under Conferences, DevOps