Category Archives: DevOps

Pertaining to agile system administration concepts and techniques.

Use Gauntlt to test for Heartbleed

Heartbleed is making headlines and everyone is making a mad dash to patch and rebuild. Good, you should. This is definitely a nightmare scenario but instead of using more superlatives to scare you, I thought it would be good to provide a pragmatic approach to test and detect the issue.

@FiloSottile wrote a tool in Go to check for the Heartbleed vulnerability. It was provided as a website in addition to a tool, but when I tried to use the site, it seemed over capacity. Probably because we are all rushing to find out if our systems are vulnerable. To get around this, you can build the tool locally from source using the install instructions on the repo. You need Go installed and the GOPATH environment variable set.

go get github.com/FiloSottile/Heartbleed
go install github.com/FiloSottile/Heartbleed

Once it is installed, you can easily check to see if your site is vulnerable.
Heartbleed example.com:443

Cool! But, lets do one better and implement this as a gauntlt attack so that we can make sure we don’t have regressions and so that we can automate this a bit further. Gauntlt is a rugged testing framework that I helped create. The main goal for gauntlt is to facilitate security testing early in the development lifecycle. It does so by wrapping security tools with sane defaults and uses Gherkin (Given, When, Then) syntax so it easily understood by dev, security and ops groups.

In the latest version of gauntlt (gauntlt 1.0.9) there is support for Heartbleed–it should be noted that gauntlt doesn’t install tools, so you will still have to follow the steps above if you want the gauntlt attacks to work. Lets check for Heartbleed using gauntlt.

gem install gauntlt
gauntlt --version

You should see 1.0.9. Now lets write a gauntlt attack. Create a text file called heartbleed.attack and add the following contents:

@slow
Feature: Test for the Heartbleed vulnerability

Scenario: Test my website for the Heartbleed vulnerability (see heartbleed.com for more info)

Given "Heartbleed" is installed
And the following profile:
| name | value |
| domain | example.com |
When I launch a "Heartbleed" attack with:
"""
Heartbleed <domain>:443
"""
Then the output should contain "SAFE"

You now have a working gauntlt attack that can be hooked into your CI/CD pipeline that will test for Heartbleed. To see this example attack file on github, go to https://github.com/gauntlt/gauntlt/blob/master/examples/heartbleed/heartbleed.attack.

To run the attack

$ gauntlt ./heartbleed.attack

You should see output like this
$ gauntlt ./examples/heartbleed/heartbleed.attack
Using the default profile...
@slow
Feature: Test for the Heartbleed vulnerability

Scenario: Test my website for the Heartbleed vulnerability (see heartbleed.com for more info) # ./examples/heartbleed/heartbleed.attack:4
Given "Heartbleed" is installed # lib/gauntlt/attack_adapters/heartbleed.rb:4
And the following profile: # lib/gauntlt/attack_adapters/gauntlt.rb:9
| name | value |
| domain | example.com |
When I launch a "Heartbleed" attack with: # lib/gauntlt/attack_adapters/heartbleed.rb:1
"""
Heartbleed <domain>:443
"""
Then the output should contain "SAFE" # aruba-0.5.4/lib/aruba/cucumber.rb:131

1 scenario (1 passed)
4 steps (4 passed)
0m3.223s

Good luck! Let me (@wickett) know if you have any problems.

2 Comments

Filed under DevOps, Security

DevOps Updated

Since I had some time and have been thinking about it lately, I’ve upgraded and expanded my definition of DevOps on theagileadmin. Healthy debate welcome!

Leave a comment

by | March 20, 2014 · 3:48 pm

DevOpsDays Austin Is Coming!

The third annual DevOpsDays conference in Austin will be May 5-6 (Cinco de Mayo!) at the Marchesa, where it was held last year! As many of you know, the DevOpsDays conferences are a super popular format – half talks from practitioners, half openspaces, all fun – held in many cities around the world since the first one in Ghent launched the DevOps movement proper.

  • You can register – all the early bird tickets are sold out but the regular ones are only half gone.
  • You can also propose a talk!  There’s 35-minute full talk slots but we’re even more in need of 5-minute Ignite! style lightning talks! RFP ends 3/26 sp
  • You can sponsor! The Gold sponsorships are half gone already. And we have some special options this year…

DevOpsDays Austin has been bigger and better every year since its inception and should have something good for everyone this year. Come out and join your comrades from the trenches who are trying to forge a new way of delivering and maintaining software!

1 Comment

Filed under Conferences, DevOps

Agile Austin asked me to help re-launch their blog, so I’ve contributed a piece on “What Is DevOps?” for them!

Leave a comment

by | March 16, 2014 · 9:58 am

Agile Organization Incorporating Various Disciplines

I started thinking about this recently because there was an Agile Austin QA SIG meeting that I sadly couldn’t attend entitled “How does a QA manager fit into an Agile organization?” which wondered about how to fit members of other disciplines (in this case QA) in with agile teams. Over the last couple years I’ve tried this, and seen it tried, in several ways with DevOps, QA, Product Management, and other disciplines, and I thought I’d elaborate on the pros and cons of some of these approaches.

Two Fundamental Discoveries

There are two things I’ve learned from this process that are pretty universal in terms of their truth.

1. Conway’s Law is true. To summarize, it states that a product will tend to reflect the structure of the organization that produces it. The corollary is that if your organization has divisions which are of no practical value to the product’s consumer, you will be creating striations within your product that impact client satisfaction. Hence basic ITSM and Agile doctrines on creating teams around owning a service/product.

2. People want to form teams and stay with them. This should be obvious from basic psychology/sociology, but if you set up an organization that is too flexible it strongly degrades the morale of your workers. In my previous role we conducted frequent engineer satisfaction surveys and the most prominent truth drawn from them is the more frequently people are asked to change roles, reorg, move to different teams, the less happy they are. Even people that want to move around to new challenges frequently are happier if they are moving to those new challenges with a team they’ve had an opportunity to move through Tuckman’s stages of group development with.

I have seen enough real-world quantified proof of both of these assertions that I will treat them as assumptions going forward.

Organizational Options

We tried out all four of these models within the same organization of high performing engineers and thus had a great opportunity to compare their results.

Separate Teams

When we started, we had the traditional model of separate teams which would hand off work to each other.  “Dev,” “Ops,” “QA,” “Product” were all under separate management up through several levels and operated as independent teams; individual affinities with specific products were emergent and simply matters of convenience (e.g. “Oh, he knows a lot about that BI stuff, let him handle that request”) and not a matter of being dedicated to specific product(s).

Embedded Crossfunctional Service Teams

Our first step away from the pure separate team model was to take those separate teams and embed specific members from them into service oriented teams, while still having them report to a manager or director representing their discipline. In some cases, the disciplinary teams would reserve some number of staff for tool development or other cross-cutting concerns. So QA, for example, had several engineers assigned to each product team, even though they were regarded as part of the permanent QA org primarily.

We (very loosely) considered our approach to be decentralized and microservice based; Martin Fowler is doing a good article in installments on Microservice Architecture if you want more on that topic.

Fully Integrated Service Teams

With our operations staff we went one step farther and simply permanently assigned them to product teams and removed the separate layer of management entirely. Dev and “DevOps” engineers reported to the same engineering manager and were a permanent part of a given product team. Any common tooling needed was created by a separate “platform” engineering team which was similarly integrated.

Project Based Organization

Due to the need to surge effort at times, we also had some organizations that were project, not product, based. Engineers would be pulled either from existing teams or entire teams would additionally be pulled into a short term (1, 2, 3 month) effort to try to make significant headway across multiple products, and then dissolve afterwards.

Hmm, this looks like it’s getting big (and I need to do some diagrams).  I’ll break it up into separate articles for each type of org and its pros and cons, and then a conclusion.

12 Comments

Filed under Agile, DevOps

Stupid webappsec Tricks Talk at LASCON with Zane Lackey

Zane Lackey spoke at LASCON 2013 about how they do data driven security at Etsy. At the conference Ernest took some notes and blogged them in this post: . Now that the LASCON vids are out we thought this would be a good time to revisit this stellar talk. Enjoy!

Leave a comment

Filed under DevOps

Clean up your cookbook mess with meez

Is your kitchen a mess? Meez will help you get things straightened out.

There is a new gem in town, and it’s here to clean up the mess you made out of your cookbooks.  Its called meez.  

If you are like me, maybe you started writing some chef cookbooks, and then later decided to add some testing and you followed some blog posts to set up some different tools.  Some where along the way you figured out that the cool kids don’t use Librarian (although I still am fond of it) so you decide to use Berkshelf (I am learning to like it).  You also figured out that you need a linting tool and some sort of way to do TDD for your infrastructure. Man, this cookbook is starting to get pretty crowded with a bunch of files that have nothing to do with actually installing the code you want to install.  You also start looking around and wondering why you have to learn all these esoteric frameworks/tools to write a simple chef cookbook (technically you don’t have to, but the technohipsters frown on you if you don’t).

What are you to do?

Enter meez. Meez sets up an opinionated cookbook replete with all the testing tools and frameworks a modern chef requires: chefspec, foodcritic, rubocop, berkshelf, kitchenci, …  Once you tell meez to create a cookbook for you, it sets up all the different frameworks and gets you ready to start actually writing your recipes and working on your cookbook.  No more remembering how to setup all the testing tools and frameworks.  Sweet!


gem install meez
meez --cookbook-path /tmp -C "James Wickett" -m james@wickett.me mycookbook

What this will do is set up ‘mycookbook’ with all the testing tools you need.  By giving it my name and email, it autofills all that in the relevant spots as well.  Once meez finishes running, it tells you what to do next:


You must run `bundle install' to fetch any new gems.
Cookbook mycookbook created successfully
Next steps...
$ cd /tmp/mycookbook
$ bundle install
$ bundle exec berks install
$ bundle exec strainer test

Follow those steps and you are now ready to start working on cookbooks and stop worrying about all the testing frameworks and tools surrounding TDD and chef.

Meez was created as a gem after @pczarkowski‘s excellent sysadvent blog post “The Lazy SysAdmin’s Guide to Test Driven Chef Cookbooks.” Reading that will give you more context behind what meez is doing.

Moar Links

1 Comment

Filed under DevOps

Trusted Software Alliance launches new podcast and news series

The Trusted Software Alliance News Network launched this week and is featuring 5 minute daily doses of AppSec and DevOps news. The show is run by @eusp along with weekly co-hosts @damonedwards@cote and yours truly (@wickett).  Check out the inaugural post and follow the blog at trustedsoftwarealliance.com.

Leave a comment

by | January 17, 2014 · 9:56 am

Evolution of Bazaarvoice’s Architecture to 500M Unique Users Per Month

Check out this article by @victortrac on High Scalability on how we have scaled our infrastructure at Bazaarvoice to be serving out a billion product reviews a day!

Leave a comment

by | December 2, 2013 · 3:07 pm

A DevOps Thanksgiving

This last week at the Agile Austin DevOps SIG, our topic was simple – “A DevOps Thanksgiving.” We all shared what we’re thankful for from the DevOps world this year – things that have made our lives better.

It was a nice and refreshing discussion!  People mentioned the things making their lives better. Group members expressed their thanks for such diverse things as DevOps Weekly, rspec-puppet, The Phoenix Project, Vagrant, Docker, test-kitchen with serverspec and bats, provisioned IOPS in AWS, DevOps Cafe, The Ship Show, increasing crossplatform support in DevOps tools and thinking, DevOps tracks springing up at conferences like Agile 2013 and AppSec, DevOpsDays… Thanks to all the people who put in lots of their hard work to make them all possible!

In retrospect we have a lot to be thankful for.  Even though the techno-hipsters don’t even want to say the word “DevOps” any more, it’s a very real change bringing better things to our tools, products, and even lives. I know I’ve seen a lot of change in the teams I’ve worked with that have implemented it – fewer “all hands overnight releases,” less psychotic oncall, less inter-group hatefulness – DevOps has brought us all a lot of good things, and it’s just starting to take hold out there in the industry.

How about you?  What DevOps thing were you thankful for this year?  Add into the comments here, blog it up yourself, tweet it (I suggest #devopsthanksgiving as the hashtag)…  Spread the thanks!

Leave a comment

Filed under DevOps