Category Archives: Security

Lean Security

James and I have been talking lately about the conjunction of Lean and Security.  The InfoSec world is changing rapidly, and just as DevOps has incorporated Lean techniques into the systems world, we feel that security has a lot to gain from doing the same.

We did a 20 minute talk on the subject at RSA, you can check out the slides and/or watch the video:

While we were there we were interviewed by Derek Weeks.  Read his blog post with a transcript of the interview, and/or watch the interview video!

Back here in Austin, I did an hour-long extended version of the talk for the local OWASP chapter.  Here’s a blog writeup from Kate Brew, and the slides and video:

We’ll be writing more about it here, but we wanted to get a content dump out to those who want it!

1 Comment

Filed under DevOps, Security

Links on Bridging Security and DevOps

If you remember, I (@wickett) said I would be doing more blogging for Signal Sciences in the new year. We still are in January, but I am glad to say that so far so good. Here are a couple highlights from recent posts:

That’s all for now.  Happy Friday everyone!

Leave a comment

Filed under Conferences, DevOps, Security

Pragmatic Security and Rugged DevOps

Turns out James (@wickett) is too shy to pimp his own stuff properly here on The Agile Admin, so I’ll do it!

As you may know James is one of the core guys behind the open source tool Gauntlt that helps you add security testing to your CI/CD pipeline.  He just gave this presentation yesterday at Austin DevOps, and it was originally a workshop at SXSW Interactive, which is certainly the big leagues.  It’s got a huge number of slides, but also has a lab where you can download Docker containers with Gauntlt and test apps installed and learn how to use it.

277 pages, 8 labs – set aside some time! Once you’re done you’re doing thorough security testing using a bunch of tools on every code deploy.

1 Comment

Filed under DevOps, Security

Use Gauntlt to test for Heartbleed

Heartbleed is making headlines and everyone is making a mad dash to patch and rebuild. Good, you should. This is definitely a nightmare scenario but instead of using more superlatives to scare you, I thought it would be good to provide a pragmatic approach to test and detect the issue.

@FiloSottile wrote a tool in Go to check for the Heartbleed vulnerability. It was provided as a website in addition to a tool, but when I tried to use the site, it seemed over capacity. Probably because we are all rushing to find out if our systems are vulnerable. To get around this, you can build the tool locally from source using the install instructions on the repo. You need Go installed and the GOPATH environment variable set.

go get github.com/FiloSottile/Heartbleed
go install github.com/FiloSottile/Heartbleed

Once it is installed, you can easily check to see if your site is vulnerable.
Heartbleed example.com:443

Cool! But, lets do one better and implement this as a gauntlt attack so that we can make sure we don’t have regressions and so that we can automate this a bit further. Gauntlt is a rugged testing framework that I helped create. The main goal for gauntlt is to facilitate security testing early in the development lifecycle. It does so by wrapping security tools with sane defaults and uses Gherkin (Given, When, Then) syntax so it easily understood by dev, security and ops groups.

In the latest version of gauntlt (gauntlt 1.0.9) there is support for Heartbleed–it should be noted that gauntlt doesn’t install tools, so you will still have to follow the steps above if you want the gauntlt attacks to work. Lets check for Heartbleed using gauntlt.

gem install gauntlt
gauntlt --version

You should see 1.0.9. Now lets write a gauntlt attack. Create a text file called heartbleed.attack and add the following contents:

@slow
Feature: Test for the Heartbleed vulnerability

Scenario: Test my website for the Heartbleed vulnerability (see heartbleed.com for more info)

Given "Heartbleed" is installed
And the following profile:
| name | value |
| domain | example.com |
When I launch a "Heartbleed" attack with:
"""
Heartbleed <domain>:443
"""
Then the output should contain "SAFE"

You now have a working gauntlt attack that can be hooked into your CI/CD pipeline that will test for Heartbleed. To see this example attack file on github, go to https://github.com/gauntlt/gauntlt/blob/master/examples/heartbleed/heartbleed.attack.

To run the attack

$ gauntlt ./heartbleed.attack

You should see output like this
$ gauntlt ./examples/heartbleed/heartbleed.attack
Using the default profile...
@slow
Feature: Test for the Heartbleed vulnerability

Scenario: Test my website for the Heartbleed vulnerability (see heartbleed.com for more info) # ./examples/heartbleed/heartbleed.attack:4
Given "Heartbleed" is installed # lib/gauntlt/attack_adapters/heartbleed.rb:4
And the following profile: # lib/gauntlt/attack_adapters/gauntlt.rb:9
| name | value |
| domain | example.com |
When I launch a "Heartbleed" attack with: # lib/gauntlt/attack_adapters/heartbleed.rb:1
"""
Heartbleed <domain>:443
"""
Then the output should contain "SAFE" # aruba-0.5.4/lib/aruba/cucumber.rb:131

1 scenario (1 passed)
4 steps (4 passed)
0m3.223s

Good luck! Let me (@wickett) know if you have any problems.

2 Comments

Filed under DevOps, Security

Product Security the Netflix Way with Jason Chan

Jason Chan did a great presentation at LASCON last fall in Austin on product security at Netflix offically titled ‘From Gates to Guardrails: Alternate Approaches to Product Security.’  You may have even seen the Agile Admin’s coverage of LASCON and @ernestmueller‘s interview of Jason Chan.  The LASCON videos are now online and we thought we would share some of our favorites from the conference.

Leave a comment

by | January 18, 2014 · 8:30 am

LASCON Interview: Jason Chan

 IMG_1513Jason Chan (@chanjbs) is an Engineering Director of the Cloud Security team at Netflix.

Tell me about your current gig!

I work on the Cloud Security team at Netflix, we’re responsible for the security of the streaming service at Netflix.  We work with some other teams on platform and mobile security.

What are the biggest threats/challenges you face there?

Protecting the personal data of our members of course.  Also we have content we want to protect – on the client side via DRM, but mainly the pipeline of how we receive the content from our studio partners. Also, due to the size of the infrastructure, its integrity – we don’t want to be a botnet or have things injected to our content that can our clients.

How does your team’s approach differ from other security teams out there?

We embody the corporate culture more, perhaps, than other security teams do. Our culture is a big differentiator between us and different companies.  So it’s very important that people we hire match the culture. Some folks are more comfortable with strong processes and policies with black and white decisions, but here we can’t just say now, we have to help the business get things done safely.

You build a security team and you have certain expertise on it.  It’s up to the company how you use that expertise. They don’t necessarily know where all the risk is, so we have to provide objective guidance and then mutually come to the right decision of what to do in a given situation.

Tell us about how you foster your focus on creating tools over process mandates?

We start with recruiting, to understand that policy and process isn’t the solution.  Adrian [Cockroft] says process is usually organizational scar tissue. By doing it with tools and automation makes it more objective and less threatening to people. Turning things into metrics makes it less of an argument. There’s a weird dynamic in the culture that’s a form of peer pressure, where everyone’s trying to do the right thing and no one wants to be the one to negatively impact that.  As a result people are willing to say “Yes we will” – like, you can opt out of Chaos Monkey, but people don’t because they don’t want to be “that guy.”

We’re starting to look at availability in a much  more refined way.  It’s not just “how long were you down.”  We’re establishing metrics over real impact – how many streams did we miss?  How many start clicks went unfulfilled.  We can then assign rough values to each operation (it’s not perfect, but based on shared understanding) and then we can establish real impact and make tradeoffs. (It’s more story point-ish instead of hard ROI). But you can get what you need to do now vs what can wait.

Your work  – how much is reactive versus roadmapped tool development?

It’s probably 50/50 on our team.  We have some big work going on now that’s complex and has been roadmapped for a while.  We need to have bandwidth as things pop up though, so we can’t commit everyone 100%. We have a roadmap we’ve committed to that we need to build, and we keep some resource free so that we can use our agile board to manage it. I try to build the culture of “let’s solve a problem once,” and share knowledge, so when it recurs we can handle it faster/better.  I feel like we can be pretty responsive with the agile model, our two week sprints and quarterly planning give us flexibility. We get more cross-training too, when we do the mid-sprint statuses and sprint meetings. We use our JIRA board to manage our work and it’s been very successful for us.

What’s it like working at Netflix?

It’s great, I love it.  It’s different because you’re given freedom to do the right thing, use your expertise, and be responsible for your decisions. Each individual engineer gets to have a lot of impact on a pretty large company.  You get to work on challenging problems and work with good colleagues.

How do you conduct collaboration within your team and with other teams?

Inside the team, we instituted once a week or every other week “deep dives” lunch and learn presentation of what you’re working on for other team members. Cross-team collaboration is a challenge; we have so many tools internally no one knows what they all are!

You are blazing trails with your approach – where do you think the rest of the security field is going?

I don’t know if our approach will catch on, but I’ve spent a lot of my last year recruiting, and I see that the professionalization of the industry in general is improving.  It’s being taught in school, there’s greater awareness of it. It’s going to be seen as less black magic, “I must be a hacker in my basement first” kind of job.

Development skills are mandatory for security here, and I see a move away from pure operators to people with CS degrees and developers and an acceleration in innovation. We’ve filed three patents on the things we’ve built. Security isn’t’ a solved problem and there’s a lot left to be done!

We’re working right now on a distributed scanning system that’s very AWS friendly, code named Monterey. We hope to be open sourcing it next year.  How do you inventory and assess an environment that’s always changing? It’s a very asynchronous problem. We thought about it for a while and we’re very happy with the result – it’s really not much code, once you think the problem through properly your solution can be elegant.

1 Comment

Filed under Cloud, Conferences, Security