Tag Archives: interview

LASCON Interview: Jason Chan

 IMG_1513Jason Chan (@chanjbs) is an Engineering Director of the Cloud Security team at Netflix.

Tell me about your current gig!

I work on the Cloud Security team at Netflix, we’re responsible for the security of the streaming service at Netflix.  We work with some other teams on platform and mobile security.

What are the biggest threats/challenges you face there?

Protecting the personal data of our members of course.  Also we have content we want to protect – on the client side via DRM, but mainly the pipeline of how we receive the content from our studio partners. Also, due to the size of the infrastructure, its integrity – we don’t want to be a botnet or have things injected to our content that can our clients.

How does your team’s approach differ from other security teams out there?

We embody the corporate culture more, perhaps, than other security teams do. Our culture is a big differentiator between us and different companies.  So it’s very important that people we hire match the culture. Some folks are more comfortable with strong processes and policies with black and white decisions, but here we can’t just say now, we have to help the business get things done safely.

You build a security team and you have certain expertise on it.  It’s up to the company how you use that expertise. They don’t necessarily know where all the risk is, so we have to provide objective guidance and then mutually come to the right decision of what to do in a given situation.

Tell us about how you foster your focus on creating tools over process mandates?

We start with recruiting, to understand that policy and process isn’t the solution.  Adrian [Cockroft] says process is usually organizational scar tissue. By doing it with tools and automation makes it more objective and less threatening to people. Turning things into metrics makes it less of an argument. There’s a weird dynamic in the culture that’s a form of peer pressure, where everyone’s trying to do the right thing and no one wants to be the one to negatively impact that.  As a result people are willing to say “Yes we will” – like, you can opt out of Chaos Monkey, but people don’t because they don’t want to be “that guy.”

We’re starting to look at availability in a much  more refined way.  It’s not just “how long were you down.”  We’re establishing metrics over real impact – how many streams did we miss?  How many start clicks went unfulfilled.  We can then assign rough values to each operation (it’s not perfect, but based on shared understanding) and then we can establish real impact and make tradeoffs. (It’s more story point-ish instead of hard ROI). But you can get what you need to do now vs what can wait.

Your work  – how much is reactive versus roadmapped tool development?

It’s probably 50/50 on our team.  We have some big work going on now that’s complex and has been roadmapped for a while.  We need to have bandwidth as things pop up though, so we can’t commit everyone 100%. We have a roadmap we’ve committed to that we need to build, and we keep some resource free so that we can use our agile board to manage it. I try to build the culture of “let’s solve a problem once,” and share knowledge, so when it recurs we can handle it faster/better.  I feel like we can be pretty responsive with the agile model, our two week sprints and quarterly planning give us flexibility. We get more cross-training too, when we do the mid-sprint statuses and sprint meetings. We use our JIRA board to manage our work and it’s been very successful for us.

What’s it like working at Netflix?

It’s great, I love it.  It’s different because you’re given freedom to do the right thing, use your expertise, and be responsible for your decisions. Each individual engineer gets to have a lot of impact on a pretty large company.  You get to work on challenging problems and work with good colleagues.

How do you conduct collaboration within your team and with other teams?

Inside the team, we instituted once a week or every other week “deep dives” lunch and learn presentation of what you’re working on for other team members. Cross-team collaboration is a challenge; we have so many tools internally no one knows what they all are!

You are blazing trails with your approach – where do you think the rest of the security field is going?

I don’t know if our approach will catch on, but I’ve spent a lot of my last year recruiting, and I see that the professionalization of the industry in general is improving.  It’s being taught in school, there’s greater awareness of it. It’s going to be seen as less black magic, “I must be a hacker in my basement first” kind of job.

Development skills are mandatory for security here, and I see a move away from pure operators to people with CS degrees and developers and an acceleration in innovation. We’ve filed three patents on the things we’ve built. Security isn’t’ a solved problem and there’s a lot left to be done!

We’re working right now on a distributed scanning system that’s very AWS friendly, code named Monterey. We hope to be open sourcing it next year.  How do you inventory and assess an environment that’s always changing? It’s a very asynchronous problem. We thought about it for a while and we’re very happy with the result – it’s really not much code, once you think the problem through properly your solution can be elegant.

1 Comment

Filed under Cloud, Conferences, Security

LASCON Interview: Nick Galbreath

IMG_1509Nick Galbreath (@ngalbreath) is VP of Engineering with client9, LLC.

What are you doing nowadays since leaving Etsy?

I am managing a small DevOps team for a company whose engineering team is based in Moscow, from Tokyo, Japan. Some other executives and our biggest customer is from there. And, I love Japan!

I know you from Velocity and the other DevOps conferences. Why are you here at a security conference?

I’ve been active at Black Hat, DEFCON, etc. as well as DevOps conferences. I’ve found that if your company is in operational chaos you don’t need security.  Once you have a good operational component and it’s not in chaos – standardized infrastructure, automation – you get up to the level where you can be effective at security.  I used the same approach at Etsy – I started there working on security, stopped, worked in infrastructure until that was basically squared away, and only then started working on security again. You have to work your way up Maslow’s hierarchy.

It’s the same with development. My background is originally development and when you’re programming in C/C++ your main effort is stability, but all those NPEs and other bugs are also security issues.  I don’t know any company doing well at security and not well at development, I’m not sure you can do it. Nail the basics and then the advanced topics are achievable.

What’s your opinion on how much the security space has left developers behind?

Look at the real core issues behind security. Dev teams have trouble with writing secure code, ops folks have problems with patching – at security conferences you don’t see anything for solving those problems.  Working on offense/breaking and blocking tools is lucrative but inhibits us from going after the root causes.

For many security pros, working in a team instead of solo is a different skill set. “We don’t want to bother the developers with this” – siloed approaches are killing us.

What do you see as the most interesting thing going on in the security landscape right now?

What has happened in the last 3-4 months, as much as I hate to say it, with all the leaking of documents – we’ve been lazy about encryption and privacy and other foundational elements and we assumed it worked, now we’re doing some healthy review to do a next generation of those. It brought that discussion to the forefront. The certificate authority problems, and the NSA stuff – we need to spend some time and think about this.  The next generation of SSL and certificate transparency are very interesting.

In terms of pure language work… Improvement of cryptography. Also, we’re making more business level APIs for common problems like PHP5’d password hashing APIs.  If your’e building a Web app and need auth you’re starting from zero most of the time and now you’re starting to see things put into the languages that solve these problems.

Out in the larger DevOpsey world, what are the things to watch, what is your team excited about?

Stuff that we’re excited about is traditional devops stuff like really treating our infrastructure like code.  No button clicking, infrastructure completely specified in config files in source control, code reviews, and then the file pushed to production to allocate/deallocate hardware and deploy software.  That’s a big change.

How do we disseminate best practices/prevent worst practices through those who aren’t the technical “1%?”

Well, best practices are harder

People went into server programming because they don’t like doing user interface stuff. But the joke’s on us, there is still a user interface, it’s configuration files, installers, etc. which are nontrivial. We should either be bundling audit software or server-side config healthchecks to provide warnings. “Why do you have SSL v2 enabled?” “Why are your .htaccess files visible by default?” [Ed: Where the hell did apache chkconfig go?]

People in ops can write these but retroactively folks won’t use them… But the future can have them.  If you at least get warned that your Apache config is using suboptimal security configs it’s your deliberate negligence to not do it right.

Maybe take the module approach (Apache wouldn’t want it in their core I’m sure) – if you want to work on it give me a call!

What message do you want to send to other security folks?

For security people, the message is, “It’s really important you start bringing your  non-security friends to these security conferences.” Devs and ops and business and QA. They’ll find it interesting and get involved. It’s really important.

Last year, we had a dozen people from my company come out to AppSec. But except for me and our security team, they’re not back this year. There just wasn’t enough content to hold the interest of the devs. What can we do about that?

Really!  Interesting.  Maybe we need more of a proper dev track, with more things like Karthik’s talk.

A project I’ve wanted to do for a very long time – most people in business and development don’t have  real idea of how much damage can be done, it’s why we have Red Teams. If someone’s really good at SQLi, etc. do a talk showing how much damage can be done.

Also – if you work at any company, you depend on an immense set of open source software and they don’t have a security person or anything.  Get involved in their process, try to help them and make it better and it’ll improve quality of everyone’s systems. We could do a hackathon during the convention to improve some existing projects.

Leave a comment

Filed under Conferences, DevOps, Security