Tag Archives: rugged

Security and the Rise (and Fall?) of DevOps

As I’ve been involved with DevOps and its approach of blending development and operations staff together to create better products, I’ve started to see similar trends develop in the security space. I think there’s some informative parallels where both can learn from each other and perhaps avoid some pitfalls.

Here’s a recent article entitled “Agile: Most security guys are useless” that states the problem succinctly. In successful and agile orgs, the predominant mindset is that if you’re not touching the product, you are semi-useless overhead. And there’s some truth to that. When people are segregated into other “service” orgs – like operations or security – the us vs. them mindset predominates and strangles innovation in its crib.

The main initial drive of agile was to break down that wall between the devs and the “business”, but walls remain that need similar breaking down. With DevOps, operations organizations faced with this same problem are innovating new approaches; a collaborative approach with developers and operations staff working together on the product as part of the same team. It’s working great for those who are trying it, from the big Web shops like Facebook to the enterprise guys like us here at NI. The movement is gathering steam and it seems clear to those of us doing it this way that it’s going to be a successful and disruptive pattern for adopters.

But let’s not pat ourselves on the back too much just yet. We still have a lot of opportunity to screw it up. Let’s review an example from another area.

In the security world, there is a whole organization, OWASP (the Open Web Application Security Project) whose goal is to promote and enable application security. Security people and developers, working together!  Dev+Sec already exists! Or so the plan was.

However, recently there have been some “shots across the bow” in the OWASP community.  Read Security People vs Developers and especially OWASP: Has It Reached A Tipping Point? The latter is by Mark Curphey, who started OWASP. He basically says OWASP is becoming irrelevant because it’s leaving developers behind. It’s becoming about “security professionals” selling tools and there’s few developers to be found in the community any more.

And this is absolutely true.  We host the Austin OWASP chapter here at NI’s Austin campus, and two of the officers are NI employees. We make sure and invite NI developers to come to OWASP. Few do, at least not after the first couple times.  I asked some of the devs on our team why not, and here’s some answers I got.

  • I want to leave sessions by saying, “I need to think about this the next time I code”. I leave sessions by saying, “that was cool, I can talk about this at a happy hour”. If I could do the former, I’d probably attend most/all the sessions.
  • A lot of the sessions don’t seem business focused and it is hard to relate. Demos are nicer; but a lot of times they are so specific (for example: specific OS + specific Java Version + specific javascript library = hackable) that it’s not actionable.
  • “Security people” think “developers” don’t know what they are doing and don’t care about security. Which to developers is offensive. We like to write secure applications; sometimes we just find the bugs too late….
  • I’ve gone to, I think, 4 OWASP meetings.  Of those, I probably would only have recommended one of them to others – Michael Howard’s.  I think it helped that he was a well-known speaker and seemed to have a developer focus. So, well-known speakers, with a compelling and relevant subject..  Even then, the time has to be weighed against other priorities.  For example, today’s meeting sounds interesting, but not particularly relevant.  I’ll probably skip it.

In the end, the content at these meetings is more for security pros like pen testers, or for tool buyers in security or sysadmin groups. “How do I code more securely” is the alleged point of the group but frankly 90% of the activity is around scanners and crackers and all kinds of stuff that is fine but should be simple testing steps after the code’s written securely in the first place.

As a result there have been interesting ideas coming from the security community that are reminiscent of DevOps concepts. Pen tester @atdre did a talk here to the Austin OWASP chapter about how security testers engaging with agile teams “from the outside” are failing, and shouldn’t we instead embed them on the team as their “security buddy.” (I love that term.  Security buddy. I hate my “compliance auditor” without even meeting the poor bastard, but I like my security buddy already.) At the OWASP convention LASCON, Matt Tesauro delivered a great keynote similarly trying to refocus the group back on the core problem of developing secure software; in fact, they’re co-sponsoring a movement called “Rugged” that has a manifesto similar to the Agile Manifesto but is focused on security, availability, reliability, et cetera. (As a result it’s of interest to us sysadmin types, who are often saddled with somehow applying those attributes in production to someone else’s code…)

The DevOps community is already running the risk of “leaving the devs behind” too.  I love all my buddies at Opscode and DTO and Puppet Labs and Thoughtworks and all. But a lot of DevOps discussions have started to be completely sysadmin focused as well; a litany of tools you can use for provisioning or monitoring or CI. And that wouldn’t be so bad if there was a real entry point for developers – “Here’s how you as a developer interact with chef to deploy your code,” “Here’s how you make your code monitorable”. But those are often fringe discussions around the core content which often mainly warms the cockles of a UNIX sysadmin’s heart. Why do any of my devs want to see a presentation on how to install Puppet?  Well, that’s what they got at a recent Austin Cloud User Group meeting.

As a result, my devs have stopped coming to DevOps events.  When I ask them why, I get answers similar to the ones above for why they’re not attending OWASP events any more. They’re just not hearing anything that is actionable from the developer point of view. It’s not worth the two hours of their valuable time to come to something that’s not at all targeted at them.

And that’s eventually going to scuttle DevOps if we let it happen, just as it’ll scuttle OWASP if it continues there. The core value of agile is PEOPLE over processes and tools, COLLABORATION over negotiation. If you are leaving the collaboration behind and just focusing on tools, you will eventually fail, just in a more spectacular and automated fashion.

The focus at DevOpsDays US 2010 was great, it was all about culture, nothing about tools. But that culture talk hasn’t driven down to anything more actionable, so tools are just rising up to fill the gap.

In my talk at that DevOpsDays I likened these new tools and techniques to the introduction of the Minie ball to rifles during the Civil War. In that war, they adopted new tools and then retained their same old tactics, walking up close in lines designed for weapons with much shorter ranges and much lower accuracy – and the slaughter was profound.

All our new DevOps tools are great, but in the same way, if we don’t adapt our way of thinking to them, they will make our lives worse, not better, for all their vaunted efficiency. You can do the wrong thing en masse and more quickly. The slaughter will similarly be profound.

A sysadmin suddenly deciding to code his own tools isn’t really the heart of DevOps.  It’s fine and good, and I like seeing more tools created by domain experts. But the heart of DevOps, where you will really see the benefits in hard ROI, is developers and operations folks collaborating on real end-consumer products.

If you are doing anything DevOpsey, please think about “Why would a developer care about this?” How is it actionable to them, how does it make their lives easier?  I’m a sysadmin primarily, so I love stuff that makes my job easier, but I’ve learned over the years that when I can get our devs to leverage something, that’s when it really takes off and gives value.

The same thing applies to the people on the security side.  Why do we have this huge set of tools and techniques, of OWASP Top 10s and Live CDs and Metasploits and about a thousand wonderful little gadgets, but code is pretty much as shitty and insecure as it was 20 years ago? Because all those things try to solve the problem from the outside, instead of targeting the core of the matter, which is developers developing secure code in the first place.  And to do that, it’s more of a hearts-and-minds problem than a tools-and-processes problem.

That’s a core realization that operations folks, and security folks, and testing folks, and probably a bunch of other folks need to realize, deeply internalize, and let it change the way they look at the world and how they conduct their work.


Filed under DevOps, Security

Rugged Software Manifesto: An Interview With Dan Cornell

I had a chance to talk with Dan Cornell from the Open Web Application Security Project (OWASP) and the Denim Group.  Dan has over 13 years of experience in development and is one of the founders at the Denim Group, a company that does security consulting day in and day out for all types of clients. Dan wrote Sprajax, a security analyzer for AJAX applications, and is the chair of the OWASP Global Membership Committee.

While a lot of us work with security, it is usually in addition to other dev or ops responsibilities and isn’t our sole focus.  However, Dan Cornell and his company are in the weeds of app security daily.  It is a pleasure to have him on the agile admin blog and we are chatting with him on his thoughts on the Rugged Software Manifesto. We ran across this manifesto at the Lonestar Application Security Conference (LASCON) and heard it referred to by both OWASP board members and Department of Homeland Security types so we thought we should look it up, and it seemed interesting.  So here’s Dan’s thoughts on the subject.

@wickett: Hi Dan, welcome to the agile admin blog.  Our first question for you is, what do we mean by Rugged Software?

@danielcornell:  I think of software that is going to run as intended, regardless of the conditions it encounters when it is deployed.  A big part of this is security, but that isn’t the only thing.  You also have to take into account reliability, survivability and a variety of other ‘-ity’ properties.  The Internet, and networks in general, are dangerous places where you need to be conscious of both malicious actors as well as other, unintentional adverse conditions such as service outages, congestion, untrained users making mistakes and so on.  Rugged Software is software that is designed and built to deal with these conditions – foreseen or not.

@wickett: When we have talked about Rugged Software with developers, we get a fair amount of pushback and hear statements like, “Rugged Software isn’t really real and that it is just a PR play.”  Some developers and engineers we;ve talked to say that it lacks depth, clarity and process, but I wonder if these engineers have even seen the original Agile Manifesto.  In my mind, I look back at the original Agile Manifesto and I am equally under-whelmed by it as I imagine they are by the Rugged Software Manifesto.  Do you see any parallels between the original Agile Manifesto and the Rugged Software Manifesto?

@danielcornell: The two biggest challenges I see for the Rugged Software movement are to provide a business justification for being Rugged to management and to answer the “so what?” question for developers.  Dealing with the first will go a long way toward solving the second.

If you ask most executives and managers if they’d like their software to be Rugged they’ll likely give an answer somewhat along the lines of: “Sure I’d love to be ‘Rugged.’  And ‘Agile’ and ‘Pragmatic’ and whatever other buzzwords the industry has come up with lately.  All that stuff is great but what I really want is to be on time and on budget and I want more features than our competitors.”  Thinking about security and quality and performance and anything else comes later.

So this is an area where the Rugged movement needs to do a better job of explaining why Rugged makes good business sense and you run into the same challenges that information security folks have had making a business justification for security for years.  Allowing an organization to avoid pain when the auditors or compliance folks come around is of value – but it often isn’t enough.  Reducing downtime and disruption in the future because you have avoided a breach or outage is of value – but it often isn’t enough.  David Rice has done some great thinking in this area in his book Geekonomics but, as an industry, I think we are still looking for the universal truth to be revealed that will make everyone sit up and take notice and I think we’re going to be waiting for a long time.

That said, some organizations have made the decision that this is an area that merits focus and when executives and managers make Rugged (or security or whatever) a priority then it is much easier to get the troops to fall in line.  I’m reminded of a secure coding training class I ran recently for an ISV where the VP in charge of engineering stood up at the beginning of the class and said “This is a priority for us and the material in this class represents the base level of knowledge everyone is expected to have.”  Then he sat in the entire two day class and at the end told his people “I hope you paid attention because you are going to be held accountable for knowing and implementing this material.”  Wow!  This was one of the best classes I taught in a long time; everyone showed up on time, paid attention, asked questions and so on.  I’d love to think it is because I’m such a funny and engaging guy but what really made the difference is that the leadership demonstrated with actions that security is a priority.  A similar model is always going to be the most effective way to drive Rugged into an organization.  Bottom-up, grass roots efforts are valuable because they raise awareness and help to identify like-minded potential champions, but to drive adoption at scale management needs to feel that this is a priority.

Speaking of priorities, I think this is an area where the Agile Manifesto style can provide some guidance.  The Rugged Manifesto is a great call to arms, but it is very self-referential: “I am Rugged… and more importantly my code is Rugged”  And so on.  If you look at the Agile Manifesto it calls out trade offs: “Individuals and interactions over processes and tools,” for example.  It accepts that processes and tools have merit, but chooses to value individuals and interactions more.  Acknowledging that decisions must be made and providing guidance on why one is superior to another is a stronger basis for a conversation and this is going to be key.  Right now those trade-offs are presented as “Prefer code without vulnerabilities over code with vulnerabilities”  Duh!  Who wouldn’t want that?  But the real question is what are they going to have to give up to get it.

@wickett: Over the next five years, do you see the Rugged Software Manifesto becoming a standard?  Will we have Rugged development practices and be operating on a new paradigm for development or will it get lumped under Agile?

@danielcornell: Well the Rugged Software Manifesto is just that – a Manifesto.  It isn’t a standard.  And I don’t think we are ever going to see a standard for software development because “development” is too big a thing to be done in a standard way.  You can’t expect folks building banking applications for East Coast financials to follow the same process as a couple of folks in a garage trying to build the next Facebook.  What I hope is developed and adopted, however, are so-called “best practices” such as threat modeling, static analysis for quality and security, security code review, penetration testing and so on.

Also, Agile and Rugged are two separate movements.  Agile is about creating customer-centric software in a flexible and timely manner and Rugged is about making software that doesn’t break.   Their goals are not exclusive, but they’re also different.

@wickett: When thinking about Rugged Software, what are practical steps that individual developers can adopt into their current practices whether they use Agile, Waterfall, or other development methodology?

@danielcornell: Well first I think it is important to note that individual developers can have a positive impact on their own code, but this is going to be largely ineffective if the rest of the development team doesn’t share the same goals.  The “weakest link in the chain” analogy comes to mind.  That said…

Threat Modeling is a practice that can be adopted into any development methodology.  There are lots of resources out there that explain how to use it to improve the security of complicated systems and it can also be used for the greater purpose of Rugged software by asking questions such as “what if this data flow stop flowing” or “what if this data store becomes unavailable”  This is best done by the team as a whole but it is also a practice that an individual developer can use on their own portions of a system.

Penetration testing and resiliency testing are other practices that teams can implement to help test the Ruggedness of their applications and systems.

@wickett: What open source tools do you think should fit into the tool chain for a rugged developer?  What about for a rugged operations/sysadmin?

@danielcornell: Static analysis tools like FindBugs, PMD and FxCop can be really valuable.  For dynamic testing a web proxy like WebScarab can be really useful to testing how software is going to respond to unexpected inputs.

On the operations/sysadmin thing a critical Rugged practice is to proactively monitor systems in order to anticipate problems.  From the standpoint, you should be instrumenting systems and not just tracking failures.  When an HTTP server goes down it is often too late to prevent an outage and that isn’t very Rugged.  Instead you also need to be tracking disk and processor usage, traffic and response time trends and so on in order to be able to anticipate problems.  Fortunately there are a lot of open source tools available for DIY folks as well as SaaS offerings that can make this sort of instrumentation less of a hill to climb that it was in the past.

@wickett: We appreciate having time with Dan and hearing from an industry leader on Rugged Software.  To keep up Dan on the interweb, you can follow him on twitter or check out his blog.

Leave a comment

Filed under Security