Tag Archives: rugged

by | May 7, 2026 · 9:13 pm

AI and DevOps Maturity

I just got back from DevOpsDays Austin, and it’s always great to go out and talk with a wide array of folks from the tech community to see what’s up.

The takeaways were interesting. Austin is an advanced tech hub, but hasn’t been bitten by the AI bug as hard as SFO. On the one hand that worries me that our community might be getting behind, but there’s some justified bubble avoidance too.

Takeaway One: AI adoption is on a curve like everything else. It is a sharp curve, but out in real businesses there’s still a sharp divide between AI’ers and non-AIers. Will Longenecker from elite tech recruiting firm New Iron shared some data from their work and it provides a much more moderate view of AI and the workforce. There’s some shops pro, some shops anti, some shops going nuts with tokenmaxxing, some shops containing costs, and so on.

from Will Longenecker of New Iron

Takeaway Two: Attitudes to AI are maturing. It’s been a rough road with AI in that there’s a lot of real awesome tech there, but the oligarch set has been blowing it into a bunch of ridiculous claims, so it’s a giant spike that’s 50% real value but 50% bubble. I found that the group was in general very AI forward and most people were using it in some way, but that there’s a real appreciation for the challenges and ROI evaluation and not just bandwagon hype.

Will did a flash survey of the group, and the interesting takeaway is that over 50% of the respondents felt security was the #1 concern with using AI agents. Other speakers and groups of practitioners at the conference talked about cost, quality, and so on.

But they weren’t just talking about it in terms of “it means it’s bad,” these folks are already trying to develop practices and tooling to mitigate it.

From a DevOps perspective, the basic description of what’s been going on in the industry it that for a long time, the technology value chain had one big, long, expensive link – software development – and a bunch of smaller links like product, testing, operations, and so on. It’s been 100 devs to a handful of SREs, security pros, and so on. Which worked when it took a long time to produce software. So companies hired up truckfuls of devs, and paid them whatever they wanted, because that’s what made the value flow.

Now producing software is much faster and less expensive, and more can be done by Product and those other links in the chain. So you see developer layoffs. But even if you lay off 20% of your devs, they are all suddenly generating code at 10x the rate. The ratio in the chain of security, quality, operations, and so on was not great in the first place but now it’s completely untenable. This is forcing an evening out of that value chain, both with more tools but also the staffing and focus has to step up. AI is not generating secure, quality, performant software – the models are trained on all the crappy software out there in the wild!

So the talks and startup tools were on how do you test more effectively,

from Daniel Ward of Lean TECHniques

How do you secure more effectively.

from James Wickett of Dry Run Security

operate more effectively.

from Peco Karayanev of Autoptic

Build more effectively,

from Sam Alba of Mendral

Manage your cost effectively.

from Ian Littman

And so on. Basically the good old rugged software model.

from Wickett again

The “tokenmaxxing” and “it must be alive” hype cycle is running out and companies and tech practitioners are buckling down to the realities of “OK, so this will get me a 20% overall advantage. Great!” 10x speed is 10x slop speed and is expensive once the token subsidies end and puts your company at risk. But, smart orgs will use the resources freed up by the “faster typing” part of the benefit to find much better solutions than they’ve had so far in these areas – to be honest the hard part has always been making software enterprise ready and many orgs have suffered calamity from not doing it well. With software now pouring out of every open faucet, leveling up in these disciplines is the new path to success.

That’s not a surprise, it’s the same “Operations is the new secret sauce” message from the Web growth era, but is forcing advances in thesse other fields as well – security, ops, etc. had to adapt to keep pace in Web 2.0 but they are still slower and more manual than is needed in this new world.

I think of it like the transition from servers to virtual servers to cloud servers to containers to kubernetes – it pushed the entire industry from manual config and runbooks to configuration management to declarative infrastructure to pervasive orchestration. AI is that same forcing function – it’s not eliminating net tech jobs or anything like that, it’s just moving them around in an inevitable way. And sure, half the silly chatbots added to every product will eventually fade away as adding no value, but AI as a ubiquitous system component is here to stay.

Leave a comment

Filed under AI, Conferences

Tagged as , , , ,

by | April 11, 2011 · 12:10 pm

Security and the Rise (and Fall?) of DevOps

As I’ve been involved with DevOps and its approach of blending development and operations staff together to create better products, I’ve started to see similar trends develop in the security space. I think there’s some informative parallels where both can learn from each other and perhaps avoid some pitfalls.

Here’s a recent article entitled “Agile: Most security guys are useless” that states the problem succinctly. In successful and agile orgs, the predominant mindset is that if you’re not touching the product, you are semi-useless overhead. And there’s some truth to that. When people are segregated into other “service” orgs – like operations or security – the us vs. them mindset predominates and strangles innovation in its crib.

The main initial drive of agile was to break down that wall between the devs and the “business”, but walls remain that need similar breaking down. With DevOps, operations organizations faced with this same problem are innovating new approaches; a collaborative approach with developers and operations staff working together on the product as part of the same team. It’s working great for those who are trying it, from the big Web shops like Facebook to the enterprise guys like us here at NI. The movement is gathering steam and it seems clear to those of us doing it this way that it’s going to be a successful and disruptive pattern for adopters.

But let’s not pat ourselves on the back too much just yet. We still have a lot of opportunity to screw it up. Let’s review an example from another area.

In the security world, there is a whole organization, OWASP (the Open Web Application Security Project) whose goal is to promote and enable application security. Security people and developers, working together!  Dev+Sec already exists! Or so the plan was.

However, recently there have been some “shots across the bow” in the OWASP community.  Read Security People vs Developers and especially OWASP: Has It Reached A Tipping Point? The latter is by Mark Curphey, who started OWASP. He basically says OWASP is becoming irrelevant because it’s leaving developers behind. It’s becoming about “security professionals” selling tools and there’s few developers to be found in the community any more.

And this is absolutely true.  We host the Austin OWASP chapter here at NI’s Austin campus, and two of the officers are NI employees. We make sure and invite NI developers to come to OWASP. Few do, at least not after the first couple times.  I asked some of the devs on our team why not, and here’s some answers I got.

  • I want to leave sessions by saying, “I need to think about this the next time I code”. I leave sessions by saying, “that was cool, I can talk about this at a happy hour”. If I could do the former, I’d probably attend most/all the sessions.
  • A lot of the sessions don’t seem business focused and it is hard to relate. Demos are nicer; but a lot of times they are so specific (for example: specific OS + specific Java Version + specific javascript library = hackable) that it’s not actionable.
  • “Security people” think “developers” don’t know what they are doing and don’t care about security. Which to developers is offensive. We like to write secure applications; sometimes we just find the bugs too late….
  • I’ve gone to, I think, 4 OWASP meetings.  Of those, I probably would only have recommended one of them to others – Michael Howard’s.  I think it helped that he was a well-known speaker and seemed to have a developer focus. So, well-known speakers, with a compelling and relevant subject..  Even then, the time has to be weighed against other priorities.  For example, today’s meeting sounds interesting, but not particularly relevant.  I’ll probably skip it.

In the end, the content at these meetings is more for security pros like pen testers, or for tool buyers in security or sysadmin groups. “How do I code more securely” is the alleged point of the group but frankly 90% of the activity is around scanners and crackers and all kinds of stuff that is fine but should be simple testing steps after the code’s written securely in the first place.

As a result there have been interesting ideas coming from the security community that are reminiscent of DevOps concepts. Pen tester @atdre did a talk here to the Austin OWASP chapter about how security testers engaging with agile teams “from the outside” are failing, and shouldn’t we instead embed them on the team as their “security buddy.” (I love that term.  Security buddy. I hate my “compliance auditor” without even meeting the poor bastard, but I like my security buddy already.) At the OWASP convention LASCON, Matt Tesauro delivered a great keynote similarly trying to refocus the group back on the core problem of developing secure software; in fact, they’re co-sponsoring a movement called “Rugged” that has a manifesto similar to the Agile Manifesto but is focused on security, availability, reliability, et cetera. (As a result it’s of interest to us sysadmin types, who are often saddled with somehow applying those attributes in production to someone else’s code…)

The DevOps community is already running the risk of “leaving the devs behind” too.  I love all my buddies at Opscode and DTO and Puppet Labs and Thoughtworks and all. But a lot of DevOps discussions have started to be completely sysadmin focused as well; a litany of tools you can use for provisioning or monitoring or CI. And that wouldn’t be so bad if there was a real entry point for developers – “Here’s how you as a developer interact with chef to deploy your code,” “Here’s how you make your code monitorable”. But those are often fringe discussions around the core content which often mainly warms the cockles of a UNIX sysadmin’s heart. Why do any of my devs want to see a presentation on how to install Puppet?  Well, that’s what they got at a recent Austin Cloud User Group meeting.

As a result, my devs have stopped coming to DevOps events.  When I ask them why, I get answers similar to the ones above for why they’re not attending OWASP events any more. They’re just not hearing anything that is actionable from the developer point of view. It’s not worth the two hours of their valuable time to come to something that’s not at all targeted at them.

And that’s eventually going to scuttle DevOps if we let it happen, just as it’ll scuttle OWASP if it continues there. The core value of agile is PEOPLE over processes and tools, COLLABORATION over negotiation. If you are leaving the collaboration behind and just focusing on tools, you will eventually fail, just in a more spectacular and automated fashion.

The focus at DevOpsDays US 2010 was great, it was all about culture, nothing about tools. But that culture talk hasn’t driven down to anything more actionable, so tools are just rising up to fill the gap.

In my talk at that DevOpsDays I likened these new tools and techniques to the introduction of the Minie ball to rifles during the Civil War. In that war, they adopted new tools and then retained their same old tactics, walking up close in lines designed for weapons with much shorter ranges and much lower accuracy – and the slaughter was profound.

All our new DevOps tools are great, but in the same way, if we don’t adapt our way of thinking to them, they will make our lives worse, not better, for all their vaunted efficiency. You can do the wrong thing en masse and more quickly. The slaughter will similarly be profound.

A sysadmin suddenly deciding to code his own tools isn’t really the heart of DevOps.  It’s fine and good, and I like seeing more tools created by domain experts. But the heart of DevOps, where you will really see the benefits in hard ROI, is developers and operations folks collaborating on real end-consumer products.

If you are doing anything DevOpsey, please think about “Why would a developer care about this?” How is it actionable to them, how does it make their lives easier?  I’m a sysadmin primarily, so I love stuff that makes my job easier, but I’ve learned over the years that when I can get our devs to leverage something, that’s when it really takes off and gives value.

The same thing applies to the people on the security side.  Why do we have this huge set of tools and techniques, of OWASP Top 10s and Live CDs and Metasploits and about a thousand wonderful little gadgets, but code is pretty much as shitty and insecure as it was 20 years ago? Because all those things try to solve the problem from the outside, instead of targeting the core of the matter, which is developers developing secure code in the first place.  And to do that, it’s more of a hearts-and-minds problem than a tools-and-processes problem.

That’s a core realization that operations folks, and security folks, and testing folks, and probably a bunch of other folks need to realize, deeply internalize, and let it change the way they look at the world and how they conduct their work.

5 Comments

Filed under DevOps, Security

Tagged as , , , , , ,

by | January 31, 2011 · 4:52 pm

Rugged Software Manifesto: An Interview With Dan Cornell

I had a chance to talk with Dan Cornell from the Open Web Application Security Project (OWASP) and the Denim Group.  Dan has over 13 years of experience in development and is one of the founders at the Denim Group, a company that does security consulting day in and day out for all types of clients. Dan wrote Sprajax, a security analyzer for AJAX applications, and is the chair of the OWASP Global Membership Committee.

While a lot of us work with security, it is usually in addition to other dev or ops responsibilities and isn’t our sole focus.  However, Dan Cornell and his company are in the weeds of app security daily.  It is a pleasure to have him on the agile admin blog and we are chatting with him on his thoughts on the Rugged Software Manifesto. We ran across this manifesto at the Lonestar Application Security Conference (LASCON) and heard it referred to by both OWASP board members and Department of Homeland Security types so we thought we should look it up, and it seemed interesting.  So here’s Dan’s thoughts on the subject.

@wickett: Hi Dan, welcome to the agile admin blog.  Our first question for you is, what do we mean by Rugged Software?

@danielcornell:  I think of software that is going to run as intended, regardless of the conditions it encounters when it is deployed.  A big part of this is security, but that isn’t the only thing.  You also have to take into account reliability, survivability and a variety of other ‘-ity’ properties.  The Internet, and networks in general, are dangerous places where you need to be conscious of both malicious actors as well as other, unintentional adverse conditions such as service outages, congestion, untrained users making mistakes and so on.  Rugged Software is software that is designed and built to deal with these conditions – foreseen or not.

@wickett: When we have talked about Rugged Software with developers, we get a fair amount of pushback and hear statements like, “Rugged Software isn’t really real and that it is just a PR play.”  Some developers and engineers we;ve talked to say that it lacks depth, clarity and process, but I wonder if these engineers have even seen the original Agile Manifesto.  In my mind, I look back at the original Agile Manifesto and I am equally under-whelmed by it as I imagine they are by the Rugged Software Manifesto.  Do you see any parallels between the original Agile Manifesto and the Rugged Software Manifesto?

@danielcornell: The two biggest challenges I see for the Rugged Software movement are to provide a business justification for being Rugged to management and to answer the “so what?” question for developers.  Dealing with the first will go a long way toward solving the second.

If you ask most executives and managers if they’d like their software to be Rugged they’ll likely give an answer somewhat along the lines of: “Sure I’d love to be ‘Rugged.’  And ‘Agile’ and ‘Pragmatic’ and whatever other buzzwords the industry has come up with lately.  All that stuff is great but what I really want is to be on time and on budget and I want more features than our competitors.”  Thinking about security and quality and performance and anything else comes later.

So this is an area where the Rugged movement needs to do a better job of explaining why Rugged makes good business sense and you run into the same challenges that information security folks have had making a business justification for security for years.  Allowing an organization to avoid pain when the auditors or compliance folks come around is of value – but it often isn’t enough.  Reducing downtime and disruption in the future because you have avoided a breach or outage is of value – but it often isn’t enough.  David Rice has done some great thinking in this area in his book Geekonomics but, as an industry, I think we are still looking for the universal truth to be revealed that will make everyone sit up and take notice and I think we’re going to be waiting for a long time.

That said, some organizations have made the decision that this is an area that merits focus and when executives and managers make Rugged (or security or whatever) a priority then it is much easier to get the troops to fall in line.  I’m reminded of a secure coding training class I ran recently for an ISV where the VP in charge of engineering stood up at the beginning of the class and said “This is a priority for us and the material in this class represents the base level of knowledge everyone is expected to have.”  Then he sat in the entire two day class and at the end told his people “I hope you paid attention because you are going to be held accountable for knowing and implementing this material.”  Wow!  This was one of the best classes I taught in a long time; everyone showed up on time, paid attention, asked questions and so on.  I’d love to think it is because I’m such a funny and engaging guy but what really made the difference is that the leadership demonstrated with actions that security is a priority.  A similar model is always going to be the most effective way to drive Rugged into an organization.  Bottom-up, grass roots efforts are valuable because they raise awareness and help to identify like-minded potential champions, but to drive adoption at scale management needs to feel that this is a priority.

Speaking of priorities, I think this is an area where the Agile Manifesto style can provide some guidance.  The Rugged Manifesto is a great call to arms, but it is very self-referential: “I am Rugged… and more importantly my code is Rugged”  And so on.  If you look at the Agile Manifesto it calls out trade offs: “Individuals and interactions over processes and tools,” for example.  It accepts that processes and tools have merit, but chooses to value individuals and interactions more.  Acknowledging that decisions must be made and providing guidance on why one is superior to another is a stronger basis for a conversation and this is going to be key.  Right now those trade-offs are presented as “Prefer code without vulnerabilities over code with vulnerabilities”  Duh!  Who wouldn’t want that?  But the real question is what are they going to have to give up to get it.

@wickett: Over the next five years, do you see the Rugged Software Manifesto becoming a standard?  Will we have Rugged development practices and be operating on a new paradigm for development or will it get lumped under Agile?

@danielcornell: Well the Rugged Software Manifesto is just that – a Manifesto.  It isn’t a standard.  And I don’t think we are ever going to see a standard for software development because “development” is too big a thing to be done in a standard way.  You can’t expect folks building banking applications for East Coast financials to follow the same process as a couple of folks in a garage trying to build the next Facebook.  What I hope is developed and adopted, however, are so-called “best practices” such as threat modeling, static analysis for quality and security, security code review, penetration testing and so on.

Also, Agile and Rugged are two separate movements.  Agile is about creating customer-centric software in a flexible and timely manner and Rugged is about making software that doesn’t break.   Their goals are not exclusive, but they’re also different.

@wickett: When thinking about Rugged Software, what are practical steps that individual developers can adopt into their current practices whether they use Agile, Waterfall, or other development methodology?

@danielcornell: Well first I think it is important to note that individual developers can have a positive impact on their own code, but this is going to be largely ineffective if the rest of the development team doesn’t share the same goals.  The “weakest link in the chain” analogy comes to mind.  That said…

Threat Modeling is a practice that can be adopted into any development methodology.  There are lots of resources out there that explain how to use it to improve the security of complicated systems and it can also be used for the greater purpose of Rugged software by asking questions such as “what if this data flow stop flowing” or “what if this data store becomes unavailable”  This is best done by the team as a whole but it is also a practice that an individual developer can use on their own portions of a system.

Penetration testing and resiliency testing are other practices that teams can implement to help test the Ruggedness of their applications and systems.

@wickett: What open source tools do you think should fit into the tool chain for a rugged developer?  What about for a rugged operations/sysadmin?

@danielcornell: Static analysis tools like FindBugs, PMD and FxCop can be really valuable.  For dynamic testing a web proxy like WebScarab can be really useful to testing how software is going to respond to unexpected inputs.

On the operations/sysadmin thing a critical Rugged practice is to proactively monitor systems in order to anticipate problems.  From the standpoint, you should be instrumenting systems and not just tracking failures.  When an HTTP server goes down it is often too late to prevent an outage and that isn’t very Rugged.  Instead you also need to be tracking disk and processor usage, traffic and response time trends and so on in order to be able to anticipate problems.  Fortunately there are a lot of open source tools available for DIY folks as well as SaaS offerings that can make this sort of instrumentation less of a hill to climb that it was in the past.

@wickett: We appreciate having time with Dan and hearing from an industry leader on Rugged Software.  To keep up Dan on the interweb, you can follow him on twitter or check out his blog.

Leave a comment

Filed under Security

Tagged as , , , ,