Tag Archives: DevOps

by | September 8, 2011 · 11:29 am

Analysts on DevOps

DevOps is getting enough traction that there are papers coming out on if from the various analyst groups. I thought I’d spur a roundup of this research – they can be very valuable in converting your upper management types into understanding and seeing the value of DevOps.

Chime in below with more to add to the list!  Analyst stuff, not random blogs, please – something that you would put in front of upper management.

Leave a comment

Filed under DevOps

Tagged as ,

by | September 2, 2011 · 12:12 pm

Addressing the IT Skeptic’s View on DevOps

A recent blog post on DevOps by the IT Skeptic entitled DevOps and traditional ITSM – why DevOps won’t change the world anytime soon got the community a’frothing. And sure, the article is a little simmered in anti-agile hate speech (apparently the Agilistias and cloud hypesters and cowboys are behind the whole DevOps thing and are leering at his wife and daughter and dropping his property values to boot) but I believe his critiques are in general very perceptive and that they are areas we, the DevOps movement, should work on.

Go read the article – it’s really long so I won’t sum the whole thing up here.

Here’s the most germane critiques and what we need to do about them. He also has some poor and irrelevant or misguided critiques, but why would I waste time on those?  Let’s take and action on the good stuff that can make DevOps better!

Lack of a coherent definition

This is a very good point. I went to the first meeting of an Austin DevOps SIG lately and was treated to the usual debate about “the definition of DevOps” and all the varied viewpoints going into that.  We need to emerge more of a structured definition that either includes and organizes or excludes the various memetic threads. It’s been done with Agile, and we can do it too. My imperfect definition of DevOps on this site tries to clarify this by showing there are different levels (principles, methods, and practices) that different thoughts about DevOps slot into.

Worry about cowboys

This is a valid concern, and one I share. Here at NI, back in the day programmers had production passwords, and they got taken away for real good reasons.  “Oh, let’s just give the programmers pagers and the root password” is not a responsible interpretation of DevOps but it’s one I’ve heard bandied about; it’s based on a false belief that as long as you have “really smart” developers they’ll never jack everything up.

Real DevOps shops that are uptaking practices that could be risky, like continuous deployment, are doing it with extreme levels of safeguard put into place (automated testing, etc.).  This is similar to the overall problem in agile – some people say “agile? Great!  I’ll code at random,” whereas really you need to have a very high percentage of unit test coverage. And sure, when you confront people with this they say “Oh, sure, you need that” but there is very little constructive discussion or tooling around it. How exactly do I build a good systems + app code integration/smoke test rig? “Uh you could write a bunch of code hooked to Hudson…” This should be one of the most discussed and best understood parts of the chain, not one of the least, to do DevOps responsibly.

We’re writing our own framework for this right now – James is doing it in Ruby, it’s called Sparta, and devs (and system folks) provide test chunks that the framework runs and times in an automated fashion. It’s not a well solved problem (and the big-dollar products that claim to do test automation are nightmares and not really automated in the “devs easily contribute tests to integrate into a continuous deploy” sense.

Team size

Working at a large corporation, I also share his concern about people’s cunning DevOps schemes that don’t scale past a 12 person company.  “We’ll just hire 7 of the best and brightest and they’ll do everything, and be all crossfunctional, and write code and test and do systems and ops and write UIs and everything!” is only a legit plan for about 10 little hot VC funded Web 2.0 companies out there.  The rest of us have to scale, and doing things right means some specialization and risks siloization.

For example, performance testing.  When we had all our developers do their own performance testing, the limit of the sophistication of those tests was “I’ll run 1000 hits against it and time how long it takes to finish.  There, 4 seconds.  Done, that’s my performance testing!”  The only people who think Ops, QA, etc. are such minor skill sets that someone can just do them all is someone who is frankly ignorant of those fields. Oh, P.S. The NoOps guys fall into this category, please don’t link them to DevOps.

We have struggled with this.  We’ve had to work out what testing our devs do versus how we closely align with external test teams.  Same with security, performace, etc.  The answer is not to completely generalize or completely silo – Yahoo! had a great model with their performance team, where their is a central team of super-experts but there are also embedded folks on each product team.

Hiring people

Very related to the previous point – again unless you’re one of the 10 hottest Web 2.0 plays and you can really get the best of the best, you are needing to staff your organization with random folks who graduated from UT with a B average. You have to have and manage tiers as well as silos – some folks are only ready to be “level 1 support” and aren’t going to be reading some dev’s Java code.

Traditional organizations and those following ITIL very closely can definitely create structures that promote bad silos and bad tiering. But just assuming everyone will be of the same (high) skill level and be able to know everything is a fallacy that is easy to fall into, since it’s those sort of elite individuals who are the leading uptakers of DevOps.  Maybe Gene Kim’s book he’s working on (“Visible DevOps” or similar) will help with that.

Tools fixation

Definitely an issue.  An enhanced focus on automation is valuable.  Too many ops shops still just do the same crap by hand day after day, and should be challenged to automate and use tools.  But a lot of the DevOps discussions do become “cool tool litanies” and that’s cart before the horse.  In my terminology, you don’t want to drive the principles out of the practices and methods – tooling is great but it should serve the goals.

We had that problem on our team. I had to talk to our Ops team and say “Hey, why are we doing all these tool implementations?  What overall goal are they serving? ”  Tools for the sake of tools are worse than pointless.

Process

It is true that with agile and with DevOps that some folks are using it as an excuse to toss out process.  It should simply be a different kind of process! And you need to take into account all the stuff that should be in there.

A great example is Michael Howard et al. at Microsoft with their Security Development Lifecycle.  The first version of it was waterfall.  But now they’ve revamped it to have an agile security development lifecycle, so you know when to do your threat modeling etc.

Build instead of buy

Well, there are definitely some open source zealots involved with most movements that have any sysadmins involved. We would like to buy instead of build, but the existing tools tend to either not solve today’s problems or have poor ROI.

In IT, we implemented some “ITIL compliant” HP tools for problem tracking, service desk, and software deployment. They suck, and are very rigid, and cost a lot of money, and required as much if not more implementation time than writing something from scratch that actually addressed our specific requirements. And in general that’s been everyone’s experience. The Ops world has learned to fear the HP/IBM/CA/etc systems management suites because it’s just one of those niches that is expensive and bad (like medical or legal software).

But having said that, we buy when we can! Splunk gave us a lot more than cobbling together our own open source thing.  Cloudkick did too. Sure, we tend to buy SaaS a lot more than on prem software now because of the velocity that gives us, but I agree that you need to analyze the hidden costs of building as part of a build/buy – you just need to also see the hidden costs and compromised benefits of a buy.

Risk Control

This simply goes back to the cowboy concern. It’s clearly shown that if you structure your process correctly, with the right testing and signoff gates, then agile/devops/rapid deploys are less risky.

We came to this conclusion independently as well.  In IT, we ran (still do) these Web go lives once a month.  Our Web site consists of 200+ applications  and we have 70 or so programmers, 7 Web ops, a whole Infrastructure department, a host of third party stuff (Oracle and many more)… Every release plan was 100 lines long and the process of planning them and executing on them was horrific. The system gets complex enough, both technically and organizationally, that rollbacks + dependencies + whatnot simply turn into paralysis, and you have to roll stuff out to make money.  When the IT apps director suggested “This is too painful – we should just do these quarterly instead, and tell the business they get to wait 2 more months to make their money,” the light went on in my mind. Slower and more rigorous is actually worse.  It’s not more efficient to put all the product you’re shipping for the month onto a huge ass warehouse on the back of a giant truck and drive it around doing deliveries, either; this should be obvious in retrospect. Distribution is a form of risk management. “All the eggs in one big basket that we’ll do all at one time” is the antithesis of that.

The Future

We started DevOps here at NI from the operations guys.  We’d been struggling for years to get the programmers to take production responsibility for their apps. We had struggled to get them access to their own logs, do their own deploys (to dev and test), let business users input Apache redirects into a Web UI rather than have us do it… We developed a whole process, the Systems Development Framework, that we used to engage with dev teams and make sure all the performance, reliability, security, manageability, provisioning, etc. stuff was getting taken care of… But it just wasn’t as successful as we felt like it could be.  Realizing that a more integrated model was possible, we realized success was actually an option. Ask most sysadmin shows if they think success is actually a possible outcome of their work, and you’ll get a lot of hedging kinds of “well success is not getting ruined today” kinds of responses.

By combining ops and devs onto one team, by embedding ops expertise onto other dev teams, by moving to using the same tools and tracking systems between devs and ops, and striving for profound automation and self service, we’ve achieved a super high level of throughput within a large organization. We have challenges (mostly when management decides to totally change track on a product, sigh) but from having done it both ways – OMG it’s a lot better. Everything has challenges and risks and there definitely needs to to be some “big boy” compatible thinking  on DevOps – but it’s like anything else, those who adopt early will reap the rewards and get competitive advantage on the others. And that’s why we’re all in. We can wait till it’s all worked out and drool-proof, but that’s a better fit for companies that don’t actually have to produce/achieve any more (government orgs, people with more money than God like oil and insurance…).

1 Comment

Filed under DevOps

Tagged as , ,

by | August 4, 2011 · 5:16 pm

Report from NIWeek

Hey all, sorry it’s been quiet around here – Peco and I took our families on vacation to Bulgaria!  Plus, we’ve been busy in the run-up to our company convention, NIWeek. I imagine most of the Web type folks out there don’t know about NIWeek, but it’s where scientists and engineers who use our products come to learn. It’s always awesome to see the technology innovation going on out there, from the Stormchasers getting data on tornadoes and lightning that no one ever has before, to high school kids solving real problems.

There were a couple things that are really worth checking out.  The first is the demo David Fuller did of NI’s system designer prototype (you can skip ahead to 5:00 in if you want to) . Though the examples he is using is of engineering type systems, you can easily imagine using that same interface for designing Web systems – no ‘separate Visio diagram’ BS any more. Imagine every level from architectural diagram to physical system representation to the real running code all being part of one integrated drill-down. It looks SUPER SWEET. Seems like science fiction to those of us IT-types.

A quick guide to the demo – so first a Xilinx guy talks about their new ARM-based chip, and then David shows drill-up and down to the real hardware parts of a system.  NI now has the “traditional systems” problem in that people buy hardware, buy software, and are turning it into large distributed scalable architectures.  Not being hobbled by preconceptions of how that should be done, our system diagram team has come up with a sweet visualization where you can swap between architecture view (8:30 in), actual pictures and specs of hardware, then down (10:40 in) into the “implementation” box-and-line system and network diagram, and then down into the code (12:00 in for VHDL and 13:20 in for LabVIEW). LabVIEW code is natively graphical, so in the final drilldown he also shows programming using drawing/gestures.

Why have twenty years of “systems management” and design tools from IBM/HP/etc not given us anything near this awesome for other systems?  I don’t know, but it’s high time. We led a session at DevOpsDays about diagramming systems, and “I make a Visio on the side” is state of the art.  There was one guy who took the time to make awesome UML models, but real integration of design/diagram to real system doesn’t exist. And it needs to. And not in some labor intensive  “How about UML oh Lord I pooped myself” kind of way, but an easy and integral part of building the system.

I am really enjoying working in the joint engineering/IT world.  There’s some things IT technology has figured out that engineering technology is just starting to bumble into (security, for example, and Web services). But there are a lot of things that engineering does that IT efforts look like the work of a bumbling child next to. Like instrumentation and monitoring, the IT state of the art is vomitous when placed next to real engineering data metric gathering (and analysis, and visualization) techniques.  Will system design also be revolutionized from that quarter?

The other cool takeaway was how cloud is gaining some foothold in the engineering space.  I was impressed as hell with Maintainable Software, the only proper Web 3.0 company in attendance. Awesome SaaS product, and I talked with the guys for a long time and they are doing all the cool DevOps stuff – automated provisioning, continuous deployment, feature knobs, all that Etsy/Facebook kind of whizbang shit. They’re like what I want our team here to become, and it was great meeting someone in our space who is doing all that – I love goofy social media apps or whatever but it can sometimes be hard to convey the appropriateness of some of those practices to our sector. “If it’s good enough to sell hand knitted tea cozies or try to hook up with old high school sweethearts, then certainly it’s good enough to use in your attempt to cure cancer!” anyway, Mike and Derek were great guys and it was nice to see that new kind of thinking making inroads into our sometimes too-traditional space.

Leave a comment

Filed under Conferences, General

Tagged as , , , , , , ,

by | July 27, 2011 · 2:25 pm

DevOps at OSCON

I was watching the live stream from OSCON Ignite and saw a good presentation on DevOps Antipatterns by Aaron Blew.  Here’s the video link, it’s 29:15 in, go check it out.  And I was pleasantly surprised to see that he mentions us here at the agile admin in his links!  Hi Aaron and OSCONners!

1 Comment

Filed under DevOps

Tagged as , ,

by | April 13, 2011 · 8:34 am

Our Cloud Products And How We Did It

Hey, I’m not a sales guy, and none of us spend a lot of time on this blog pimping our company’s products, but we’re pretty proud of our work on them and I figured I’d toss them out there as use cases of what an enterprise can do in terms of cloud products if they get their act together!

Some background.  Currently all the agile admins (myself, Peco, and James) work together in R&D at National Instruments.  It’s funny, we used to work together on the Web Systems team that ran the ni.com Web site, but then people went their own ways to different teams or even different companies. Then we decided to put the dream team back together to run our new SaaS products.

About NI

Some background.  National Instruments (hereafter, NI) is a 5000+ person global company that makes hardware and software for test & measurement, industrial control, and graphical system design. Real Poindextery engineering stuff. Wireless sensors and data acquisition, embedded and real-time, simulation and modeling. Our stuff is used to program the Lego Mindstorms NXT robots as well as control CERN’s Large Hadron Collider. When a crazed highlander whacks a test dummy on Deadliest Warrior and Max the techie looks at readouts of the forces generated, we are there.

About LabVIEW

Our main software product is LabVIEW.  Despite being an electrical engineer by degree, we never used LabVIEW in school (this was a very long time ago, I’ll note, most programs use it nowadays), so it wasn’t till I joined NI I saw it in action. It’s a graphical dataflow programming language. I assumed that was BS when I heard it. I had so many companies try to sell be “graphical” programming over the years, like all those crappy 4GLs back in the ‘9o’s, that I figured that was just an unachieved myth. But no, it’s a real visual programming language that’s worked like a champ for more than 20 years. In certain ways it’s very bad ass, it does parallelism for you and can be compiled and dropped onto a FPGA. It’s remained niche-ey and hasn’t been widely adopted outside the engineering world, however, due to company focus more than anything else.

Anyway, we decided it was high time we started leveraging cloud technologies in our products, so we created a DevOps team here in NI’s LabVIEW R&D department with a bunch of people that know what they’re doing, and started cranking on some SaaS products for our customers! We’ve delivered two and have announced a third that’s in progress.

Cloud Product #1: LabVIEW Web UI Builder

First out of the gate – LabVIEW Web UI Builder. It went 1.0 late last year. Go try it for free! It’s a Silverlight-based RIA “light” version of LabVIEW – you can visually program, interface with hardware and/or Web services. As internal demos we even had people write things like “Duck Hunt” and “Frogger” in it – it’s like Flash programming but way less of a pain in the ass. You can run in browser or out of browser and save your apps to the cloud or to your local box. It’s a “freemium” model – totally free to code and run your apps, but you have to pay for a license to compile your apps for deployment somewhere else – and that somewhere else can be a Web server like Apache or IIS, or it can be an embedded hardware target like a sensor node. The RIA approach means the UI can be placed on a very low footprint target because it runs in the browser, it just has to get data/interface with the control API of whatever it’s on.

It’s pretty snazzy. If you are curious about “graphical programming” and think it is probably BS, give it a spin for a couple minutes and see what you can do without all that “typing.”

A different R&D team wrote the Silverlight code, we wrote the back end Web services, did the cloud infrastructure, ops support structure, authentication, security, etc. It runs on Amazon Web Services.

Cloud Product #2: LabVIEW FPGA Compile Cloud

This one’s still in beta, but it’s basically ready to roll. For non-engineers, a FPGA (field programmable gate array) is essentially a rewritable chip. You get the speed benefits of being on hardware – not as fast as an ASIC but way faster than running code on a general purpose computer – as well as being able to change the software later.

We have a version of LabVIEW, LabVIEW FPGA, used to target LabVIEW programs to an FPGA chip. Compilation of these programs can take a long time, usually a number of hours for complex designs. Furthermore the software required for the compilation is large and getting more diverse as there’s more and more chips out there (each pretty much has its own dedicated compiler).

So, cloud to the rescue. The FPGA Compile Cloud is a simple concept – when you hit ‘compile’ it just outsources the compile to a bunch of servers in the cloud instead of locking up your workstation for hours (assuming you’ve bought a subscription).  FPGA compilations have everything they need with them, there’s not unique compile environments to set up or anything, so it’s very commoditizable.

The back end for this isn’t as simple as the one for UI Builder, which is just cloud storage and load balanced compile servers – we had to implement custom scaling for the large and expensive compile workers, and it required more extensive monitoring, performance, and security work. It’s running on Amazon too. We got to reuse a large amount of the infrastructure we put in place for systems management and authentication for UI Builder.

Cloud Product #3: Technical Data Cloud

It’s still in development, but we’ve announced it so I get to talk about it! The idea behind the Technical Data Cloud is that more and more people need to collect sensor data, but they don’t want to fool with the management of it. They want to plop some sensors down and have the acquired data “go to the cloud!” for storage, visualization, and later analysis. There are other folks doing this already, like the very cool Pachube (pronounced “patch-bay”, there’s a LabVIEW library for talking to it), and it seems everyone wants to take their sensors to the cloud, so we’re looking at making one that’s industrial strength.

For this one we are pulling our our big guns, our data specialist team in Aachen, Germany. We are also being careful to develop it in an open way – the primary interface will be RESTful HTTP Web services, though LabVIEW APIs and hardware links will of course be a priority.

This one had a big technical twist for us – we’re implementing it on Microsoft Windows Azure, the MS guys’ cloud offering. Our org is doing a lot of .NET development and finding a lot of strategic alignment with Microsoft, so we thought we’d kick the tires on their cloud. I’m an old Linux/open source bigot and to be honest I didn’t expect it to make the grade, but once we got up to speed on it I found it was a pretty good bit of implementation. It did mean we had to do significant expansion of our underlying platform we are reusing for all these products – just supporting Linux and Windows instance in Amazon already made us toss a lot of insufficiently open solutions in the garbage bin, and these two cloud worlds are very different as well.

How We Did It

I find nothing more instructive than finding out the details – organizational, technical, etc. – of how people really implement solutions in their own shops.  So in the interests of openness and helping out others, I’m going to do a series on how we did it!  I figure it’ll be in about three parts, most likely:

  • How We Did It: People
  • How We Did It: Process
  • How We Did It: Tools and Technologies

If there’s something you want to hear about when I cover these areas, just ask in the comments!  I can’t share everything, especially for unreleased products, but promise to be as open as I can without someone from Legal coming down here and Tasering me.

5 Comments

Filed under Cloud, DevOps

Tagged as , , , , , , ,

by | April 11, 2011 · 12:10 pm

Security and the Rise (and Fall?) of DevOps

As I’ve been involved with DevOps and its approach of blending development and operations staff together to create better products, I’ve started to see similar trends develop in the security space. I think there’s some informative parallels where both can learn from each other and perhaps avoid some pitfalls.

Here’s a recent article entitled “Agile: Most security guys are useless” that states the problem succinctly. In successful and agile orgs, the predominant mindset is that if you’re not touching the product, you are semi-useless overhead. And there’s some truth to that. When people are segregated into other “service” orgs – like operations or security – the us vs. them mindset predominates and strangles innovation in its crib.

The main initial drive of agile was to break down that wall between the devs and the “business”, but walls remain that need similar breaking down. With DevOps, operations organizations faced with this same problem are innovating new approaches; a collaborative approach with developers and operations staff working together on the product as part of the same team. It’s working great for those who are trying it, from the big Web shops like Facebook to the enterprise guys like us here at NI. The movement is gathering steam and it seems clear to those of us doing it this way that it’s going to be a successful and disruptive pattern for adopters.

But let’s not pat ourselves on the back too much just yet. We still have a lot of opportunity to screw it up. Let’s review an example from another area.

In the security world, there is a whole organization, OWASP (the Open Web Application Security Project) whose goal is to promote and enable application security. Security people and developers, working together!  Dev+Sec already exists! Or so the plan was.

However, recently there have been some “shots across the bow” in the OWASP community.  Read Security People vs Developers and especially OWASP: Has It Reached A Tipping Point? The latter is by Mark Curphey, who started OWASP. He basically says OWASP is becoming irrelevant because it’s leaving developers behind. It’s becoming about “security professionals” selling tools and there’s few developers to be found in the community any more.

And this is absolutely true.  We host the Austin OWASP chapter here at NI’s Austin campus, and two of the officers are NI employees. We make sure and invite NI developers to come to OWASP. Few do, at least not after the first couple times.  I asked some of the devs on our team why not, and here’s some answers I got.

  • I want to leave sessions by saying, “I need to think about this the next time I code”. I leave sessions by saying, “that was cool, I can talk about this at a happy hour”. If I could do the former, I’d probably attend most/all the sessions.
  • A lot of the sessions don’t seem business focused and it is hard to relate. Demos are nicer; but a lot of times they are so specific (for example: specific OS + specific Java Version + specific javascript library = hackable) that it’s not actionable.
  • “Security people” think “developers” don’t know what they are doing and don’t care about security. Which to developers is offensive. We like to write secure applications; sometimes we just find the bugs too late….
  • I’ve gone to, I think, 4 OWASP meetings.  Of those, I probably would only have recommended one of them to others – Michael Howard’s.  I think it helped that he was a well-known speaker and seemed to have a developer focus. So, well-known speakers, with a compelling and relevant subject..  Even then, the time has to be weighed against other priorities.  For example, today’s meeting sounds interesting, but not particularly relevant.  I’ll probably skip it.

In the end, the content at these meetings is more for security pros like pen testers, or for tool buyers in security or sysadmin groups. “How do I code more securely” is the alleged point of the group but frankly 90% of the activity is around scanners and crackers and all kinds of stuff that is fine but should be simple testing steps after the code’s written securely in the first place.

As a result there have been interesting ideas coming from the security community that are reminiscent of DevOps concepts. Pen tester @atdre did a talk here to the Austin OWASP chapter about how security testers engaging with agile teams “from the outside” are failing, and shouldn’t we instead embed them on the team as their “security buddy.” (I love that term.  Security buddy. I hate my “compliance auditor” without even meeting the poor bastard, but I like my security buddy already.) At the OWASP convention LASCON, Matt Tesauro delivered a great keynote similarly trying to refocus the group back on the core problem of developing secure software; in fact, they’re co-sponsoring a movement called “Rugged” that has a manifesto similar to the Agile Manifesto but is focused on security, availability, reliability, et cetera. (As a result it’s of interest to us sysadmin types, who are often saddled with somehow applying those attributes in production to someone else’s code…)

The DevOps community is already running the risk of “leaving the devs behind” too.  I love all my buddies at Opscode and DTO and Puppet Labs and Thoughtworks and all. But a lot of DevOps discussions have started to be completely sysadmin focused as well; a litany of tools you can use for provisioning or monitoring or CI. And that wouldn’t be so bad if there was a real entry point for developers – “Here’s how you as a developer interact with chef to deploy your code,” “Here’s how you make your code monitorable”. But those are often fringe discussions around the core content which often mainly warms the cockles of a UNIX sysadmin’s heart. Why do any of my devs want to see a presentation on how to install Puppet?  Well, that’s what they got at a recent Austin Cloud User Group meeting.

As a result, my devs have stopped coming to DevOps events.  When I ask them why, I get answers similar to the ones above for why they’re not attending OWASP events any more. They’re just not hearing anything that is actionable from the developer point of view. It’s not worth the two hours of their valuable time to come to something that’s not at all targeted at them.

And that’s eventually going to scuttle DevOps if we let it happen, just as it’ll scuttle OWASP if it continues there. The core value of agile is PEOPLE over processes and tools, COLLABORATION over negotiation. If you are leaving the collaboration behind and just focusing on tools, you will eventually fail, just in a more spectacular and automated fashion.

The focus at DevOpsDays US 2010 was great, it was all about culture, nothing about tools. But that culture talk hasn’t driven down to anything more actionable, so tools are just rising up to fill the gap.

In my talk at that DevOpsDays I likened these new tools and techniques to the introduction of the Minie ball to rifles during the Civil War. In that war, they adopted new tools and then retained their same old tactics, walking up close in lines designed for weapons with much shorter ranges and much lower accuracy – and the slaughter was profound.

All our new DevOps tools are great, but in the same way, if we don’t adapt our way of thinking to them, they will make our lives worse, not better, for all their vaunted efficiency. You can do the wrong thing en masse and more quickly. The slaughter will similarly be profound.

A sysadmin suddenly deciding to code his own tools isn’t really the heart of DevOps.  It’s fine and good, and I like seeing more tools created by domain experts. But the heart of DevOps, where you will really see the benefits in hard ROI, is developers and operations folks collaborating on real end-consumer products.

If you are doing anything DevOpsey, please think about “Why would a developer care about this?” How is it actionable to them, how does it make their lives easier?  I’m a sysadmin primarily, so I love stuff that makes my job easier, but I’ve learned over the years that when I can get our devs to leverage something, that’s when it really takes off and gives value.

The same thing applies to the people on the security side.  Why do we have this huge set of tools and techniques, of OWASP Top 10s and Live CDs and Metasploits and about a thousand wonderful little gadgets, but code is pretty much as shitty and insecure as it was 20 years ago? Because all those things try to solve the problem from the outside, instead of targeting the core of the matter, which is developers developing secure code in the first place.  And to do that, it’s more of a hearts-and-minds problem than a tools-and-processes problem.

That’s a core realization that operations folks, and security folks, and testing folks, and probably a bunch of other folks need to realize, deeply internalize, and let it change the way they look at the world and how they conduct their work.

5 Comments

Filed under DevOps, Security

Tagged as , , , , , ,

by | March 16, 2011 · 11:17 am

DevOps In Action Podcast

While at SXSW, I recorded an episode of the IT Management and Cloud Podcast with Michael Coté and John Willis. You can find it on Coté’s People Over Process blog! Sorry about the background noise, we were doing it in the upstairs bar at the Driskill.

Leave a comment

Filed under Cloud, Conferences, DevOps

Tagged as , ,

by | March 14, 2011 · 10:08 am

SXSW Interactive 2011 Day Two

Day 2 started off a bit rocky but then got really good.  I tried to get to the Lean Startup: King of the Apps Showdown session, but the shuttle buses take a long route and are few and far between, so I got there pretty late.  I saw snapappointments.com, snapwebsites, Planzai, Icebreakr,and mapomatic all get their app and pitch critiqued by a panel of folks – Robert Scoble, Eric Ries, Dave McClure, Bill Boebel, and Stacey Higginbotham. They demanded mapomatic release right away, and told snapwebsites to remove half of his features.  I was ambivalent about this advice – I can see streamlining the UI to hide advanced functionality, but “remove functionality”? Sure, a simple app has a wider reach, but if you can effectively provide an advanced mode/version with more functionality that professionals need, wouldn’t that be better? I just don’t like how all these apps are degenerating into one very narrow function “plus it checks you into Foursquare!!!!”  Bah.

They joked about how many of the candidates had “Snap” in their name, and then on the bus on the way back I met a girl from TeamSnap. They have a site that lets you arrange sports teams, even take Paypal payments for team dues, very slick.

And congratulations to friend of the blog Lenny Rachitsky (@lennysan) of Localmind, who has been singled out by Scoble as “best SXSW app so far!” You should all go download it from the App Store now.

Then I tried to go to the session on How To Innovate At Big Companies, with Gene Kim of Visible Ops fame – I left the AT&T center early and went all the way to the Hyatt. But it was so full they had 30 people standing outside only allowed in if anyone left. So I went over to the Hilton to try to catch Mistakes I Made Building Netflix for the iPhone. And it also had a line of “one in, one out” standby people. Son of a bitch.  They better be videoing this stuff. Anyway, at this point it was too far into the slot so I just went to the Screenburn Arcade, which was fun. Got my picture taken with Loren Wiseman of Traveller fame – apparently the venerable RPG is being turned into an iPhone MMO!

To recuperate from our disappointing morning, Peco and I went and had a burger at Casino El Camino, which takes a ridiculously long time but is worth it, they have the best burgers in Austin. We hooked up with Cote and John Willis there, and then went to the Etsy Code as Craft: Moving Fast At Scale event which was brilliant and DevOpsey. Continuous deployment, only coding off trunk, logging and metrics, “dashboard driven development”, and more. A lot of the topics have info on them on the Etsy Code as Craft blog – they didn’t record the talk but promise to record it the next time they give it and put it up.

I took a lot of notes but the high points were:

  • They have open sourced their log collection/graphing tool, Logster, and their stats collection daemon, statsd.
  • They perform hundreds of releases a month but only had six “bad deploys” over the course of a year by combining one-button deploy, testing, feature flags, and a  theory of roll-forward. No more organized releases, release managers, rollbacks, all that stuff.
  • They have a cute “deployinator” dashboard that lets people (and not just coders, they have more people deploying than they have developers) do the deploys – they’ll open source it, but stressed that it’s not much technically, the culture and practices are 99% of the work to get to this.
  • They use a lot of chef but don’t use it for code deploys because that’s a simple task that they need more control over.

That was a two hour event, so then after a little VIP partyin’ atop Fogo de Chao, we packed it in for the day.

Leave a comment

Filed under Conferences

Tagged as , , ,

by | March 12, 2011 · 8:50 pm

DevOps at CloudCamp at SXSWi!

Isn’t that some 3l33t jargon.  Anyway, Dave Nielsen is holding a CloudCamp here in Austin during SXSW Interactive on Monday, March 14 (followed by the Cloudy Awards on March 15) and John Willis of Opscode and I reached out to him, and he has generously offered to let us have a DevOps meetup in conjunction.

John’s putting together the details but is also traveling, so this is going to be kind of emergent.  If you’re in Austin (normally or just for SXSW) and are interested in cloud, DevOps, etc., come on out to the CloudCamp (it’s free and no SXSW badge required) and also participate in the DevOps meetup!

In fact, if anyone’s interested in doing short presentations or show-and-tells or whatnot, please ping John (@botchagalupe) or me (@ernestmueller)!

Leave a comment

Filed under Cloud, Conferences, DevOps

Tagged as , , , ,

by | March 10, 2011 · 11:46 am

SXSW Interactive Is Here!

SXSW Interactive is going on in Austin tomorrow through next Tuesday, and loads of great cloud and DevOps folks will be in town for it. Looking forward to talking with @cote, @lennysan, @botchgalupe, @davenielsen, @ehuddleston, and many more.

Here’s what I think the best cloud/devops/high tech related tickets are, let me know what I’m missing! A lot of the off premise events don’t even require badges and are mostly free.

Random Off Premise SXSW Interactive Stuff

Other events – not job related but make me happy:

Sessions

Sessions are less important than the other stuff so they’re on here second!  No time for links, search on ’em.

What’s the good stuff I haven’t mentioned?  DevOps, Cloud, noSQL, and other cool stuff report below!

Leave a comment

Filed under Conferences

Tagged as , , ,