Monthly Archives: November 2010

Austin Cloud User Group Nov 17 Meeting Notes

This month’s ACUG meeting was cooler than usual – instead of having one speaker talk on a cloud-related topic, we had multiple group members do short presentation on what they’re actually doing in the cloud.  I love talks like that, it’s where you get real rubber meets the road takeaways.

I thought I’d share my notes on the presentations.  I’ll write up the one we did separately, but I got a lot out of these:

  1. OData to the Cloud, by Craig Vyal from Pervasive Software
  2. Moving your SaaS from Colo to Cloud, by Josh Arnold from Arnold Marziani (previously of PeopleAdmin)
  3. DevOps and the Cloud, by Chris Hilton from Thoughtworks
  4. Moving Software from On Premise to SaaS, by John Mikula from Pervasive Software
  5. The Programmable Infrastructure Environment, by Peco Karayanev and Ernest Mueller from National Instruments (see next post!)

My editorial comments are in italics.  Slides are linked into the headers where available.

OData to the Cloud

OData was started by Microsoft (“but don’t hold that against it”) under the Open Specification Promise.  Craig did an implementation of it at Pervasive.

It’s a RESTful protocol for CRUDdy GET/POST/DELETE of data.  Uses AtomPub-based feeds and returns XML or JSON.  You get the schema and the data in the result.

You can create an OData producer of a data source, consume OData from places that support it, and view it via stuff like iPhone/Android apps.

Current producers – Sharepoint, SQL Azure, Netflix, eBay, twitpic, Open Gov’t Data Initiative, Stack Overflow

Current consumers – PowerPivot in Excel, Sesame, Tableau.  Libraries for Java (OData4J), .NET 4.0/Silverlight 4, OData SDK for PHP

It is easier for “business user” to consume than SOAP or REST.  Craig used OData4J to create a producer for the Pervasive product.

Questions from the crowd:

Compression/caching?  Nothing built in.  Though normal HTTP level compression would work I’d think. It does “page” long lists of results and can send a section of n results at a time.

Auth? Your problem.  Some people use Oauth.  He wrote a custom glassfish basic HTTP auth portal.

Competition?  Gdata is kinda like this.

Seems to me it’s one part REST, one part “making you have a DTD for your XML”.  Which is good!  We’re very interested in OData for our data centric services coming up.

Moving your SaaS from Colo to Cloud

Josh Arnold was from PeopleAdmin, now he’s a tech recruiter, but can speak to what they did before he left.  PeopleAdmin was a Sungard type colo setup.  Had a “rotting” out of country DR site.

They were rewriting their stack from java/mssql to ruby/linux.

At the time they were spending $15k/mo on the colo (not including the cost of their HW).  Amazon estimated cost was 1/3 that but really they found out after moving it’s 1/2.  What was the surprise cost?  Lower than expected perf (disk io) forced more instances than physical boxes of equivalent “size.”

Flexible provisioning and autoscaling was great, the colo couldn’t scale fast enough.  How do you scale?

The cloud made having an out of country DR site easy, and not have it rot and get old.

Question: What did you lose in the move?  We were prepared for mental “control issues” so didn’t have those.  There’s definitely advanced functionality (e.g. with firewalls) and native hardware performance you lose, but that wasn’t much.

They evalled Rackspace and Amazon (cursory eval).  They had some F5s they wanted to use and the ability to mix in real hardware was tempting but they mainly went straight to Amazon.  Drivers were the community around it and their leadership in the space.

Timeline was 2 years (rewrite app, slowly migrate customers).  It’ll be more like 3-4 before it’s done.  There were issues where they were glad they didn’t mass migrate everyone at once.

Technical challenges:

Performance was a little lax (disk performance, they think) and they ended up needing more servers.  Used tricks like RAIDed EBSes to try to get the most io they could (mainly for the databases).

Every customer had a SSL cert, and they had 600 of them to mess with.  That was a problem because of the 5 Elastic IP limit.  Went to certs that allow subsidiary domains – Digicert allowed 100 per cert (other CAs limit to much less) so they could get 100 per IP.

App servers did outbound LDAP conns to customer premise for auth integration and they usually tried to allow those in via IP rules in their corporate firewalls, but now on Amazon outbound IPs are dynamic.  They set up a proxy with a static (elastic) Ip to route all that through.

Rightscale – they used it.  They like it.

They used nginx for the load balancing, SSL termination.  Was a single point of failure though.

Remember that many of the implementations you are hearing about now were started back before Rackspace had an API, before Amazon had load balancers, etc.

In discussion about hybrid clouds, the point was brought up a lot of providers talk about it – gogrid, opsource, rackspace – but often there are gotchas.

DevOps and the Cloud

Chris Hilton from Thoughtworks is all about the DevOps, and works on stuff like continuous deployment for a living.

DevOps is:

  • collaboration between devs and operations staff
  • agile sysadmin, using agile dev tools
  • dev/ops/qa integration to achieve business goals

Why DevOps?

Silos.  agile dev broke down the wall between dev/qa (and biz).

devs are usually incentivized for change, and ops are incentivized for stability, which creates an innate conflict.

but if both are incentivized to deliver business value instead…

DevOps Practices

  • version control!
  • automated provisioning and deployment (Puppet/chef/rPath)
  • self healing
  • monitoring infra and apps
  • identical environments dev/test/prod
  • automated db mgmt

Why DevOps In The Cloud?

cloud requires automation, devops provides automation


  • “Continuous Delivery” Humble and Farley
  • Rapid and Reliable Releases InfoQ
  • Refactoring Databases by Ambler and Sadalage

Another tidbit: they’re writing puppet lite in powershell to fill the tool gap – some tool suppliers are starting, but the general degree of tool support for people who use both Windows and Linux is shameful.

Moving Software from On Premise to SaaS

John Mikula of Pervasive tells us about the Pervasive Data Cloud.  They wanted to take their on premise “Data Integrator” product, basically a command line tool ($, devs needed to implement), to a wider audience.

Started 4 years ago.  They realized that the data sources they’re connecting to and pumping to, like Quickbooks Online, Salesforce, etc are all SaaS from the get go.   “Well, let’s make our middle part the same!”

They wrote a Java EE wrapper, put it on Rackspace colo initally.

It gets a customer’s metadata, puts it on a queue, another system takes it off and process it.  A very scaling-friendly architecture.  And Rackspace (colo) wasn’t scaling fast enough, so they moved it to Amazon.

Their initial system had 2 glassfish front ends, 25 workers

For queuing, they tried Amazon SQS but it was limited, then went to Apache Zookeeper

First effort was about “deploy a single app” – namely salesforce/quickbooks integration.  Then they made a domain specific model and refactored and made an API to manage the domain specific entities so new apps could be created easily.

Recommended approach – solve easy problems and work from there.  That’s more than enough for people to buy in.

Their core engine’s not designed for multitenancy – have batches of workers for one guy’s code – so their code can be unsafe but it’s in its own bucket and doesn’t mess up anyone else.

Changing internal business processes in a mature company was a challenge – moving from perm license model to per month just with accounting and whatnot was a big long hairy deal.

Making the API was rough.  His estimate of a couple months grew to 6.  Requirements gathering was a problem, very iterative.  They weren’t agile enough – they only had one interim release and it wasn’t really usable; if they did it again they’d do the agile ‘right thing’ of putting out usable milestones more frequently to see what worked and what people really needed.

In Closing

Whew!  I found all the presentations really engaging and thank everyone for sharing the nuts and bolts of how they did it!

Leave a comment

Filed under Cloud

Our First Cloud Product Released!

Hey all, I just wanted to take a moment to share with you that our first cloud-based product just went live!  LabVIEW Web UI Builder is National Instruments’ first SaaS application.  It’s actually free to use, go to and “Try It Now”, all you have to do is make an account.  It’s a freemium model, so you can use it, save your code, run it, etc. all you want; we charge to get the “Build & Deploy” functionality that enables you to compile, download, and deploy the bundled app to an embedded device or whatnot.

Essentially it’s a Silverlight app (can be installed out of browser on your box or just launched off the site) that lets you graphically program test & measurement, control, and simulation type of programs.  You can save your programs to the cloud or locally to your own machine.  The programs can interact via Web services with anything, but in our case it’s especially interesting when they interact with data acquisition devices.  There’s some sample programs on that page that show what can be done, though those are definitely tuned to engineers…  We have apps internally that let you play frogger and duck hunt, or do the usual Web mashup kinds of things calling google maps apis.  So feel free and try out some graphical programming!

Cool technology we used to do this:

And it’s 100% DevOps powered.  Our implementation team consists of developers and sysadmins, and we built the whole thing using an agile development methodology.  All our systems are created by model-driven automation from assets and definitions in source control.  We’ll post more about the specifics now that we’ve gotten version 1 done!  (Of course, the next product is just about ready too…)

Leave a comment

Filed under Cloud, DevOps

LASCON 2010: Why The Cloud Is More Secure Than Your Existing Systems

Why The Cloud Is More Secure Than Your Existing Systems

Saving the best of LASCON 2010 for last, my final session was the one I gave!  It was on cloud security, and is called “Why The Cloud Is More Secure Than Your Existing Systems.”  A daring title, I know.

You can read the slides (sadly, the animations don’t come through so some bits may not make sense…).  In general my premise is that people that worry about cloud security need to compare it to what they can actually do themselves.  Mocking a cloud provider’s data center for not being ISO 27001 compliant or having a two hour outage only makes sense if YOUR data center IS compliant and if your IT systems’ uptime is actually higher than that.  Too much of the discussion is about the FUD and not the reality.  Security guys have this picture in their mind of a super whizbang secure system and judge the cloud against that, even though the real security in the actual organization they work at is much less.  I illustrate this with ways in which our cloud systems are beating our IT systems in terms of availablity, DR, etc.

The cloud can give small to medium businesses – you know, the guys that form 99% of the business landscape – security features that heretofore were reserved for people with huge money and lots of staff.  Used to be, if you couldn’t pay $100k for Fortify, for instance, you just couldn’t do source code security scanning.  “Proper security” therefore has an about $1M entry fee, which of course means it’s only for billion dollar companies.  But now, given the cloud providers’ features, and new security as a service offerings, more vigorous security is within reach of more people.  And that’s great -building on the messages in previous sessions from Matt’s keynote and Homeland Security’s talk, we need pervasive security for ALL, not just for the biggest.

There’s more great stuff in there, so go check it out.

1 Comment

Filed under Cloud, Conferences, Security

LASCON 2010: HTTPS Can Byte Me

HTTPS Can Byte Me

This paper on the security problems of HTTPS was already presented at Black Hat 2010 by Robert Hansen, aka “RSnake”, of SecTheory and Josh Sokol of our own National Instruments.

This was a very technical talk so I’m not going to try to reproduce it all for you here.  Read the white paper and slides.  But basically there are a lot of things about how the Web works that makes HTTPS somewhat defeatable.

First, there are insecure redirects, DNS lookups, etc. before you ever get to a “secure” connection.  But even after that you can do a lot of hacking from traffic characterization – premapping sites, watching “encrypted” traffic and seeing patterns in size, get vs post, etc.  A lot of the discussion was around doing  things like making a user precache content to remove noisiness via a side channel (like a tab; browsers don’t segment tabs).  Anyway, there’s a lot of middle ground between “You can read all the traffic” and “The traffic is totally obscured to you,” and it’s that middle ground that it can be profitable to play in.

Leave a comment

Filed under Conferences, Security

LASCON 2010: Tell Me Your IP And I’ll Tell You Who You Are

Tell Me Your IP And I’ll Tell You Who You Are

Noa Bar-Yosef from Imperva talked about using IP addresses to identify attackers – it’s not as old and busted as you may think.  She argues that it is still useful to apply IP intelligence to security problems.

Industrialized hacking is a $1T business, not to mention competitive hacking/insiders, corporate espionage…  There’s bad people trying to get at you.

“Look at the IP address” has gotten to where it’s not considered useful, due to pooling from ISPs, masquerading, hopping… You certainly can’t use them to prove in court who someone is.

But… home users’ IPs persist 65% more than a day, 15% persist more than a week.  A lot of folks don’t go through aggregators, and not all hopping matters (the new IP is still in the same general location).  So the new “IP Intelligence” consists of gathering info, analyzing it, and using it intelligently.

Inherent info an IP gives you – its type of allocation, ownership, and geolocation.  You can apply reputation-based analytics to them usefully.

Geolocation can give context – you can restrict IPs by location, sure, but also it can provide “why are they hitting that” fraud detection.  Are hits from unusual locations, simultaneous from different locations,  or from places really different from what the account’s information would indicate?  Maybe you can’t block on them – but you can influence fuzzy decisions.  Flag for analysis. Trigger adaptive authentication or reduced functionality.

Dynamically allocated addresses aren’t aggregators, and 96% of spam comes from them.

Thwart masquerading – know the relays, blacklist them.  Check accept-language headers, response time, path…  Services provide “naughty” lists of bad IPs – also, whitelists of good guys.  Use realtime blacklist feeds (updated hourly).

Geolocation data can be obtained as a service (Quova) or database (Maxmind). Reputation data is somewhat fragmented by “spammer” or whatnot, and is available from various suppliers (who?)

I had to bail at this point unfortunately…  But in general a sound premise, that intel from IPs is still useful and can be used in a general if not specific sense.

Leave a comment

Filed under Conferences, Security

LASCON 2010: Mitigating Business Risks With Application Security

Mitigating Business Risks With Application Security

This talk was by Joe Jarzombek, Department of Homeland Security.  Normally I wouldn’t go to a management-track session called something like this, when I looked at the program this was my third choice out of all three tracks.  But James gave me a heads up that he had talked with Joe at dinner the previous night and he was engaging and knew his stuff, and since there were plenty of other NI’ers there to cover the other sessions, I took a chance, and I wasn’t disappointed!

From a pure “Web guy” standpoint it wasn’t super thrilling, but in my National Instruments hat, where we make hardware and software used to operate large hadron colliders and various other large scale important stuff where you would be very sad if things went awry with it, and by sad I mean “crushed to death,” it was very interesting.

Joe runs the DHS National Cyber Security Division’s new Software Assurance Program.  It’s a government effort to get this damn software secure, because it’s pretty obvious that events on a 9/11 kind of scale are more and more achievable via computer compromise.

They’re attempting to leverage standards and, much like OWASP’s approach with the Web security “Top 10,” they are starting out by pushing on the Top 25 CWE (Common Weakness Enumeration) errors in software.  What about the rest?  Fix those first, then worry about the rest!

Movement towards cloud computing has opened up people’s eyes to trust issues.  The same issues are relevant to every piece of COTS software you get as part of your supply chain!  It requires a profound shift from physical to virtual security.

“We need a rating scheme!”  Like food labels, for software.  They’re thinking about it in conjunction with NIST and OWASP as a way to raise product assurance expectations.

He mentioned that other software areas like embedded and industrial control might have different views on the top 25 and they’re very interested in how to include those.

They’re publishing a bunch of pocket guides to try to make the process accessible.  There’s a focus on supply risk chain management, including services.

Side note – don’t disable compiler warnings!  Even the compiler guys are working with the sec guys.  If you disable compiler warnings you’re on the “willful disregard” side of due diligence.

You need to provide security engineering and risk-based analysis throughout the lifecycle (plan, design, build, deploy) – that generates more resilient software products/systems.

  • Plan – risk assessment
  • Design – security design review
  • Build – app security testing
  • Deploy – SW support, scanning, remediation

They’re trying to incorporate software assurance programs into higher education.

Like Matt, he mentioned the Rugged Software Manifesto.  Hearing this both from “OWASP guy” and “Homeland security guy” convinced me it was something that bore looking into.  I like the focus on “rugged” – it’s more than just being secure, and “security” can seem like an ephemeral concept to untrained developers.  “Rugged” nicely encompasses reliable, secure, resilient…  I like it.

You can do software assurance self assessment they provide on their Web site to get started.

It was interesting, at times it seemed like Government Program Bureaucratese but then he’d pull out stuff like the CWE top 25 and the Rugged Software Manifesto – they really seem to be trying to leverage “real” efforts and help use the pull of Homeland Security’s Cyber Security Division to spread them more widely.

Leave a comment

Filed under Conferences, Security

LASCON 2010: Why Doesn’t Get Hacked

Why Doesn’t Get Hacked

The first LASCON session I went to was Why Doesn’t Get Hacked by James Flom (who with rsnake is  By its nature, it gets like 500-1000 hack attempts a week, but they’ve kept it secure for six years now.

From the network perspective, they use dual firewalls running the openBSD open source pf, which does Cisco-style traffic inspection.  Systems inside have no egress, and they have the user traffic and admin traffic segmented to different firewalls  sets and switches.

On the systems, they use chroot jails mounted read only.  Old school!  Jails are virtualization on the cheap, and if combined with a read only filesystem, give you a single out of band point of update, and you can do upgrades with minimal downtime.  They monitor them from the parent host.

Rsnake has done a whole separate presentation on how he’s secured his browser – the biggest attack vector is often “compromise the browser of an admin” and not direct attack on the asset.

They went to WordPress for their software – how to secure that?  Obviously code security’s a nightmare there.  So they set up a defense in depth scheme where they check the source ip, cert, and user/pass auth at the firewall, then to admin proxy check source IP, path, htaccess user/pass, and finally do the app auth.

Other stuff they do:

  • Secure logging to OSSEC – pflogd, waf logs, os logs, apache logs, parent logs, it goes off host so it’s reasonably tamper-proof
  • On-host WAF – custom, more of a “Web IDS” really, which feeds back “naughty people” to the firewall for blocking
  • For Apache – have your content owned by a different user, in their case there’s not even a user in the jail that can write to the files.
  • Use file ACLs, too.

Use case – they found an Apache flaw, reported it, and as is too often the case, Apache didn’t care.  So they modded their pf to detect the premise of the attack and block it (not just the specific attack).  (Heard of slowloris?)

Their ISP has been an issue – as they’ve moved their ISPs have shut them down out of cluelessness sometimes (Time Warner Business Class FTL).

They are moving to relayd for load balancing and SSL.  The PCI rule about “stay encrypted all the way to the box” is dumb, because it would prevent them from doing useful security inspection at that layer.

A good talk, though sadly a lot of the direct takeaways would mean “go to FreeBSD,” which I would rather not do.  But a lot of the concepts can port to other OSes and pure virtualization/cloud scenarios.  And note how joining network security, OS security, and appsec gets you way more leverage than having “separate layers” where each layer only worries about itself.n

And may I just say that I love how Apache can be run “read only” – sadly, most software, even other open source software like Tomcat, can’t be.  It all wants to write down into its config and its running directories itself, and it’s a horrible design practice and security risk.  If you’re writing software, remember that if it’s compromised and it can write to its own exes/config/etc. you’re owned.  Make your software run on a read only FS (with read/write in /tmp for stuff acceptable).  It’s the right thing to do.

Leave a comment

Filed under Conferences, Security

LASCON 2010: Why Does Bad Software Happen To Good People?

Why does bad software happen to good people?

First up at LASCON was the keynote by Matt Tesauro from Praetorian (and OWASP Foundation board member), speaking on “Why does bad software happen to good people?”  The problem in short is:

  • Software is everywhere, in everything
  • Software has problems
  • Why do we have these problems, and why can’t we create a secure software] ecosystem?

The root causes boil down to:

  • People trust software a lot nowadays
  • Blame developers for problems
  • Security of software is hidden
  • Companies just CYA in their EULAs
  • Lack of market reward for secure software
  • First mover advantage, taking time on security often not done
  • Regulation can’t keep up

So the trick is to address visibility of application security, and in a manner that can take root despite the market pressures against it.  We have to break the “black box” cycle of trust and find ways to prevent problems rather than focusing on coping with the aftermath.

He made the point that the physical engineering disciplines figured out safety testing long ago, like the “slump test” for concrete.  We don’t have the equivalent kind of standards and pervasive testability for software safety.  How do we make software testable, inspectable, and transparent?

Efforts underway:

  • They got Craig Youngkins, a big python guy, to start Python, which has been successful as a developer-focused grass roots effort
  • The Rugged Software Manifesto at is similar to the Agile Manifesto and it advocates resilient (including secure) software at the ideological level.

I really liked this talk and a number of things resonated with me.  First of all, working for a test & measurement company that serves the “real engineering” disciplines, I often have noted that software engineering needs best practices taken from those disciplines.  If it happens for jumbo jets then it can happen for your shitty business application.  Don’t appeal to complexity as a reason software can’t be inspected.

Also, the Rugged Software Manifesto dovetails well with a lot of our internal discussion on reliability.  And having “rugged” combine reliability, security, and other related concepts and make it appealing to grass roots developers is great.  “Quality initiatives” suck.  A “rugged manifesto” might just work.  It’s how agile kicked “CMMI”‘s ass.

The points about how pervasive software are now are well taken, including the guy with the mechanical arms who died in a car crash – software fault?  We’ll never know.  As we get more and more information systems embedded with/in us we have the real possibility of a “Ghost In The Shell” kind of world, and software security isn’t just about your credit card going missing but about your very real physical safety.

He threw in some other interesting tidbits that I noted down to look up later, including the ToorCon “Real Men Carry Pink Pagers” presentation about hacking the Girl Tech IM-Me toy into a weaponized attack tool, and some open source animated movie called Sintel.

It was a great start to the conference, raised some good questions for thought and I got a lot out of it.

Leave a comment

Filed under Conferences, Security

LASCON 2010 Conference Report

LASCON 2010 was awesome.  It’s an Austin app security conference put on by the Austin OWASP chapter. Josh Sokol and James Wickett did a great job of putting the thing together; for a first time convention it was really well run and went very smoothly.  The place was just about full up, about 200 people.  I saw people I knew there from Austin Networking, the University of Texas, HomeAway, and more.  It was a great crowd, all sorts of really sharp people, both appsec pros and others.

And the swag was nice, got a good quality bugout bag and shirt, and the OWASP gear they were selling was high quality – no crappy black geek tshirts.

I wish I had more time to talk with the suppliers there; I did make a quick run in to talk to Fortify and Veracode.  Both now have SaaS offerings where you can buy in for upload scanning of your source (Fortify) or your binaries (Veracode) without having to spring for their big ass $100k software packages, which is great – if proper security is only the purview of billion dollar companies, then we’ll never be secure.

At the happy hour they brought in a mechanical bull!  We had some friends in from Cloudkick in SF and they asked me with some concern, “Do all conferences in Austin do this?”  Nope, first time I’ve seen it, but it was awesome!  After some of the free drinks, I was all about it.  They did something really clever with the drinks – two drink tickets free, but you could get more by going and talking to the vendors at their booths.  That’s a win-win!  No “fill out a grade school passport to get entered into a drawing” kind of crap.

Speaking of drawings, they had a lot of volunteers working hard to run the con, they did a great job.

I took notes from the presentations I went to, they’re coming as separate posts.  I detected a couple common threads I found very interesting.  The Rugged Software Manifesto was mentioned by speakers in multiple sessions including by the Department of Homeland Security.  It’s clear that as software becomes more and more pervasive in our lives that health, safety, national security, and corporate livelihood are all coming to depend on solid, secure software and frankly we’re not well on the right track towards that happening.

Also, the need for closer cooperation between developers, appsec people, and traditional netsec people was a clear call to action.  This makes me think about the ongoing call for developer/ops collaboration from DevOps – truly, it’s a symptom of a larger need to find a better way for everyone to work together to generate these lovely computerized monstrosities we work on.

So check out my notes from the sessions – believe me, if it was boring I wouldn’t bother to write it down.

I hear the conference turned a profit and it was a big success from my point of view, so here’s hoping it’s even bigger and better in 2011!  Two days!  It’s calling to you!

Leave a comment

Filed under Conferences, Security