SaaS Headaches

There’s a lot of promise in the new SaaS (software as a service; what used to be called ASPs, or Application Service providers, till Microsoft crapped all over that acronym) and newer PaaS (platform as a service) spaces (and look for a steady stream of new “aaS”es to come).  However, there are a lot of gotchas in signing on with a SaaS vendor.  You’d like to be able to believe that they have decent performance, uptime, security, etc., especially after they tell you “Oh, all kinds of big companies use us; Dell, IBM…”  This is exacerbated by SaaS often being an “end run” around IT in the enterprise, so naive users can get sold a bill of goods without proper technical oversight.  SaaS is a big buzzword now, and there are a lot of startups springing up that do not necessarily have experience running large scale sites.  Think about how many MMORPG games still get scuttled due to poor operational performance.  SaaS is the same.

Here’s some things to keep in mind when selecting a SaaS vendor, laced with real life horror stories from our experiences.

1.  Performance/Availability.  Set a hard performance/availability SLA in the contract.  Many vendors won’t even have an SLA clause, or they’ll have one that says “99.9% uptime!” without any remedy clause for what if they don’t hit that.  You want a clear SLA with a clear measurement method and clear “money back” if they don’t hit it.  We use a 2 second global performance SLA as measured by a Keynote Global 35 monitor.  But the SLA isn’t the whole story – you are counting on these people to accomplish your goals.

True story.  We did an implementation with a new SaaS supplier.  Everything looked fine but the team had not done performance testing.  With only a week to go live, they finally loaded in a full set of data into the system and saw performance that was horrible, clearly in the ~30 second page load time even in the US (much worse internationally).  But it isn’t as simple as “Oh, you’re not hitting the SLA, goodbye…”  We have months of time invested in this supplier. In this case, the supplier just could not solve the performance issues and we had to cut bait with them, a very expensive thing to do (but not more expensive than going live with an awful solution on our site).

2.  Privacy policy.  Make sure you know what they’re going to do with your user data they collect and make sure that matches what your privacy policy says.  We’re international and so have to abide by a fairly restrictive EU-compliant privacy policy.  Many SaaS vendors don’t know what exactly that entails, and so can run you afoul of the law – remember, you’re legally responsible for your site whether you’ve outsourced parts of it or not.

3.  Security.  One, if you have compliance concerns like PCI etc. you need to make sure they’re complaint as well, and certified as such by an auditor (don’t take “Oh, yeah, we’re PCI complaint” at anyone’s word).  Two, you are almost certainly going to have to exchange user credentials, so a user can log in across you site and the SaaS site with the same login, and the “right ways” to do this, like SAML, are supported by about one tenth of one percent of SaaS vendors.  You need to carefully review how you’re going to do it to avoid security problems.

4.  Contingency plan.  Companies go out of business, or the relationship between them degrades.  You have to have a plan in place for when your SaaS vendor dies, gets bought by HP and has their price quadrupled, or you decide you hate each other, or the cost isn’t what you anticipated (see point 5 below).   You need some of this in your contract – in any event, you get your data and if they are dying, a perpetual license to their software.  (If you’re really lucky it’s one of these firms that has a software package and an ASP version both.)

True story.  Our ni.com forums were hosted for many years by a company called QUIQ.  In the big dotcom bubble bust, they went down and had the revenooers show up to unplug them.   We were very lucky in that they were interested in helping us despite their own problems, and that we had bad ass technical folks on staff.  We paid to have a QUIQ engineer come down and load up their forum software on our systems (and this wasn’t a bundled software solution, it was their internal-only stuff) and transition our data over.  We then supported this as our forum solution for a couple years, often having to take measures like decompiling the Java to make changes.   But that was better than being dead in the water.  Now with our new forums vendor, we do things like get regular backups of our data, gateway the forum to NNTP, etc.

5.  Cost.  SaaS vendors almost universally charge you by usage.  Which is “fair” – but the Internet is a wild place.  What about when that new Chinese experimental spider (soso, you suck!) decides to grant you a couple hundred thousand extra page views one day?  You get stuck with the bill.  Corporations have to be able to budget for their expenditures, and there is risk in this business model that one time events and/or unexpected growth will fundamentally alter your cost and ROI structure.

There are several potential mitigations here.  One is to get a SaaS vendor that implements throttling, so you can meter back untoward amounts of traffic.  Another is to have specific payment scales that mitigate this – you want to avoid “cell phone” type plans that give you a certain amount and then an abusive overage charge like the plague.  Look for plans like ISPs, where you pay for the 90th percentile peak of traffic, for example.  Or have a prepay agreement, and/or specify what kinds of traffic you’ll pay for.  (On many Internet sites, spiders account for 30% or so of the traffic and thus the expense.)

6.  Quality.  Do a pilot/proof of concept.  DO ONE!!!  A SaaS vendor is not yet a commodity in most areas.  You are getting as deep in bed with them as if you bought hardware and software and in house programming.  Don’t sign the contract until you have seen it work for you.  Build deliverables and payment schedule into the contract – “You get 1/3 upon requirement completion, 1/3 upon signoff of a test version, and 1/3 upon go live” is popular.  We have one purchasing agent who likes to push “Sign the contract, and have a 30 day ‘out’ clause if things don’t go well.”  But this ignores the deep investment into even a SaaS vendor (and the fact that most implementations aren’t fully baked in 30 days).  As a result, we have several business units even now signing up with ASPs without doing a formal pilot, and every one of them will come to regret it bitterly.  Only the rich and the idiotic buy a car without a test drive and a mechanic checkup.  Any SaaS solution will be much more expensive than any car.

None of this is to say that SaaS is bad or should be avoided.  In fact, increasingly, SaaS is the best solution to many functional areas. But it needs to be evaluated just like any other solution.  Is the 5 year TCO really better than in house, not just the first year cost?  And does it really do what you need – functionality wise, but also in the areas of performance, security, and these other areas which make your functionality meaningful to the users?  If you can’t answer these questions, you are betting a lot of money and your reputation on an untried horse.  Find out the answers. Then you can say “yes” with confidence.

2 Comments

Filed under Cloud

2 responses to “SaaS Headaches

  1. Josh

    Great blog post. Any time you are dealing with any sort of 3rd party vendor (SaaS or otherwise), you should always make sure that their polcies are at least as restrictive as your own.

  2. Pingback: Beware the Deceptive SLA, My Friend - Web Admin Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s