Monthly Archives: January 2009

A DoS We Can Believe In

We knew that the historic inauguration of Barack Obama would be generating a lot more Internet traffic than usual, both in general and specifically here at NI.  Being prudent Web Admin types, we checked around to make sure we thought that there wouldn’t be any untoward effects on our Web site.  Like many corporate sites, we use the same pipe for inbound Internet client usage and outbound Web traffic, so employees streaming video to watch the event could pose a problem.  We got all thumbs up after consulting with our networking team, and decided to not even send any messaging asking people to avoid streaming.  But, we monitored the situation carefully as the day unwound.  Here’s what we saw, just for your edification!

Our max inbound Internet throughput was 285 Mbps, about double our usual peak.  We saw a ni.com Web site performance degradation of about 25% for less than two hours according to our Keynote stats.  ni.com ASPs were affected proportionately which indicates the slowdown was Internet-wide and not unique to our specific Internet connection here in Austin.  The slowdown was less pronounced internationally, but still visible.  So in summary – not a global holocaust, but a noticeable bump.

Cacti graphs showing our Internet connection traffic:

obamabumpcactihrlyobamabumpcactidaily

Keynote graph of several of our Web assets, showing global response time in seconds:obamabumpkeynoteLooking at the traffic specifically, there were two main standouts.  We had TCP 1935, which is Flash RTMP, peaking around 85 Mbps, and UDP 8247, which is a special CNN port (they use a plugin called “Octoshape” with their Flash streaming), peaking at 50 Mbps.   We have an overall presence of about 2500 people here at our Austin HQ on an average day, but we can’t tell exactly how many were streaming.  (Our NetQoS setup shows us there were 13,600 ‘flows,’ but every time a stream stops and starts that creates a new one – and the streams were hiccupping like crazy.  We’d have to do a bunch of Excel work to figure out max concurrent, and have better things to do.)

In terms of the streaming provider breakdown – since everyone uses Akamai now, the vast majority showed as “Akamai”.  We could probably dig more to find out, but we don’t really care all that much.  And, many of the sources were overwhelmed, which helped some.

We just wanted to share the data, in case anyone finds it helpful or interesting.

Leave a comment

Filed under General, Uncategorized

Beware The Wolf In Supplier’s Clothing

As you all know, the economic climate of 2009 is a cold, cold winter indeed.  And like wolves starved by the cold and hardship of the season, our suppliers have turned feral.

When everyone’s sales slip due to the down economy, companies (and individual sales reps) are desperate to make their numbers.  How are they doing it?  By trying to jack up maintenance costs, in some cases by more than 100%!  It’s way more than isolated incidents; all our maintenance renewals coming up are meeting with hugely inflated quotes.  And not fly-by-night companies either, I don’t want to name names but let’s just say I am confident everyone out there has heard of all of them.

So protect yourself.  In your dealings with your supplier reps, start making it clear way ahead of time that your economic situation sucks too and you certainly expect that there’s a price freeze in place.  Don’t put up with it either – they know they’re going to make plenty of money off all the goons they send quotes to who will just rubber-stamp it and send it on so they can return to ESPNZone (I’m looking at you, State of Texas).  If you put up enough resistance they’ll go looking for easier pickings, just like those mean ol’ wolves do.    We had one outfit that wanted to jack up our maintenance cost by $125k a year, but luckily our IT director is a firm lady who has no problems with browbeating a sales rep until he cries.  In the end, we let them have a 5% increase because we ended up feeling sorry for them.

And have a backup plan.  If they really do have you over a barrel, then you’re low on leverage – you can try offering reference calls, presenting at conferences, and other handy non-cash incentives to them.  But when it comes down to it, you need to be able to walk away from them.  And to do this you need to plan ahead.  There are very few things that there’s only one of.  Have multiple suppliers lined up, and have a plan to change hardware or software if you have to.  Also look into open source, or third party support – even if it’s “not as good,” these days you have to decide how much good is worth how much money.

Now don’t get me wrong, we like to partner with our suppliers and treat them friendly.  Win-win and all that.  But good fences build good neighbors, and there’s nothing friendly about showing up and saying  “Hey, your operations will grind to a halt without our product, so stick ’em up and give me double this year!”

Be advised, that gleam in Bob the Sales Rep’s eyes will be a little hungrier than usual these days, and he’s gotta eat one of God’s little forest creatures to live.  Just make sure it’s not you.

1 Comment

Filed under General, Uncategorized