Category Archives: General

Random musings on technology or management or whatever.

A DoS We Can Believe In

We knew that the historic inauguration of Barack Obama would be generating a lot more Internet traffic than usual, both in general and specifically here at NI.  Being prudent Web Admin types, we checked around to make sure we thought that there wouldn’t be any untoward effects on our Web site.  Like many corporate sites, we use the same pipe for inbound Internet client usage and outbound Web traffic, so employees streaming video to watch the event could pose a problem.  We got all thumbs up after consulting with our networking team, and decided to not even send any messaging asking people to avoid streaming.  But, we monitored the situation carefully as the day unwound.  Here’s what we saw, just for your edification!

Our max inbound Internet throughput was 285 Mbps, about double our usual peak.  We saw a ni.com Web site performance degradation of about 25% for less than two hours according to our Keynote stats.  ni.com ASPs were affected proportionately which indicates the slowdown was Internet-wide and not unique to our specific Internet connection here in Austin.  The slowdown was less pronounced internationally, but still visible.  So in summary – not a global holocaust, but a noticeable bump.

Cacti graphs showing our Internet connection traffic:

obamabumpcactihrlyobamabumpcactidaily

Keynote graph of several of our Web assets, showing global response time in seconds:obamabumpkeynoteLooking at the traffic specifically, there were two main standouts.  We had TCP 1935, which is Flash RTMP, peaking around 85 Mbps, and UDP 8247, which is a special CNN port (they use a plugin called “Octoshape” with their Flash streaming), peaking at 50 Mbps.   We have an overall presence of about 2500 people here at our Austin HQ on an average day, but we can’t tell exactly how many were streaming.  (Our NetQoS setup shows us there were 13,600 ‘flows,’ but every time a stream stops and starts that creates a new one – and the streams were hiccupping like crazy.  We’d have to do a bunch of Excel work to figure out max concurrent, and have better things to do.)

In terms of the streaming provider breakdown – since everyone uses Akamai now, the vast majority showed as “Akamai”.  We could probably dig more to find out, but we don’t really care all that much.  And, many of the sources were overwhelmed, which helped some.

We just wanted to share the data, in case anyone finds it helpful or interesting.

Leave a comment

Filed under General, Uncategorized

Beware The Wolf In Supplier’s Clothing

As you all know, the economic climate of 2009 is a cold, cold winter indeed.  And like wolves starved by the cold and hardship of the season, our suppliers have turned feral.

When everyone’s sales slip due to the down economy, companies (and individual sales reps) are desperate to make their numbers.  How are they doing it?  By trying to jack up maintenance costs, in some cases by more than 100%!  It’s way more than isolated incidents; all our maintenance renewals coming up are meeting with hugely inflated quotes.  And not fly-by-night companies either, I don’t want to name names but let’s just say I am confident everyone out there has heard of all of them.

So protect yourself.  In your dealings with your supplier reps, start making it clear way ahead of time that your economic situation sucks too and you certainly expect that there’s a price freeze in place.  Don’t put up with it either – they know they’re going to make plenty of money off all the goons they send quotes to who will just rubber-stamp it and send it on so they can return to ESPNZone (I’m looking at you, State of Texas).  If you put up enough resistance they’ll go looking for easier pickings, just like those mean ol’ wolves do.    We had one outfit that wanted to jack up our maintenance cost by $125k a year, but luckily our IT director is a firm lady who has no problems with browbeating a sales rep until he cries.  In the end, we let them have a 5% increase because we ended up feeling sorry for them.

And have a backup plan.  If they really do have you over a barrel, then you’re low on leverage – you can try offering reference calls, presenting at conferences, and other handy non-cash incentives to them.  But when it comes down to it, you need to be able to walk away from them.  And to do this you need to plan ahead.  There are very few things that there’s only one of.  Have multiple suppliers lined up, and have a plan to change hardware or software if you have to.  Also look into open source, or third party support – even if it’s “not as good,” these days you have to decide how much good is worth how much money.

Now don’t get me wrong, we like to partner with our suppliers and treat them friendly.  Win-win and all that.  But good fences build good neighbors, and there’s nothing friendly about showing up and saying  “Hey, your operations will grind to a halt without our product, so stick ’em up and give me double this year!”

Be advised, that gleam in Bob the Sales Rep’s eyes will be a little hungrier than usual these days, and he’s gotta eat one of God’s little forest creatures to live.  Just make sure it’s not you.

1 Comment

Filed under General, Uncategorized

Google Chrome Hates You (Error 320)

The 1.0 release of Google Chrome has everyone abuzz.  Here at NI, loads of people are adopting it.  Shortly after it went gold, we started to hear from users that they were having problems with our internal collaboration solution, based on the Atlassian Confluence wiki product.  They’d hit a page and get a terse error, which if you clicked on “More Details” you got the slightly more helpful, or at least Googleable, string  “Error 320 (net::ERR_INVALID_RESPONSE): Unknown error.”

At first, it seemed like if people reloaded or cleared cache the problem went away.  It turned out this wasn’t true – we have two load balanced servers in a cluster serving this site.  One server worked in Chrome and the other didn’t; reloading or otherwise breaking persistence just got you the working server for a time.  But both servers worked perfectly in IE and Firefox (every version we have lying around).

So we started researching.  Both servers were as identical as we could make them.  Was it a Confluence bug?  No, we have phpBB on both servers and it showed the same behavior – so it looked like an Apache level problem.

Sure enough, I looked in the logs.  The error didn’t generate an Apache error, it was still considered a 200 OK response, but when I compared the log strings the box that Chrome was erroring on showed that the cookie wasn’t being passed up; that field was blank (it was populated with the cookie value on the other box and on both boxes when hit in IE/Firefox).  Both boxes had an identically compiled Apache 2.0.61.  I diffed all the config files- except for boxname and IP, no difference.  The problem persisted for more than a week.

We did a graceful Apache restart for kicks – no effect.  Desperate, we did a full Apache stop/start – and the problem disappeared!  Not sure for how long.  If it recurs, I’ll take a packet trace and see if Chrome is just not sending the cookie, or sending it partially, or sending it and it’s Apache jacking up…  But it’s strange there would be an Apache-end problem that only Chrome would experience.

I see a number of posts out there in the wide world about this issue; people have seen this Chrome behavior in YouTube, Lycos, etc.  Mostly they think that reloading/clearing cache fixes it but I suspect that those services also have large load balanced clusters, and by luck of the draw they’re just getting a “good” one.

Any other server admins out there having Chrome issues, and can confirm this?  I’d be real interested in knowing what Web servers/versions it’s affecting.  And a packet trace of a “bad” hit would probably show the root cause.  I suspect for some reason Chrome is partially sending the cookie or whatnot, choking the hit.

3 Comments

Filed under General, Uncategorized

Vignette Village 2008

Vignette, the Austin-based Web content management company,  has an annual show called Vignette Village.  A whole crew went from our company; Mark and I represented the Web Admins.

I got a lot out of Village though I wasn’t expecting to.  There was excitement in the air and clear commitment to continued development of their core Vignette Content Management (VCM V7) product and other products which had been lacking for the last couple years.  To be honest, I had begun to expect that it was a matter of time unti the Plone/Drupal/Joomla crowd outstripped VCM, but they seem to be making the changes required to keep the product as the true enterprise choice.  We already moved off Vignette Dialog, which was a very good email marketing package, because of the lackof support and new development.  I don’t know the details, but basically Vignette went all meathead and turned away from their core products to chase medical/legal document management money a couple years ago, combined with financial problems and layoffs, and so the products started to suck.  They seem to have turned that around, though, and everyone I spoke to inside Vignette is excited about their new leadership, especially Bertrand de Coatpont, the new VCM product manager.

The new Vignette Recommendations (OEMed from Baynote Systems) looks really good, and will expose some new data to us that I think can be used in a lot of different and innovative ways.  From previous descriptions I had thought “Yeah, whatever, BazaarVoice but from Vignette, which doesn’t necessarily inspire confidence in me” (frankly, we Web Admins have learned to be suspicious about additional offerings from Oracle, Vignette, HP, etc. as they will try to sell you crap on the strength of their brand name and alleged integration).  But the reality, which is an extremely elegant way of collecting and immediately reusing usage info, is brilliant and especially with their social search aspect to it, I feel like they have an actual vision they’re working towards.  So two thumbs up there!

Also two thumbs up on the Transfer Tool, which allows you to easily clone VCM installs to other servers – it’ll allow for frequent and efficient refreshes.  We had to have that working, so we Web Admins had devised a complicated two-day process to clone an environment; this should be much better.

VCM 7.6 is planned to be complete this year, and it has a lot of compelling features – you can migrate Content Type Definitions (change a CTD and the content changes inside the VCM to fit), lots of performance, availability, and console GUI fixes…  Then “Ace,” which everyone knows is VCM V8 but they don’t want to own up to that yet, has a total GUI overhaul.  Most of the issues we have with VCM are content contributor usability, so that’s great.

All in all, two days well spent.  It definitely exceeded my expectations (and I’ve been to Village in years previous).

5 Comments

Filed under General, Uncategorized

Our Search Implementation In The News!

InformationWeek did a big story on enterprise search, and used NI as their lead example!  Note all the system info in the article that I fed them. And we’re getting a lot of fun out of Graff’s quote about how it’s easy to sign off on more resources forus, we’re including that in every purchase req now. 🙂

One of the reasons that our FAST enterprise search program has been so successful here is that the programmers and the Web Admins have worked pretty much 50/50 on the platform.  Also, FAST is a great product and has great support (we’re waiting with bated breath to see if Microsoft screws it up; we’ve been with FAST since way way before they got bought), and we have some very visionary search business folks who saw its potential early on.

Nowadays, search is more than it was considered traditionally.  We have a normal “search box”, of course.   But we also run our faceted navigation off search (e.g. our Data Acquisition product line page), pull things like related links and other resources (see resources tab on this page).  Search, in many ways, can be used the way people have used databases in the past.  With some metadata added, a search index is kinda like a big database, highly denormalized for speed, focusing on text search.  In fact, I think there’s a master’s thesis in there somewhere as to when search makes sense vs. when a database makes sense.  Databases make sense with lots of numerical information, but on the Web that’s frankly a fringe use case!   On the Web it’s all about text, from name/address to links to articles to product info…  When we did things like query related links out of a database table, and I mean an oracle database table on a big ass Solaris box, it was painfully slow.  Pulling from search, it’s 15 milliseconds.

As a result, our internal search use is even more killer.  We pull Intranet pages, documents from Notes repositories, data from our Oracle ERP system, files off file shares, etc. all into one place and let people delve through it.  They’ve even implemented “screens” on top of some of the data (mainly because Oracle ERP is painful to use).  Our entire sales force is gaga over it.

Anyway, so yay to modern search technology, yay to FAST, and yay us!

1 Comment

Filed under General, Uncategorized

No No, You Really DO Want To Use Live Search

It’s been in the news that Microsoft is pushing “rewards programs” for people to use Live Search and the Live Toolbar.  But did you know they’re trying to get your local IT department to do it for you?

Yep, the program’s called the “Search@Work Rewards Program”.  If your IT department puts IE, with Live Search as the default search, and the Live Toolbar installed, and some kind of tracker plugin called the “Search Rewards Client,” on your company PCs, then they get Microsoft service credits!  Yay.  I can only assume my ISP is next.

Here’s the exact service description from Microsoft.  Note that they’re tracking Yahoo and Google ad impressions too!  The rest of it’s “fair enough” at least by usual IT industry standards but that’s kinda shady I think.

Continue reading

Leave a comment

Filed under General, Uncategorized

Beware the Deceptive SLA, My Friend

We’re trying to come to an agreement with a SaaS vendor about performance and availability service level agreements (SLAs).  I discussed this topic some in my previous “SaaS Headaches” post.  I thought it would be instructive to show people the standard kind of “defense in depth” that suppliers can have to protect against being held responsible for what they host for you.

We’ve been working on a deal with one specific supplier.  As part of it, they’ll be hosting some images for our site.  There’s a business team primarily responsible for evaluating their functionality etc., we’re just in the mix as the faithful watchdogs of performance and availability for our site.

Round 1 – “What are these SLAs you speak of?”  The vendor offers no SLA.  “Unacceptable,” we tell the project team.  They fret about having to worry about that along with the 100 other details of coming to an agreement with the supplier, but duly go back and squeeze them.  It takes a couple squeezes because the supplier likes to forget about this topic – send a list of five questions with one of them being “SLA,” you get four answers back, ignoring the SLA question.

Round 2 – “Oh, you said ‘SLA’!  Oh, sure, we have one of those.”  We read the SLA and it only commits to their main host being pingable.  Our service could be completely down, and it doesn’t speak to that.  Back to our project team, who now between the business users, procurement agent, and legal guy need more urging to lean on the supplier.  The supplier plays dumb for a while, and then…

Continue reading

1 Comment

Filed under Cloud, General

Oracle + BEA = ?

We use Oracle Application Server as our Java app server at NI. Yeah, yeah, I’ll wait till you stop laughing.

Why not JBoss or WebLogic or WebSphere? Well, a couple reasons. We made the decision five years ago, and JBoss wasn’t solid then, and we needed J2EE support so plain Tomcat wasn’t enough. And we’re a huge Oracle shop and figured that if we were using the same app server on the Web and our ERP tiers there’d be leverage in terms of developer knowledge etc.  Would we make that same decision today? I’m not sure about that (I can hear my team members shouting “hell no” over the cube walls).  Although since we’ve also gone with Oracle’s SOA suite for ESB and BPEL it would be harder to switch. But still tempting – Oracle has done a horrible job in getting their app server supported by other vendors. Every time we buy something and look at the supported app server section of their support matrix, and we ask “What about Oracle’s OAS?” we get expressions of mixed horror and pity from the supplier. (I liked it when the Chinese technical guy from one eComm vendor we had in responded to this question with, “You know, the Tomcat is good, and free! Maybe you use that!”)

Anyway, Oracle bought BEA a while back, which got keen interest from us. Stay with Oracle *and* use a good app server that other people support?  Tempting!  But Oracle’s been farting around for six months without coming out with a statement on what this will mean for the products. Oracle’s finally done a Webcast describing their strategy. Well, it’s half marketing and a celebration of how many million dollars they have. But there’s also a lot of product strategy in there. I’ll sum it up for you because the damn webcast is nearly two hours long, and I don’t want other people to have to waste that much time on it. Unless you like to hear someone go on about “strategic clarity” and “customer profiles,” in which case this is two hours of bliss for you and you should watch it.  Although I also had the stream break a bunch of times while watching.  Who the heck uses RealPlayer any more?  Anyway, here’s a list of the interesting product facts from the Webcast.  Some are marked with their timestamp if you want to fast forward to them and see more.

Continue reading

2 Comments

Filed under General, Uncategorized

Quick Blogging Tip

All yesterday I was being annoyed by the need to write up my blog posts in another editor and paste them over into WordPress.  You have to do that because composing text longer than about 3 sentences in a browser window is taking your life in your hands.  But I discovered even in cutting and pasting from Wordpad you get formatting inserted that drives the TinyMCE editor crazy.  And Notepad was giving me line break problems.  (And it needs not be said that you should never ever paste from Word…)

But Robert cued me in to PureText, which is a little Windows addon that strips all formatting from text when you cut/paste it.  By default you Windows+V instead of Control+V and voila, no crap.  Yay!

4 Comments

Filed under General, Uncategorized