Category Archives: Uncategorized

The Velocity 2009 Conference Experience

Velocity 2009 is well underway and going great!  Here’s my blow by blow of how it went down.

Peco, my erstwhile Bulgarian comrade, and I came in to San Jose  from Austin on Sunday.  We got situated at the fairly swank hotel, the Fairmont, and wandered out to find food.  There was some festival going on so the area was really hopping.  After a bit of wandering, we had a reasonably tasty dinner at Original Joe’s.  Then we walked around the cool pedestrian part of downtown San Jose and ended up watching “Terminator:  Salvation” at a neighborhood movie theater.

We went down at 8  AM the next morning for registration.  We saw good ol’ Steve Souders, and hooked up with a big crew from BazaarVoice, a local Austin startup that’s doing well.  (P.S. I don’t know who that hot brunette is in the lead image on their home page, but I can clearly tell that she wants me!)

This first day is an optional “workshop” day with a number of in depth 90 minute sessions.  There were two tracks, operations and performance.   Mostly I covered ops and Peco covered performance.  Next time – the first session!

Leave a comment

Filed under DevOps, Uncategorized

State of SOA

We have a pretty decent-sized SOA implementation here.  I got interviewed by InformationWeek magazine about it, and you can read my thoughts on SOAP vs REST and related topics in:

InformationWeek Analytics: State Of SOA

Leave a comment

Filed under General, Uncategorized

Browser Support – Just Do It

I am moved to post today by a gripe.  We have a lot of products and SaaS vendors that for some reason feel like they don’t need to support browsers other than whatever it is they have in their mind as the one browser they’re going to support.   I have Firefox 3, Internet Explorer 8 beta, and Chrome on my PC but still can’t use many of the darn programs I needed to use today.  (Of course, you can’t run different IE versions on the same box without resorting to virtualization or similar, so once I went to IE8 beta I knew I was in a world of hurt).

Let me share with you the top 10 browsers we see on our Web site.  These numbers are from the last 500k visits so they should be statistically representative.

  • IE7 – 34.9%
  • Firefox – 31.0%
  • IE6 – 25.9%
  • Safari (includes Chrome) – 4.1%
  • Opera 9 – 2.3%
  • IE8 beta – .9%
  • Mozilla – .4%
  • Charlotte – .1%
  • Yeti – .1%
  • IE5 – .1%

All you suppliers who think “I don’t need to support Firefox” – think again.  And you’re all doing a bad job of supporting IE8.  I know it’s new – but if you’ve already been only supporting one browser, be advised that as soon as IE8 goes gold everyone will auto-download it from Microsoft and then you’re SOL.   And there’s a lot of IE6 out there still, even if you are trying to do “IE only.”

To name names – Peopleclick.  IE7 support only.  Really?  You really only want 35% of users to use your product?  Or you think we’re going to mandate an internal company standard for your one app?  Get real.

Sharepoint.  No editing in Firefox.  When we evaluated intranet collaboration solutions here, we got down to Atlassian Confluence and Sharepoint as finalists, but then the “no Firefox” factor got Sharepoint booted for cause.  Confluence itself doesn’t support Safari until its newest version, which was annoying.  (Microsoft does promise the new version of Sharepoint out later this year will have adequate Firefox support.)

Graphs don’t work right in Firefox in Panorama, otherwise a pet favorite APM tool.

So guys – I know it’s a pain, but the Windows browser market is split and Macs are undergoing a renaissance.  Real companies don’t tell 5 to 10 percent of their customers to screw off (let alone 65%, Peopleclick).  It’s a cost of doing business.   You’re getting out of a whole bunch of client side code writing by cheating and using Web browsers for it, so be grateful for that rather than ungrateful that you have to test in a couple different browsers.  Because corporate decisionmakers like myself will ask, and we will make buying decisions based on it.


Filed under General, Uncategorized

A DoS We Can Believe In

We knew that the historic inauguration of Barack Obama would be generating a lot more Internet traffic than usual, both in general and specifically here at NI.  Being prudent Web Admin types, we checked around to make sure we thought that there wouldn’t be any untoward effects on our Web site.  Like many corporate sites, we use the same pipe for inbound Internet client usage and outbound Web traffic, so employees streaming video to watch the event could pose a problem.  We got all thumbs up after consulting with our networking team, and decided to not even send any messaging asking people to avoid streaming.  But, we monitored the situation carefully as the day unwound.  Here’s what we saw, just for your edification!

Our max inbound Internet throughput was 285 Mbps, about double our usual peak.  We saw a Web site performance degradation of about 25% for less than two hours according to our Keynote stats. ASPs were affected proportionately which indicates the slowdown was Internet-wide and not unique to our specific Internet connection here in Austin.  The slowdown was less pronounced internationally, but still visible.  So in summary – not a global holocaust, but a noticeable bump.

Cacti graphs showing our Internet connection traffic:


Keynote graph of several of our Web assets, showing global response time in seconds:obamabumpkeynoteLooking at the traffic specifically, there were two main standouts.  We had TCP 1935, which is Flash RTMP, peaking around 85 Mbps, and UDP 8247, which is a special CNN port (they use a plugin called “Octoshape” with their Flash streaming), peaking at 50 Mbps.   We have an overall presence of about 2500 people here at our Austin HQ on an average day, but we can’t tell exactly how many were streaming.  (Our NetQoS setup shows us there were 13,600 ‘flows,’ but every time a stream stops and starts that creates a new one – and the streams were hiccupping like crazy.  We’d have to do a bunch of Excel work to figure out max concurrent, and have better things to do.)

In terms of the streaming provider breakdown – since everyone uses Akamai now, the vast majority showed as “Akamai”.  We could probably dig more to find out, but we don’t really care all that much.  And, many of the sources were overwhelmed, which helped some.

We just wanted to share the data, in case anyone finds it helpful or interesting.

Leave a comment

Filed under General, Uncategorized

Beware The Wolf In Supplier’s Clothing

As you all know, the economic climate of 2009 is a cold, cold winter indeed.  And like wolves starved by the cold and hardship of the season, our suppliers have turned feral.

When everyone’s sales slip due to the down economy, companies (and individual sales reps) are desperate to make their numbers.  How are they doing it?  By trying to jack up maintenance costs, in some cases by more than 100%!  It’s way more than isolated incidents; all our maintenance renewals coming up are meeting with hugely inflated quotes.  And not fly-by-night companies either, I don’t want to name names but let’s just say I am confident everyone out there has heard of all of them.

So protect yourself.  In your dealings with your supplier reps, start making it clear way ahead of time that your economic situation sucks too and you certainly expect that there’s a price freeze in place.  Don’t put up with it either – they know they’re going to make plenty of money off all the goons they send quotes to who will just rubber-stamp it and send it on so they can return to ESPNZone (I’m looking at you, State of Texas).  If you put up enough resistance they’ll go looking for easier pickings, just like those mean ol’ wolves do.    We had one outfit that wanted to jack up our maintenance cost by $125k a year, but luckily our IT director is a firm lady who has no problems with browbeating a sales rep until he cries.  In the end, we let them have a 5% increase because we ended up feeling sorry for them.

And have a backup plan.  If they really do have you over a barrel, then you’re low on leverage – you can try offering reference calls, presenting at conferences, and other handy non-cash incentives to them.  But when it comes down to it, you need to be able to walk away from them.  And to do this you need to plan ahead.  There are very few things that there’s only one of.  Have multiple suppliers lined up, and have a plan to change hardware or software if you have to.  Also look into open source, or third party support – even if it’s “not as good,” these days you have to decide how much good is worth how much money.

Now don’t get me wrong, we like to partner with our suppliers and treat them friendly.  Win-win and all that.  But good fences build good neighbors, and there’s nothing friendly about showing up and saying  “Hey, your operations will grind to a halt without our product, so stick ’em up and give me double this year!”

Be advised, that gleam in Bob the Sales Rep’s eyes will be a little hungrier than usual these days, and he’s gotta eat one of God’s little forest creatures to live.  Just make sure it’s not you.

1 Comment

Filed under General, Uncategorized

Google Chrome Hates You (Error 320)

The 1.0 release of Google Chrome has everyone abuzz.  Here at NI, loads of people are adopting it.  Shortly after it went gold, we started to hear from users that they were having problems with our internal collaboration solution, based on the Atlassian Confluence wiki product.  They’d hit a page and get a terse error, which if you clicked on “More Details” you got the slightly more helpful, or at least Googleable, string  “Error 320 (net::ERR_INVALID_RESPONSE): Unknown error.”

At first, it seemed like if people reloaded or cleared cache the problem went away.  It turned out this wasn’t true – we have two load balanced servers in a cluster serving this site.  One server worked in Chrome and the other didn’t; reloading or otherwise breaking persistence just got you the working server for a time.  But both servers worked perfectly in IE and Firefox (every version we have lying around).

So we started researching.  Both servers were as identical as we could make them.  Was it a Confluence bug?  No, we have phpBB on both servers and it showed the same behavior – so it looked like an Apache level problem.

Sure enough, I looked in the logs.  The error didn’t generate an Apache error, it was still considered a 200 OK response, but when I compared the log strings the box that Chrome was erroring on showed that the cookie wasn’t being passed up; that field was blank (it was populated with the cookie value on the other box and on both boxes when hit in IE/Firefox).  Both boxes had an identically compiled Apache 2.0.61.  I diffed all the config files- except for boxname and IP, no difference.  The problem persisted for more than a week.

We did a graceful Apache restart for kicks – no effect.  Desperate, we did a full Apache stop/start – and the problem disappeared!  Not sure for how long.  If it recurs, I’ll take a packet trace and see if Chrome is just not sending the cookie, or sending it partially, or sending it and it’s Apache jacking up…  But it’s strange there would be an Apache-end problem that only Chrome would experience.

I see a number of posts out there in the wide world about this issue; people have seen this Chrome behavior in YouTube, Lycos, etc.  Mostly they think that reloading/clearing cache fixes it but I suspect that those services also have large load balanced clusters, and by luck of the draw they’re just getting a “good” one.

Any other server admins out there having Chrome issues, and can confirm this?  I’d be real interested in knowing what Web servers/versions it’s affecting.  And a packet trace of a “bad” hit would probably show the root cause.  I suspect for some reason Chrome is partially sending the cookie or whatnot, choking the hit.


Filed under General, Uncategorized

Vignette Village 2008

Vignette, the Austin-based Web content management company,  has an annual show called Vignette Village.  A whole crew went from our company; Mark and I represented the Web Admins.

I got a lot out of Village though I wasn’t expecting to.  There was excitement in the air and clear commitment to continued development of their core Vignette Content Management (VCM V7) product and other products which had been lacking for the last couple years.  To be honest, I had begun to expect that it was a matter of time unti the Plone/Drupal/Joomla crowd outstripped VCM, but they seem to be making the changes required to keep the product as the true enterprise choice.  We already moved off Vignette Dialog, which was a very good email marketing package, because of the lackof support and new development.  I don’t know the details, but basically Vignette went all meathead and turned away from their core products to chase medical/legal document management money a couple years ago, combined with financial problems and layoffs, and so the products started to suck.  They seem to have turned that around, though, and everyone I spoke to inside Vignette is excited about their new leadership, especially Bertrand de Coatpont, the new VCM product manager.

The new Vignette Recommendations (OEMed from Baynote Systems) looks really good, and will expose some new data to us that I think can be used in a lot of different and innovative ways.  From previous descriptions I had thought “Yeah, whatever, BazaarVoice but from Vignette, which doesn’t necessarily inspire confidence in me” (frankly, we Web Admins have learned to be suspicious about additional offerings from Oracle, Vignette, HP, etc. as they will try to sell you crap on the strength of their brand name and alleged integration).  But the reality, which is an extremely elegant way of collecting and immediately reusing usage info, is brilliant and especially with their social search aspect to it, I feel like they have an actual vision they’re working towards.  So two thumbs up there!

Also two thumbs up on the Transfer Tool, which allows you to easily clone VCM installs to other servers – it’ll allow for frequent and efficient refreshes.  We had to have that working, so we Web Admins had devised a complicated two-day process to clone an environment; this should be much better.

VCM 7.6 is planned to be complete this year, and it has a lot of compelling features – you can migrate Content Type Definitions (change a CTD and the content changes inside the VCM to fit), lots of performance, availability, and console GUI fixes…  Then “Ace,” which everyone knows is VCM V8 but they don’t want to own up to that yet, has a total GUI overhaul.  Most of the issues we have with VCM are content contributor usability, so that’s great.

All in all, two days well spent.  It definitely exceeded my expectations (and I’ve been to Village in years previous).


Filed under General, Uncategorized

Our Search Implementation In The News!

InformationWeek did a big story on enterprise search, and used NI as their lead example!  Note all the system info in the article that I fed them. And we’re getting a lot of fun out of Graff’s quote about how it’s easy to sign off on more resources forus, we’re including that in every purchase req now. 🙂

One of the reasons that our FAST enterprise search program has been so successful here is that the programmers and the Web Admins have worked pretty much 50/50 on the platform.  Also, FAST is a great product and has great support (we’re waiting with bated breath to see if Microsoft screws it up; we’ve been with FAST since way way before they got bought), and we have some very visionary search business folks who saw its potential early on.

Nowadays, search is more than it was considered traditionally.  We have a normal “search box”, of course.   But we also run our faceted navigation off search (e.g. our Data Acquisition product line page), pull things like related links and other resources (see resources tab on this page).  Search, in many ways, can be used the way people have used databases in the past.  With some metadata added, a search index is kinda like a big database, highly denormalized for speed, focusing on text search.  In fact, I think there’s a master’s thesis in there somewhere as to when search makes sense vs. when a database makes sense.  Databases make sense with lots of numerical information, but on the Web that’s frankly a fringe use case!   On the Web it’s all about text, from name/address to links to articles to product info…  When we did things like query related links out of a database table, and I mean an oracle database table on a big ass Solaris box, it was painfully slow.  Pulling from search, it’s 15 milliseconds.

As a result, our internal search use is even more killer.  We pull Intranet pages, documents from Notes repositories, data from our Oracle ERP system, files off file shares, etc. all into one place and let people delve through it.  They’ve even implemented “screens” on top of some of the data (mainly because Oracle ERP is painful to use).  Our entire sales force is gaga over it.

Anyway, so yay to modern search technology, yay to FAST, and yay us!

1 Comment

Filed under General, Uncategorized

No No, You Really DO Want To Use Live Search

It’s been in the news that Microsoft is pushing “rewards programs” for people to use Live Search and the Live Toolbar.  But did you know they’re trying to get your local IT department to do it for you?

Yep, the program’s called the “Search@Work Rewards Program”.  If your IT department puts IE, with Live Search as the default search, and the Live Toolbar installed, and some kind of tracker plugin called the “Search Rewards Client,” on your company PCs, then they get Microsoft service credits!  Yay.  I can only assume my ISP is next.

Here’s the exact service description from Microsoft.  Note that they’re tracking Yahoo and Google ad impressions too!  The rest of it’s “fair enough” at least by usual IT industry standards but that’s kinda shady I think.

Continue reading

Leave a comment

Filed under General, Uncategorized

Cloud Headaches?

The industry is abuzz with people who are freaked out about the outages that Amazon and other cloud vendors have had.  “Amazon S3 Crash Raises Doubts Among Cloud Customers,” says InformationWeek!

This is because people are going into cloud computing with retardedly high expectations.  This year at Velocity, Interop, etc. I’ve seen people just totally in love with cloud computing – Amazon’s specifically but in general as well.  And it’s a good concept for certain applications.  However, it is a computing system just like every other computing system devised previously by man.  And it has, and will have, problems.

Whether you are using in house systems, or a SaaS vendor, or building “in the cloud,” you have the same general concerns.  Am I monitoring my systems?  What is my SLA?  What is my recourse if my system is not hitting it?  What’s my DR plan?

SaaS is a special case of cloud computing in general.  And if you’re a company relying on it, when you contract with a SaaS vendor you get SLAs established and figure out what the remedy is if they breach it.  If you are going into a relationship where you are just paying money for a cloud VM, storage, etc. and there is no enforceable SLA in the relationship, then you need to build the risk of likely and unremediable outages into your business plan.

I hate to break it to you, but the IT people working at Amazon, Google, etc. are not all that smarter than the IT people working with you.  So an unjustified faith in a SaaS or cloud vendor – “Oh, it’s Amazon, I’m sure they’ll never have an outage of any sort – either across their entire system or localized to my part of it – and if they do I’m sure the $100/month I’m paying them will cause them to give a damn about me” – is an unreasonable expectation on its face.

Clouds and cloud vendors are a good innovation.  But they’re like every other computing innovation and vendor selling it to you.  They’ll have bugs and failures.  But treating them as if they won’t is a failure on your part, not theirs.


Filed under Cloud, Uncategorized