Category Archives: General

Random musings on technology or management or whatever.

Chaos In The Time Of Covid

“See, here I’m now sitting by myself, uh, er, talking to myself. That’s, that’s chaos theory.”
– Jeff Goldblum as Ian Malcom, Jurassic Park

Hello all!  I know it’s been a while between posts here at the Agile Admin. Everyone’s safe and healthy, though. We’ve been quiet but busy, and in a twist of happy fate many of us are working together again!

James, Karthik, Bill, and Ernest are all now at Verica, a chaos engineering startup founded by Casey Rosenthal and Aaron Rinehart. It’s 100% remote, with people from Portland to Portland, as we like to say (Maine to Oregon). We’re still working on the initial release of the product; my team (which includes Karthik and Bill) is working on the Kubernetes module.  Really interesting stuff!  Casey literally co-wrote the book on chaos engineering and it’s a fascinating bleeding edge part of systems engineering.  If we can lure Peco over to join us then we’ll have the entire band back together!

Community stuff has been quiet.  We had to fight to get our money back from the venue from cancelling DevOpsDays Austin 2020, but we finally prevailed there.  The CloudAustin user group is meeting virtually.  And James, Karthik, and I are running the Modern Infrastructure track for All Day DevOps in November. Karthik’s speaking at KubeCon in August. We’re doing light work on some LinkedIn Learning courses, though I have to admit with the additional burden of the pandemic I haven’t been getting a lot of side stuff done personally.

As we get on our feet with chaos experimentation and this dang lockdown eases up we hope to be back with our usual mix of technical insight and Texan cussedness!  Because let’s be honest, Twitter is where you go to get dumber.

Leave a comment

Filed under General

The Agile Admin, Now Also On Tumblr

For all the under-18 DevOps fans out there, The Agile Admin is now available via Tumblr too at http://theagileadmin.tumblr.com/! Get your parents’ permission…

Leave a comment

Filed under General

Steal This Teambuilder: Meat, Guns, and Booze

I thought I’d take a minute for a teambuilding tip from Management Corner and share a winning teambuilder from here in Austin that I’ve used a lot and has always been a solid success. We call it “Meat, Guns, and Booze.” Steal it and use it with your teams!

Coming up with good teambuilders for an engineering team is pretty difficult.  You have a mix of introverts and extroverts. Going to a movie doesn’t really allow for much personal interaction.  A group meal lets you talk to the 3-4 people next to you, which in large groups isn’t a large percentage, and even getting everyone seated together can be a challenge.  Not everyone drinks and may not appreciate hanging out at a bar. Messing around in the office is fine but sometimes you want to get out of there so that people aren’t continually distracted by work. Not everyone is active enough for something like paintball (especially in Texas weather!). Well, back some years ago while I was at National Instruments we were mulling over what a good field trip/teambuilder could be given all these limitations, and happened across a winning combination that never fails to deliver!

Meat

BBQ At Smitty's 4

Meat Frenzy

First, head out to Lockhart, about an hour from most places in Austin, for a lunch of barbeque. Austin BBQ is good, but for the truly top tier stuff you have to go out to other locations (we love Franklin’s/LA Barbecue but going and standing in line at 10 AM to get some isn’t really a team activity). You can have internal debates over whether you go to Black’s, Smitty’s, or Kreutz’; try them all out over time as everyone will passionately defend the one they think is best. For vegetarians, there’s good sides, but also there’s a cafe right across the street from Smitty’s Market with salads and whatnot. Smitty’s also has a “no utensils” rule; eating meat with your hands definitely switches the mood to informal and friendly quickly!

Pro Tip: Bring cash. Some of them – Smitty’s for sure IIRC – don’t take credit!

Blacks BBQ

Goodsprings from Fallout: New Vegas?

Besides the great BBQ, many folks haven’t seen old style rural Texas – even folks living in Austin, but especially people from other countries. I remember once we were headed into town, and it has a traditional town square with raised boardwalks and all – and the two guys from Ukraine in the back seat started wriggling around like itchy bear cubs and talking rapidly to each other in Ukranian.  “What’s up?” I asked from the driver’s seat.  “It looks just like Fallout!” they breathlessly exclaimed. I had to laugh. It’s worth a brief stroll around the town square and courthouse. There’s a western-stuff store for the greenhorns, too.

Guns

IMG_0781

My .30/.30

IMG_0775

A Ukranian Engineer Learns About Texas

If it’s raining, you can fall back to Red’s in Austin, they have several locations.  But they’re small and cramped for groups. Right near Lockhart is the Lone Star Gun Range. They have rifle and pistol ranges, outdoor but with shade.  You just need a driver’s license (or passport for the foreigners) and you can rent anything from a revolver to an AK-47 for a reasonable fee – you mainly end up paying for ammunition. (You can bring your own guns and ammo, but have to buy their ammo to use in their rental guns.) They give a quick safety briefing and off you go.  As long as you have one or two more gun-savvy people to help folks who are having trouble figuring out how to reload or whatnot, you’re good to go; I’ve been there many times with groups of 10-20 composed of mostly noobs with no problems. And don’t worry – everyone gets into it.  I’ve had people who were initially intimidated but once they get hands on for a minute they always take to it with gusto.

Even in Texas, gun owners usually don’t go out to the range all the time, so it’s a treat even for those who are more into it, and a novelty for those who aren’t.

Pro Tip: Don’t drive a super-nice car – the road to the place is not paved, it’s shell.  If you have a sweet hand-waxed Porsche you should ask someone else to drive. Also, it’s outside in Texas in (whatever season) – bring sunscreen, hats, and a cooler full of water. There is an air-conditioned store with a bathroom and all if someone gets overwhelmed by the heat.

Steve on the Barrett 1

The Barrett

Sad Note: They used to have a Barrett .50 out there at Lone Star but they sold it.  The rounds were super expensive but no one wanted to fire more than 1 or 2 anyway! But they have revolvers, automatics, rifles from .22 to ARs to AKs…

Booze

After shooting, you’re usually hot, and so stopping in at a watering hole on your way back to Austin is a good thing to do.  Anywhere that’s convenient for you is good – I’ve done the Whip In as that’s uber Austiny, has an outdoor patio with picnic tables for good social mobility, and has a lot besides just beer (food, wide non-alcoholic drink variety). You got out of the office (and out of town), had the team nourish itself on food, then have an exciting activity, and now they get to interact with the ice already well broken. People don’t drink a lot because it’s already been a long day, as opposed to a “let’s go drink” teambuilder where people are more tempted to overindulge. But they have plenty to talk about. Non-drinkers – or if your entire company doesn’t like sponsoring events with alcohol – can just drink sodas and have an appetizer or two to unwind.

And yes, to all the wags out there who inevitably respond with “but don’t mix up the order!”, you do want to do it in this order – there’s the wisdom of no drinking before gun handling, but also because this sequence of events is carefully constructed to provide a cadence of icebreaking and interaction. You “break bread” together, which is a universal signal of togetherness, and sit by some people to eat, by others in the car, interact with others while doing a stimulating activity, unwind with others after, etc. And since it’s multipart, if someone really does have to come late or leave early there are multiple natural breakpoints. We also like doing three-round K1 racing as teambuilders but that’s more disruptive if people come and go.

I’ve used this teambuilder with multiple teams, multiple companies, and it’s always a hit. It’s especially a hit with international teams – folks from Malaysia, Ukraine, etc. see all of this as a special treat; it’s very common to talk to them a year later about their trip to Austin and they’ll immediately volunteer that this was their favorite part of their visit hands down. It’s relatively inexpensive and just takes a long afternoon (we usually leave at 11 and start wrapping up at 5). It’s maybe $1000 for a full day of fun for a dozen people.

Not In Austin?

Well of course the luridly titled “Meat, Guns, and Booze” specifically evokes Texas and is tuned to our location.  But consider the same kind of event cadence and think about what’s distinctive in your area. You just want the team to share a meal, then participate in an activity that everyone can easily participate in, and then have an opportunity to relax and talk with each other afterwards.

4 Comments

Filed under General

Is It a Bug Or A Feature? Who Cares?

Today I’ve been treated to the about 1000th hour of my life debating whether something someone wants is a “bug” or a “feature.”  This is especially aggravating because in most of these contexts where it’s being debated, there is no meaningful difference.

A feature, or bug, or, God forbid, an “enhancement” or other middle road option, is simply a difference between the product you have and the product you want. People try to declare something a “bug” because they think that should justify a faster fix, but it doesn’t and it shouldn’t. I’ve seen so many gyrations of trying to qualify something as a bug. Is it a bug because the implementation differs from the (likely quite limited and incomplete) spec or requirements presented?  Is it a bug because it doesn’t meet client expectation?

In a backlog, work items should be prioritized based on their value.  There’s bugs that are important to fix first and bugs it’s important to fix never.  There’s features it’s important to have soon and features it’s important to have never.  You need (and your product people) need to be able to reconcile the cost/benefit/risk/etc across any needed change and to single stack-rank prioritize them for work in that order regardless of the imputed “type” of work it is.  This is Lean/Agile 101.

Now, something being a bug is important from an internal point of view, because it exposes issues you may have with your problem definition, or coding, or QA processes. But from a “when do we fix it” point of view, it should have absolutely no relation. Fixing a bug first because it’s “wrong” is some kind of confused version of punishment theory. If you’re distinguishing between the two meaningfully in prioritization, it’s just a fancy way of saying you like to throw good money after bad without analysis.

So stop wasting your life arguing and philosophizing about whether something in your backlog is a bug or enhancement or feature.  It’s a meaningless distinction, what matters is the value that change will convey to your users and the effort it will take to perform it.

I’m not saying one shouldn’t fix bugs – no one likes a buggy product.  But you should always clearly align on doing the highest leverage work first, and if that’s a bug that’s great but if it’s not, that’s great too.  What label you hang on the work doesn’t alter the value of the work, and you should be able to evaluate value, or else what are you even doing?

We have a process for my product team – if you want something that’s going to take more than a week of engineer time, it needs justification and to be prioritized amongst all the other things the other stakeholders want.  Is it a feature?  A bug?  A week worth of manual labor shepherding some manual process? It doesn’t matter.  It’s all work consuming my high value engineers, and we should be doing the highest value work first.  It’s a simple principle, but one that people manage to obscure all too often.

8 Comments

Filed under General

Want To Write For The Agile Admin?

At Velocity and DevOpsDays this week I had a couple people ask about contributing articles to the blog – yes, we’d love to have anyone write on cloud, DevOps, etc. Many folks don’t want to maintain a whole blog themselves (or do and want to crosspromote it, I guess!) so we’re happy to have you here, just email me!

2 Comments

Filed under General

Breaking the Silence – Agile Admin Updates!

Well I know it’s been quiet around here lately.  Two of the agile admins, myself and Peco, have switched jobs and have been crazy busy.

Peco has moved to Opnet as a SE – he always loved their APM tool Panorama and we got a lot of mileage out of it at NI. Notably, this included getting some of our suppliers, like Vignette, to buy it and use it on their products to find the performance problems before we did… We were having Vignette V7 performance issues and used Panorama to identify exactly what they were, and when we fed it back to Vignette engineering they were like “how the hell did you do that…” Opnet’s never had great marketing, they’ve been doing what New Relic and AppDynamics have been doing for a long time but aren’t as much of a recognized name. It’s an interesting move for Peco, he went from long time Ops guy to a Java dev, working on system automation like PIE, and now an SE because he wants trigger time on customer interaction.  He’s a renaissance man!  And working on his own stealth startup on the side of course.

I have moved to Bazaarvoice, a SaaS startup in Austin (well, are you still a startup when  you’re up to 400 people?) to be their Manager of Release Engineering.  They have been increasing their dev and ops forces by a very large margin (P.S. We’re hiring!) and wanted someone to run the “middle third” of their DevOps teams.  There’s one team of DevOps embedded into all the product teams, kind of a matrixed organization; then there’s my team to do the build and deploy automation and cloud and data center stuff; then there’s a support team to wrangle the pages and tickets.

I was there for a week doing new hire training when they said “So we have moved to agile doing two week sprints, and that’s going OK except we realized we can’t release every two weeks safely. Get that going.  We start biweekly releases in three weeks. Zero customer impact each time!” Therefore I’ve been in a little bit of a frenzy doing that. It was definitely on the fine line between “I like a challenge” and “Oh shit.” I lucked out and got a week of slack because we decided to IPO on the date that the first release was supposed to go, and we decided that doing both on the same day was just a wee bit too ambitious. The first biweekly release slipped from a Thursday till the next Tuesday and only had two minor issues that needed fixing after, and the next one is coming up!  We should have it tamped down to regular in a sprint or two and I hope to return to more regular blogging.

On the side of course I continue to help run the Austin Cloud User Group and the Agile Austin DevOps SIG and put together DevOpsDays Austin and do other random stuff, so sadly when the overcommittedness hits the blog has to give.

James is doing great, and just had a new baby!  So he’s been out of pocket some too. He’s still at NI, and now large and in charge of all operations for their SaaS products. They are fighting the brave fight to get PIE open sourced and out the door.

We are all going to SXSW Interactive this weekend and will regroup and decide how we’re going to bring you all the latest in hot cloud/DevOps/tech stuff going forward!

1 Comment

Filed under General

A Brief Hiatus

Sorry all, it’s been a crazy busy month around here.  But the Agile Admin is back, and we’ll be talking about the latest fun stuff in DevOps, cloud, and cloud security on a regular basis!

Some fun developments:

  • We delivered another NI SaaS product, the 1.0 version of the FPGA Compile Cloud
  • We’re working hard on another, the Technical Data Cloud
  • NI just hired Lars Ewe, former CTO of Cenzic, as a security guru, and he sits over here with our group.
  • Agile Austin has started a DevOps SIG
  • Work towards open sourcing PIE, our Programmable Infrastructure Environment, proceeds apace!

So we’ve been busy, but I know we need to take more time to discuss and engage with everyone else out there, so it’s back to posting!  Hope to see you all in the comments.

2 Comments

Filed under General

Report from NIWeek

Hey all, sorry it’s been quiet around here – Peco and I took our families on vacation to Bulgaria!  Plus, we’ve been busy in the run-up to our company convention, NIWeek. I imagine most of the Web type folks out there don’t know about NIWeek, but it’s where scientists and engineers who use our products come to learn. It’s always awesome to see the technology innovation going on out there, from the Stormchasers getting data on tornadoes and lightning that no one ever has before, to high school kids solving real problems.

There were a couple things that are really worth checking out.  The first is the demo David Fuller did of NI’s system designer prototype (you can skip ahead to 5:00 in if you want to) . Though the examples he is using is of engineering type systems, you can easily imagine using that same interface for designing Web systems – no ‘separate Visio diagram’ BS any more. Imagine every level from architectural diagram to physical system representation to the real running code all being part of one integrated drill-down. It looks SUPER SWEET. Seems like science fiction to those of us IT-types.

A quick guide to the demo – so first a Xilinx guy talks about their new ARM-based chip, and then David shows drill-up and down to the real hardware parts of a system.  NI now has the “traditional systems” problem in that people buy hardware, buy software, and are turning it into large distributed scalable architectures.  Not being hobbled by preconceptions of how that should be done, our system diagram team has come up with a sweet visualization where you can swap between architecture view (8:30 in), actual pictures and specs of hardware, then down (10:40 in) into the “implementation” box-and-line system and network diagram, and then down into the code (12:00 in for VHDL and 13:20 in for LabVIEW). LabVIEW code is natively graphical, so in the final drilldown he also shows programming using drawing/gestures.

Why have twenty years of “systems management” and design tools from IBM/HP/etc not given us anything near this awesome for other systems?  I don’t know, but it’s high time. We led a session at DevOpsDays about diagramming systems, and “I make a Visio on the side” is state of the art.  There was one guy who took the time to make awesome UML models, but real integration of design/diagram to real system doesn’t exist. And it needs to. And not in some labor intensive  “How about UML oh Lord I pooped myself” kind of way, but an easy and integral part of building the system.

I am really enjoying working in the joint engineering/IT world.  There’s some things IT technology has figured out that engineering technology is just starting to bumble into (security, for example, and Web services). But there are a lot of things that engineering does that IT efforts look like the work of a bumbling child next to. Like instrumentation and monitoring, the IT state of the art is vomitous when placed next to real engineering data metric gathering (and analysis, and visualization) techniques.  Will system design also be revolutionized from that quarter?

The other cool takeaway was how cloud is gaining some foothold in the engineering space.  I was impressed as hell with Maintainable Software, the only proper Web 3.0 company in attendance. Awesome SaaS product, and I talked with the guys for a long time and they are doing all the cool DevOps stuff – automated provisioning, continuous deployment, feature knobs, all that Etsy/Facebook kind of whizbang shit. They’re like what I want our team here to become, and it was great meeting someone in our space who is doing all that – I love goofy social media apps or whatever but it can sometimes be hard to convey the appropriateness of some of those practices to our sector. “If it’s good enough to sell hand knitted tea cozies or try to hook up with old high school sweethearts, then certainly it’s good enough to use in your attempt to cure cancer!” anyway, Mike and Derek were great guys and it was nice to see that new kind of thinking making inroads into our sometimes too-traditional space.

Leave a comment

Filed under Conferences, General

Quora vs StackExchange

As a Web ops guy, I’ve used the Stack Exchange sites, especially Server Fault, a lot.  They’re the best place to go do technical Q&A without having to go immerse yourself in a specific piece of open source software and determine what bizarre way you’re supposed to get support on it (often a crufty forum, mailing list with bizarre culture, or an IRC channel).

However, they have started to, in my opinion, come apart at the seams. They started with the “holy trinity” of Stack Overflow (coders), Server Fault (admins), and Super User (users).  But lately they have expanded their scope to sites for non-techie areas, but have also started to fragment the technical areas.  So now if I have a question, I am confronted with separate communities for Server Fault, Linux & UNIX, Ubuntu, and more. Or even worse, Stack Exchange vs. “Programmers” vs language specific lists. This basically heavily segments the population and leads to the same problems that the weird little insular mailing lists have. It makes me use the SEs a lot less. I don’t want to have to somehow engage with 10 different communities to answer my everyday questions (and I sure as hell am not going to follow 10 to answer questions), so in my opinion they are cannibalizing their success and it will implode of its own weight and become no better than any Internet forum site.

Recently I started seeing tweets about Quora from @scobleizer, and it said stuff like “is it the future of blogging?” and was being pitched as some twitter-blog hybrid which of course caused me to ignore it.  But then it started getting a lot more activity and I thought I’d go check it out.  But if you go to the Quora page, you can’t see anything without logging in.  And of course if you log in with Twitter or Facebook it wants “everything from you and your friends, ever.”  So I wandered off again.

Finally I gave in and went over and logged in, and it’s actually pretty neat – it’s Q&A, like Stack Exchange, but instead of segmentation into different sub-communities, it uses the typical tag/follow/etc. Web 2.0 paradigm. So “Stack Exchange plus Twitter” is probably the best analogy. Now on the one hand that more unmanaged approach runs the risk of becoming like “Yahoo! Answers” – utter crap, full of unanswered questions and spammers and psychos – but on the other hand, I like my topics not being pushed down into little boxes where you can’t get an answer without mastering the arcane rules of that community (like the hateful Cygwin mailing list, where the majority of new posters are chased off with bizarre acronyms telling them they are using email wrong).  The simple addition of up/down voting is 80% of the value of what SE gives over forums, so will that carry the day?

Now maybe it’s because they’re having capacity problems, but the biggest problem with Quora IMO is that you don’t get to see any of it when you go there until you log in and give them access to all your networks and whatnot, which I find obnoxious. But if they fix that, then I think given the harmful direction SE is going, it may be the next big answer for Q&A.

2 Comments

Filed under General

Oracle Declares War On Open Source

Some days, it doesn’t pay to be a Java shop.  Oracle has discontinued OpenSolaris and is suing Google over making a Java fork for their mobile phones.  Are OpenJDK, mySQL, and OpenOffice next on the destructive rampage?

We like using open source.  We also like coding our Web apps in Java; it’s heavier duty than PHP/Ruby and more open than .NET – or at least it was.  We’re actually a big Oracle shop – we use mySQL for our cloud offerings but use loads of Oracle databases (and Oracle ERP) internally.  But it’s hard to interpret this as anything other than a series of crappy moves that will result in diminishing the ecosystem we’re trying to use to create products and Web apps.

It seems like Larry Ellison is really fond of the old Microsoft “I am your monopolistic corporate overlord” kind of business relationship.  Warning – if we want one of those, we’d use Microsoft; they’re cheaper and better at it.

It’s partly the fault of the open source “companies” – they willingly sell out to the corporate-overlord set (Microsoft, Oracle, IBM, HP, CA) and then, no matter what they say, they’re not really open any more.  Java was allegedly open sourced by Sun but now Oracle is suing Google for exercising that open source license on the basis of patent infringement.  Heck, a lot of the open source companies have instead become “open core” which is often mostly a lie.

So is Open Source already dead, and this is just part of the feeding frenzy of the big boys scooping it up?

Leave a comment

Filed under General