Monthly Archives: March 2014

Amazon Cuts Prices Too

Well, if nothing else I’m happy to have Google Cloud around to provide some competition to push Amazon Web Services.  Immediately after Google announced dramatic price drops, Amazon has responded doing the same!

Now if they can only also shame them into dropping their whole crazy reserve instance scheme and go to progressive discounts like Google just did too, the world will be better.


by | March 27, 2014 · 7:10 am

Google Cloud Update

We had a little get-together here in Austin today, sponsored by MomentumSI and hosted at Capital Factory (thanks to both!), to view the Google Cloud Platform newest product announcement webcast. About 24 local engineers showed up to watch.

You can view the whole thing yourself here, or just read my notes from the event.

Cloud Is Hard

Their thesis statement was that cloud, while cool, is still too hard for many people, hindering adoption or slowing innovation. So they’ve worked on making it easier.


Cost calculation is super complex (reserve, on demand, etc.). They talk about “other industry standard clouds” by which they mean Amazon Web Services. They note the drawbacks to reserved instances, which I am all totally in agreement on (see my earlier article Why Amazon Reserve Instances Torment Me for more on that). Specifically they note that reservations constrain your design choices, which is one of the great reasons to go to the cloud in the first place – Amen, brother!

Though cloud prices have been dropping 6-8% a year, hardware’s been dropping 20-30%. Why is Moore’s Law not translating into more sweet green in our pockets? It should, they contend. Thus, they are announcing on demand price drops:

  • GCE 32% price drop
  • Storage is now .026 cents/GB for any use
  • .02 c/GB for reduced durability storage
  • bigquery 85% reduction
  • can now purchase predictable throughput

Introducing sustained use discounts – no pre-plan or reserving ahead of time, instead prices automatically drop as VM usage is sustained over 25% of the month and then progressively from there. 100% use is a 53% discount over current (remember that includes the new 32% reduction, so another 21% from current for continued use). With linear machine cost scaling, makes it simple(r) to predict and calculate your costs.

Other Tradeoffs

Current cloud (hint: AWS) forces other tradeoffs: time to market vs scalability, flexibility (iaas) vs automatic management (paas), big data vs realtime data analysis.

But first, we interrupt our messaging to talk about other random new features based on customer feedback. To wit:

  • SuSE/Red Hat support
  • Windows Server 2008 R2 (preview) support
  • Cloud DNS service, accessible via API and console

The features are nice but even nicer was that they implemented these based on customer feedback, which means they consider this a real product with real customers and not just a fun tech thing for their own ends (which to be fair 80% of Google’s offerings are, and it can be hard to tell the difference).

Time to Market vs Scalability

So on scaling… You need deployment! Troubleshooting! Use tools you know!
They have a new “gcloud” command line tool
“gcloud init” pulls down the app via git, you can just edit, git commit, git push
They have a build service integrated – it spins up a jenkins/maven and builds, deploys – you can see release status in the console.
There’s also a new unified logs viewer with basic searching – like Splunk junior, with one cool dev feature. Click on the code in the stack trace and you’re put directly into the code in the console’s source view. Fix and commit, it auto-builds, bam you’re fixed.

IaaS vs PaaS

A new halfway state – “managed VMs.” It’s the normal PaaS, but in the config, you can tell it things to apt-get install onto the instances, so you can have more third party software than the PaaS previously allowed.
Also, you can “enable debugging” on an instance and then log in interactively.

Big Data vs Realtime Data Analysis

They’ve upped BigQuery to have 100k rows/sec ingest.
Example Demo: smart monitoring of 60 events/hour from 400k glen canyon power meters (17bn events/mo), with about 128k records. They did a visualization that is updating in near real time showing all those meters geolocated and you can go click on them to get realtime data.
He showed the complex BigQuery “bigjoin” to filter by meter lat/long from sep table and then by quartile across whole population. “Doing this in NoSQL would be impossible or very slow.”

They will be doing a Google Cloud roadshow soon – see – it looks like Austin will be on the list of cities!


The good thing about getting a bunch of techies together to view this was the discussion afterwards.  The general sentiment was that:

1. The cost drops are nice and the approach to reserve/sustained use instances is much better. The reserve instance scheme is one of the worst things about AWS and if this drives them to adopt the same model, hooray!

2. The other new features (managed VMs, gcloud) are definitely nice. They are focusing on dev friendliness in their discussion but it’s a lot less clear how to operate this. If you’re really trying to stitch together a bunch of micro-services there’s not a lot of great support for that. They talk about using their PaaS and say “of course, if you use our PaaS you don’t need to carry a pager! You’d only need to do that if you’re doing IaaS and maintaining your own OSes.” That is dangerously naive and really made the whole group skittish. Most people there have done “play” things in Google’s cloud but are reticent to put mission critical items there, and this section of the presentation didn’t do a lot to improve that.

3. The BigQuery/realtime demo was impressive and multiple people would like to kick the tires on it.

Overall – it was a little light, but it was a keynote; the new features/pricing are all good; this shows more Google commitment to their cloud as a product but actual concerns still linger about maturity and suitability for realistically complex revenue-generating production applications.


Leave a comment

Filed under Cloud

DevOps Updated

Since I had some time and have been thinking about it lately, I’ve upgraded and expanded my definition of DevOps on theagileadmin. Healthy debate welcome!

Leave a comment

by | March 20, 2014 · 3:48 pm

DevOpsDays Austin Is Coming!

The third annual DevOpsDays conference in Austin will be May 5-6 (Cinco de Mayo!) at the Marchesa, where it was held last year! As many of you know, the DevOpsDays conferences are a super popular format – half talks from practitioners, half openspaces, all fun – held in many cities around the world since the first one in Ghent launched the DevOps movement proper.

  • You can register – all the early bird tickets are sold out but the regular ones are only half gone.
  • You can also propose a talk!  There’s 35-minute full talk slots but we’re even more in need of 5-minute Ignite! style lightning talks! RFP ends 3/26 sp
  • You can sponsor! The Gold sponsorships are half gone already. And we have some special options this year…

DevOpsDays Austin has been bigger and better every year since its inception and should have something good for everyone this year. Come out and join your comrades from the trenches who are trying to forge a new way of delivering and maintaining software!

1 Comment

Filed under Conferences, DevOps

Agile Austin asked me to help re-launch their blog, so I’ve contributed a piece on “What Is DevOps?” for them!

Leave a comment

by | March 16, 2014 · 9:58 am

Steal This Teambuilder: Meat, Guns, and Booze

I thought I’d take a minute for a teambuilding tip from Management Corner and share a winning teambuilder from here in Austin that I’ve used a lot and has always been a solid success. We call it “Meat, Guns, and Booze.” Steal it and use it with your teams!

Coming up with good teambuilders for an engineering team is pretty difficult.  You have a mix of introverts and extroverts. Going to a movie doesn’t really allow for much personal interaction.  A group meal lets you talk to the 3-4 people next to you, which in large groups isn’t a large percentage, and even getting everyone seated together can be a challenge.  Not everyone drinks and may not appreciate hanging out at a bar. Messing around in the office is fine but sometimes you want to get out of there so that people aren’t continually distracted by work. Not everyone is active enough for something like paintball (especially in Texas weather!). Well, back some years ago while I was at National Instruments we were mulling over what a good field trip/teambuilder could be given all these limitations, and happened across a winning combination that never fails to deliver!


BBQ At Smitty's 4

Meat Frenzy

First, head out to Lockhart, about an hour from most places in Austin, for a lunch of barbeque. Austin BBQ is good, but for the truly top tier stuff you have to go out to other locations (we love Franklin’s/LA Barbecue but going and standing in line at 10 AM to get some isn’t really a team activity). You can have internal debates over whether you go to Black’s, Smitty’s, or Kreutz’; try them all out over time as everyone will passionately defend the one they think is best. For vegetarians, there’s good sides, but also there’s a cafe right across the street from Smitty’s Market with salads and whatnot. Smitty’s also has a “no utensils” rule; eating meat with your hands definitely switches the mood to informal and friendly quickly!

Pro Tip: Bring cash. Some of them – Smitty’s for sure IIRC – don’t take credit!

Blacks BBQ

Goodsprings from Fallout: New Vegas?

Besides the great BBQ, many folks haven’t seen old style rural Texas – even folks living in Austin, but especially people from other countries. I remember once we were headed into town, and it has a traditional town square with raised boardwalks and all – and the two guys from Ukraine in the back seat started wriggling around like itchy bear cubs and talking rapidly to each other in Ukranian.  “What’s up?” I asked from the driver’s seat.  “It looks just like Fallout!” they breathlessly exclaimed. I had to laugh. It’s worth a brief stroll around the town square and courthouse. There’s a western-stuff store for the greenhorns, too.



My .30/.30


A Ukranian Engineer Learns About Texas

If it’s raining, you can fall back to Red’s in Austin, they have several locations.  But they’re small and cramped for groups. Right near Lockhart is the Lone Star Gun Range. They have rifle and pistol ranges, outdoor but with shade.  You just need a driver’s license (or passport for the foreigners) and you can rent anything from a revolver to an AK-47 for a reasonable fee – you mainly end up paying for ammunition. (You can bring your own guns and ammo, but have to buy their ammo to use in their rental guns.) They give a quick safety briefing and off you go.  As long as you have one or two more gun-savvy people to help folks who are having trouble figuring out how to reload or whatnot, you’re good to go; I’ve been there many times with groups of 10-20 composed of mostly noobs with no problems. And don’t worry – everyone gets into it.  I’ve had people who were initially intimidated but once they get hands on for a minute they always take to it with gusto.

Even in Texas, gun owners usually don’t go out to the range all the time, so it’s a treat even for those who are more into it, and a novelty for those who aren’t.

Pro Tip: Don’t drive a super-nice car – the road to the place is not paved, it’s shell.  If you have a sweet hand-waxed Porsche you should ask someone else to drive. Also, it’s outside in Texas in (whatever season) – bring sunscreen, hats, and a cooler full of water. There is an air-conditioned store with a bathroom and all if someone gets overwhelmed by the heat.

Steve on the Barrett 1

The Barrett

Sad Note: They used to have a Barrett .50 out there at Lone Star but they sold it.  The rounds were super expensive but no one wanted to fire more than 1 or 2 anyway! But they have revolvers, automatics, rifles from .22 to ARs to AKs…


After shooting, you’re usually hot, and so stopping in at a watering hole on your way back to Austin is a good thing to do.  Anywhere that’s convenient for you is good – I’ve done the Whip In as that’s uber Austiny, has an outdoor patio with picnic tables for good social mobility, and has a lot besides just beer (food, wide non-alcoholic drink variety). You got out of the office (and out of town), had the team nourish itself on food, then have an exciting activity, and now they get to interact with the ice already well broken. People don’t drink a lot because it’s already been a long day, as opposed to a “let’s go drink” teambuilder where people are more tempted to overindulge. But they have plenty to talk about. Non-drinkers – or if your entire company doesn’t like sponsoring events with alcohol – can just drink sodas and have an appetizer or two to unwind.

And yes, to all the wags out there who inevitably respond with “but don’t mix up the order!”, you do want to do it in this order – there’s the wisdom of no drinking before gun handling, but also because this sequence of events is carefully constructed to provide a cadence of icebreaking and interaction. You “break bread” together, which is a universal signal of togetherness, and sit by some people to eat, by others in the car, interact with others while doing a stimulating activity, unwind with others after, etc. And since it’s multipart, if someone really does have to come late or leave early there are multiple natural breakpoints. We also like doing three-round K1 racing as teambuilders but that’s more disruptive if people come and go.

I’ve used this teambuilder with multiple teams, multiple companies, and it’s always a hit. It’s especially a hit with international teams – folks from Malaysia, Ukraine, etc. see all of this as a special treat; it’s very common to talk to them a year later about their trip to Austin and they’ll immediately volunteer that this was their favorite part of their visit hands down. It’s relatively inexpensive and just takes a long afternoon (we usually leave at 11 and start wrapping up at 5). It’s maybe $1000 for a full day of fun for a dozen people.

Not In Austin?

Well of course the luridly titled “Meat, Guns, and Booze” specifically evokes Texas and is tuned to our location.  But consider the same kind of event cadence and think about what’s distinctive in your area. You just want the team to share a meal, then participate in an activity that everyone can easily participate in, and then have an opportunity to relax and talk with each other afterwards.


Filed under General

Agile Organization Incorporating Various Disciplines

I started thinking about this recently because there was an Agile Austin QA SIG meeting that I sadly couldn’t attend entitled “How does a QA manager fit into an Agile organization?” which wondered about how to fit members of other disciplines (in this case QA) in with agile teams. Over the last couple years I’ve tried this, and seen it tried, in several ways with DevOps, QA, Product Management, and other disciplines, and I thought I’d elaborate on the pros and cons of some of these approaches.

Two Fundamental Discoveries

There are two things I’ve learned from this process that are pretty universal in terms of their truth.

1. Conway’s Law is true. To summarize, it states that a product will tend to reflect the structure of the organization that produces it. The corollary is that if your organization has divisions which are of no practical value to the product’s consumer, you will be creating striations within your product that impact client satisfaction. Hence basic ITSM and Agile doctrines on creating teams around owning a service/product.

2. People want to form teams and stay with them. This should be obvious from basic psychology/sociology, but if you set up an organization that is too flexible it strongly degrades the morale of your workers. In my previous role we conducted frequent engineer satisfaction surveys and the most prominent truth drawn from them is the more frequently people are asked to change roles, reorg, move to different teams, the less happy they are. Even people that want to move around to new challenges frequently are happier if they are moving to those new challenges with a team they’ve had an opportunity to move through Tuckman’s stages of group development with.

I have seen enough real-world quantified proof of both of these assertions that I will treat them as assumptions going forward.

Organizational Options

We tried out all four of these models within the same organization of high performing engineers and thus had a great opportunity to compare their results.

Separate Teams

When we started, we had the traditional model of separate teams which would hand off work to each other.  “Dev,” “Ops,” “QA,” “Product” were all under separate management up through several levels and operated as independent teams; individual affinities with specific products were emergent and simply matters of convenience (e.g. “Oh, he knows a lot about that BI stuff, let him handle that request”) and not a matter of being dedicated to specific product(s).

Embedded Crossfunctional Service Teams

Our first step away from the pure separate team model was to take those separate teams and embed specific members from them into service oriented teams, while still having them report to a manager or director representing their discipline. In some cases, the disciplinary teams would reserve some number of staff for tool development or other cross-cutting concerns. So QA, for example, had several engineers assigned to each product team, even though they were regarded as part of the permanent QA org primarily.

We (very loosely) considered our approach to be decentralized and microservice based; Martin Fowler is doing a good article in installments on Microservice Architecture if you want more on that topic.

Fully Integrated Service Teams

With our operations staff we went one step farther and simply permanently assigned them to product teams and removed the separate layer of management entirely. Dev and “DevOps” engineers reported to the same engineering manager and were a permanent part of a given product team. Any common tooling needed was created by a separate “platform” engineering team which was similarly integrated.

Project Based Organization

Due to the need to surge effort at times, we also had some organizations that were project, not product, based. Engineers would be pulled either from existing teams or entire teams would additionally be pulled into a short term (1, 2, 3 month) effort to try to make significant headway across multiple products, and then dissolve afterwards.

Hmm, this looks like it’s getting big (and I need to do some diagrams).  I’ll break it up into separate articles for each type of org and its pros and cons, and then a conclusion.


Filed under Agile, DevOps