Category Archives: DevOps

Pertaining to agile system administration concepts and techniques.

Java Docker Pull Travails

Just had a problem that I thought I’d document the solution to for the world…

In our build pipeline at work, we use maven and the fabric8 docker-maven-plugin to manage our builds.  We love it, developers can just “mvn install” locally and then the Atlassian Bamboo build system just “mvn deploy”s in the exact same way.

Well, so we had some builds that suddenly weren’t able to pull the base images specified in our Dockerfiles down from Dockerhub, breaking the build with 500 error messages like:

[ERROR] DOCKER> Unable to pull 'library/debian:sid' from registry '' : received unexpected HTTP status: 500 Server Error (Internal Server Error: 500) [received unexpected HTTP status: 500 Server Error (Internal Server Error: 500)]

But it worked fine on our local box. And it could pull our custom images from Artifactory fine. What’s the problem here?  Bamboo?  The plugin? Well, some helpful community folks helped home in on it, it turns out that for some versions of Java 1.8, 8u131 and prior at least going back to 112, where there’s some problem (TLS? Root certs? Not really sure) that messes up when pulling a container from inside Java during our docker build step.  My team’s microservices aren’t Java based so the Java version doesn’t come up much – but of course maven uses Java.

Upgrading the JDK version to 8u144 made the problem go away.  We actually have an up to date curated Java version we use in Bamboo for our Java builds, but folks doing Python builds were just using the default “JDK 1.8” that Atlassian is putting on their Bamboo build agent AMI, which is of course old and suffers from this issue.


Leave a comment

Filed under DevOps

Long live ChatOps, RIP AOL IM!

I grew up in Muscat, Oman, and it was an exciting time when we got Internet at home in 1996. By 1998, all of my friends who had Internet at home were first on ICQ and then on AOL IM. AOL IM was huge when I went to college in the early 2000’s and was the primary way to connect friends together to chat. Back then, it was rare to have chat rooms, and the rooms that existed were usually long-running things set up to talk about general topics.

The first time I saw value in a chat room in a professional setting was when I got invited to a Basecamp “deploy room” by fellow Agile Admin Peco (or was it Ernest?) at NI when our quarterly release cycle was going super poorly, and all of us (100 other people) were waiting around at hour #34 trying to figure out why some random enterprise application was holding up the rest of the release process. Post invitation to the room, I was able to look at the past messages between the ops team about application failures, and then realized pretty quickly that our databases weren’t actually responding like they should. It took all of 10 minutes to ask someone on the ops side with credentials to run a database query, and figure out that the db creds were all wrong. 2 hours later, the release was all done…

That moment made me realize that 1×1 chats were great, but having a persistent chat rooms with teams of people added value to an organization.

Recently, a colleague asked me a simple question that made me reflect. He asked, “What’s the big deal about Slack?”. At work, there’s been a big push to move towards Slack, when we’ve had 1×1 chat forever. Here are my 5 most compelling reasons for doing so:

1) Collaboration++: 15 years ago, software was a simpler, and there was no cloud/microservices. You’d have 1 large binary to deploy for a platform, and typically have a few folks who understood the overall workings of platform. Today, with microservices, you require a bunch of applications to deploy, and each of these have specific owners who understand specifics. Thus, you’re going to have to have conversations with multiple folks to figure out any issues. Having this in a room setting versus a 1×1 setting gets you to a resolution faster.

2) Chat metadata: Chat is less about words, and more about conversations that include images, links, slash commands, workflows etc. Chatops tools make pasting these much easier than before, and looking at formatted code in Slack is so much easier to read than looking at the same in pidgin.

3) Chat History: Chat apps now give you history – even from when you were not online or in the chat room. This is valuable from the perspective that you can see everything from when you weren’t around, and don’t have to ask someone to keep repeating the problem over and over again. You can just scroll up, read the context, and be ready to help if you can. This is my one knock against IRC (or at least the implementation of IRC at a company I worked at); it was nice to have everyone in a spot, but it only worked when we were VPN’ed in, and had no history.

4) Pipelining with chatbots: Continuous Integration/Delivery is all the rage these days! Having a chat system that allows for your devops systems to push data is a primary requirement in order to build a pipeline of this sort. Responses to broken builds, tests, alerts are quicker when the data associated with these are transmitted to a chatroom that you’re looking at, than having to look at Jenkins all the time. Chatbots are invaluable in this scenario, and help you with information flow.

5) The new normal: A new generation of engineers already do this. It’s already part of the culture for the next generation of engineers who work on open source (for example, kubernetes slack) and there’s even chatter about slack at Universities now. The world is evolving towards broader conversation, and not having chatops tools will hurt your company in terms of hiring and retention.


Agree/Disagree, or have a different perspective? Let me know by commenting below!


Filed under Agile, DevOps

Docker 101

Working at Stackengine, and now at Oracle, I’ve been working in the Docker ecosystem for the last 5 years!

While containerization has taken the IT and devops world by storm, a lot of larger enterprises might still be on the outside looking in. If you find yourself in that boat, you’re in luck!

Here’s a quick video on getting you running your very first Docker container on your Mac in under 5 minutes.

Also, I had the pleasure of traveling back to my childhood hometown of Bengaluru and presenting a workshop at Code Conf this year. I’ll create a separate post about my travels, but I got to present a workshop lab that is an Introduction to Containers. This lab is a perfect follow on to the video above, and will help you get started on your Docker journey! Let me know if you have questions.

If you’re more of a product manager, or just looking for why you’d want to use Docker, and understand its usecases, you can take a look at this presentation I had published on Why to docker? as shown below.

Questions, comments, or concerns? Hit us up by leaving a comment below…


Filed under DevOps

HashiConf 2017

As part of our embarrassment of conference riches here in Austin this year, I just went to HashiConf 2017 last week (Sept. 19-20).  HashiConf is the company conference for HashiCorp, the guiding hand behind a whole set of cool open source projects used by many newfangled technorati. We use many of these at AlienVault and so I went to see what’s hot and new!  If you’re not familiar, here’s the open source tools Hashi runs:

  • Vagrant – Run multi-server development environments on your laptop. We’ve mostly moved on to using Docker Compose for this but if you’re not using docker yet it’s mandatory!
  • Packer – A utility for building VM images, including AWS AMIs, Azure, etc.  At AlienVault we use this extensively to bake images for virtual appliances.
  • Terraform – An infrastructure as code tool, like AWS CloudFormation but cross-cloud. We use this on one of our products; I personally don’t have a lot of experience with it though.
  • Consul – A distributed configuration store and service discovery tool.  I did a consul setup for one of our products.
  • Vault – a secret store – it stores credentials encrypted but can also dynamically provision them. I am very interested in us starting to use this.
  • Nomad – a cluster scheduler – does more than Amazon ECS but less than Kubernetes.

I wanted to go to the Vault training but it was sold out by the time I managed to poke the sullen beast sufficiently to get registered.

The good news is that all of the sessions will allegedly have video posted publicly at some point!  So you won’t have to rely on my notes below, which is good.  Here’s the conference schedule.  All the session videos are now available as a YouTube playlist at:

If I were some kind of paid blogger or this was some ad-driven site I’d say “We’re done with part one, come back next post for more!”  But I’m not, so read on to get all my debrief from the conference. If it’s too long, you’re too old!!!

Day One

The Day 1 keynote was packed with info.  It started with Mitchell Hashimoto (the Hashicorp founder, as you might suspect from the name) talking about the mad growth they’ve seen – in 5 years since founding they’re up to 130 employees, conducted 150 releases across their 6 products last year, and had 22 million downloads of these tools in the last year (1.5M in the last month).  Makes sense to me; most people I know use at least one Hashi product even if it’s just vagrant or packer.

They have Terraform, Nomad, Vault, and Consul Enterprise (hosted) offerings.

Let me take an aside to say – while it’s admirable they didn’t spend most of the conference pushing their paid products (I’m looking at you Dockercon), there was so little information on them that it was confusing.  Back a year ago when I started in on a Consul implementation I poked around their site trying to figure out their hosted offering (called Atlas at the time) and was basically thwarted. And it’s not much better now.  What do these give you?  What do they cost?  I had a *lot* of hallway conversations with people who similarly had tried to look into them and been rebuffed.  “It’s like buying a Ferrari,” said one person who I’ll leave anonymous. “There’s no list price, calling them just starts some conversation about whether you should really be a customer of theirs or not.”  Sucks, I like using hosted services rather than rolling my own if it at all makes financial sense, but who has time for that?

Anyway, so they started working their way down the products to make announcements. They’re now marketing these as “The Hashistack!”


It’s going gangbusters and they added lots of features including a UI console, imports from external and local variables, stability, remote backends, and more.  I can testify to this, before the remote backends and you had to keep state in a file – that sucked.  (For those of you familiar with AWS and not Terraform, let me explain – CloudFormation has two parts to it – the actual templates and then the storage of system state.  The AWS fabric itself keeps the state for you and you get it via API.  Since Terraform is cloud agnostic the tool needs to store state itself; and instead of starting with a database or whatnot they just started with a file and have expanded from there.  Now you can do S3, consul, various other stuff.)

Their big announcement is the Terraform Module Registry.  I like to call it the TerrorHub or even the TerrorDome (with apologies to Public Enemy). Basically, you know, where people share stuff in the community like dockerhub.

I find this super interesting.  Done right, repos like this are a huge force multiplier for modern tools.  On the Open Threat Exchange, I’ve been happy just to use AWS CloudFormation.  We’re not interested in cross-cloud at all, and we use changesets and exports and other modern CF things, so going to Terraform wouldn’t really get us much -maybe some extra modularity, but we do continuously deployed microservices and things are factored out where we don’t need a giant ass template anyway. But, AWS does a crappy job of providing and enabling the community to provide decent CloudFormation templates. They try – but they’re hard to find in the giant rat’s nest of AWS docs, they are generally somewhat problematic (e.g. the quickstart Mongo CF they provide has a node hardcoded to be the primary and the others to be secondaries – but mongo works by consensus elections of the primary man!!!). Now, it’s possibly to bork this up, the Chef community had fragmentation around this till they finally put together the Chef Supermarket in 2015; we use Puppet at AV currently and at the time we selected it with the deciding factor being easier access to high quality modules for re-use.

They had Corey Sanders from Microsoft show off their Azure console that now has an embedded “Azure Cloud Shell” (just a container they expose via browser) and now it’s coming pre-loaded with Terraform!  Azure’s been Johnny on the spot with docker and Linux and Hashi stuff over the last few years.

Terraform Enterprise apparently has a lovely visual viewer for workspaces and is API everything and will have general signups open by EOY.


In an automated environment, “don’t keep your passwords in source” is really kind of a laughable lie.  I mean, you can not hardcode them and insert them from something else, something properly also kept in source control…  Hence Vault.  Vault is an internal project they open sourced 2.5 years ago.  Network configurations have become increasingly complex – hey did you know network perimeters aren’t secure any more?
Secret management, encryption as a service, privileged access management are all critical needs, Vault has added on bunches of functionality to them – and its secure plugins open it up to you!

Dan McTeer from Adobe came out to describe how his team makes self service security solutions as an internal platform (using Vault amongst other things) so the many, many other Adobe ops teams don’t have to waste time reinventing the wheel.

Vault 0.8.3 has tight and easy kubernetes integration. Kubernetes has a secret store but… Not a good one. (And before any k8s people get their shorts in a wad, the “take it all or leave it all” attitude from some k8s people is why some folks are sticking with more composable solutions. Don’t say your components are pluggable out one side of your mouth and then give people flak for doing it out the other.)


Consul can do so many things it’s hard to talk about sometimes. “Service discovery and distributed config store?” Hard for novices to get their heads around. There’s a session I’ll describe later that does a great job of demystifying it.  For now, note that they added dead server cleanup and other new functionality.

Since Consul is built on very academic stuff like SWIM and RAFT, they’ve started a research arm and have published their first paper on Lifeguard: SWIM-ming with Situational Awareness which reduces gossip protocol false positives by 98%.

And, they have just gone 1.0 with Consul!  So it’s battle tested, mother approved.


Nomad’s their batch and service scheduler.  I have to admit, I went into the conference with a “bah who cares” attitude about Nomad, but afterwards I think it has its points.

Previous reviews I read compare it unfavorably against Kubernetes and Swarm. To be fair they are deliberately smaller and more composable, which is why we are still on ECS and haven’t taken the big ol’ step to Kubernetes ourselves yet. (Kubernetes: The OpenStack of 2017).

Anyway, new stuff.

  • Everything’s API and CLI – so now they have a UI integrated into the binary, like Consul!
  • Role based ACL policies driven by ACL tokens. “IAM for Nomad.”
  • Citadel passed 1M containers under Nomad
  • Nomad Enterprise has native namespacing for larger shared environments.

The Big Secret Announcement

Mitchell gives us a big ol’ windup. In terms of total automation we went from VMs and config management to cloud and IoC to containers and microservices to schedulers… What’s next?
How do you enforce rules? Forbid changing config outside work hours (consul)… Ensure all services have health checks (nomad)… Ensure all TLS certs are 2048 bits (vault)… Ensure all AWS instances are tagged (terraform)…
Announcing Sentinel – a policy as code engine suitable for use in a continuous delivery pipeline.

  • Define and version your compliance rules, test and automate them.
  • Language: Easy to learn and use, mostly one liners, easy logic.
  • Can do active enforcement (block) and passive (check).
  • Levels of fail – advisory, soft, hard.
  • Workflow (“simulator”) – can mock and test locally.
  • Plugins as imports.
  • Terraform Enterprise uses this, so like you can make rules before the plan and apply to validate changes
  • Consul Enterprise, use during KV modifies or service registrations
  • Vault Enterprise, role and endpoint governing policies
  • Nomad Enterprise, run on job create/update introspecting on their details (e.g. artifacts only come from this repo)

It wasn’t immediately clear if this only worked with the Enterprise offerings or not, but it appears (after talking to other confused folks over the 2 days) that the answer is yes.  Well, that sucks, as it limits adoption to the 1% who care of the 1% who have managed to get any Enterprise offerings.   Oh well.

I’ve been asked if this is like Chef’s InSpec.  Kinda, but InSpec is for compliance rules “inside the box” and Sentinel is for rules about the system, just like Chef is to Cloudformation. So IMO they’d be super complementary.

Finally, Dave McJannet, the CEO, came on to talk about the Hashicorp Partner Network for resellers and integrators.

Guiding principles:
  • Workflows not technologies
  • Automation through codification
  • Open and extensible

All right, that was the keynote!  Don’t worry, I didn’t take that many notes from other things.

Session One – Sentinel

Having just heard about Sentinel, and not being clear yet I couldn’t use it if not on all Enterprise offerings, I skedaddled over to a session on it. Again, these will eventually be online so I’m not trying to duplicate whole talks and demos in text, this is to give you some flavor and opinions.

Defining, communicating, implementing, and auditing policy gets complex with scale. And it then becomes a huge source of friction on real operational work, despite attempts to document it. So just like we’ve fixed similar problems, let’s do it as code.

Then he says all the things Mitchell did in the keynote, but slower.

  • Example policy: Terraform can’t execute when consul healthchecks are failing
  • Yay golang!
  • Made their own simplistic DSL.
  • We’re 15 minutes into a 40 minute slot. I kinda want to switch to the terraform session, but I know from the attendance here that it’s standing room only over there.

import “time” <- you can import libs it knows how to get info from (sockaddr, tfplan, job)
main = rule { time.pst.hour is not 3 } <- functioney syntax, rules are the main thing, plain english “or”s and stuff
if an element returns undefined that’s ok
variables, dynamic typing

types of policy – advisory (just tells ya), soft (can be overridden), hard (no screw you)

writing and testing policies, you can use their simulator, e.g. “sentinel test”
sentinel apply <file>
make test folder (test/<policy>/<test>.json
config, mock, say what rule you want to test, then it’ll run all the tests with “sentinel test <policy>”

It integrates with nomad enterprise, and he shows how you accidentally change a deployment count too low and it fails the rule and doesn’t happen.

Session Two – Vault

Liz Rice of Aqua (@aquasecteam) has a product written around Vault.

How to handle secret attributes and lifecycle in docker? Passing secrets into containers goes from


  • in source code
  • in image/dockerfile

To less bad but still bad:

  • Env vars – can exec in and see it, can docker inspect it and see it, can cat /proc/<pid>/environ, leak into logs…
  • mount volume with directory with secrets – tempfs is in memory. can still get it via docker exec and /proc.

Docker orchestrator support for secrets:

  • Nomad + Vault – secrets passed as files, tasks get tokens to retrieve values
  • Docker – swarm has service support but not for pocs. rotation needs restart though, and secrets go into the raft log encrypted -but the key is right there unless you lock your swarm.
  • kubernetes – in a pod yaml, namespaced, as vol or env var, can turn on RBAC with —authorization-mode RBAC. Stored in etcd and you have to make sure it’s encrypted.

A docker -pluggable secret backend is in progress.
In kubernetes – use kubernetes-vault
For others – Aqua Security secrets management!
With aqua you can’t get the secret via inspect or /proc, and it has audit logs n stuff.

Session Three – Enterprise Security with Hashistack – Palantir

Ooo this one was good.  Pic!

2017-09-19 14.36.47

The Hashistack

So you need to:

  • Secure your infrastructure
  • Secure your configuration
  • Make the secure thing the easy thing to do
    it gets pretty complicated fast.

There are 6 pillars of security in your infrastructure:

  1. encryption
  2. access segmentation
  3. patching
  4. centralized logging
  5. mfa
  6. defensive backups

He will skip talking about #1 and #2 because they are obvious and boring.


MalwareBytes shows: It now takes 4 days on average to weaponize new exploits. Patching needs to be in phases, alert on failures, roll back, run daily
They use immutable AMIs with packer/terraform. They burn a common AMI with vault/consul/nomad/filebeat and just turn on what’s needed on a given server. Versions are kept in a variable.json file.
Their terraform lays out 3 ASGs for nomad/consul, vault, and then nomad workers.

How to bounce a hashistack:

  • delete node/add node for consul/nomad
  • like that with stepdown for vault
  • nomad workers – add new ones, drain old ones
  • Ain’t nobody got time for that.

They wrote “bouncer” to do that easily and automatically, open sourced at github/palantir/bouncer. So you have it rebuild and roll in new versions daily.  Ta da!


You need them to do incident response. You want <5 min latency, many formats, long retention, source identified – and opt-out not opt-in (in other words you get all the damn logs and not just the 3 you know about).

They started with workers shipping logs to a SIEM. Then they shipped to logstash which sends to multiple locations. They have a custom rsyslog template that puts everything into json, because logging in json is always better.  [If you don’t believe that you will be tied to a chair and Charity Majors will bludgeon you with a whiskey bottle until you are reeducated. -Ed.]

Then they go journald to rsyslog for nomad etc. Other ones, filebeats watches. Telemetry should monitor high risk files and get insights into running procs, kernel versions, etc.


MFA on everything?  Sure! But… It’s bullshit. The process of getting logged in and the number of logins and MFAs you have to do to go from “sitting down at your desk” to “logged into the necessary box” is ridiculous.

2017-09-19 14.50.20

MFA is Bullshit

Need to simplify finding hosts, ssh’ing.
They wrote a go wrapper around sesh, aws, and vault commands called vault-ssh-helper.  They also use a duo pam module and a yubikey (way faster than phone mfa).
from laptop, get vault token which then are used to look up aws creds, then used to look up ssh, one MFA later you’re in.  ssh/mfa becomes the easy path.

Defensive Backups

Use a separate backup account with you not the admin. back up storage by default, not opt in.  You need those backups available for testing too.
So, RDS. We have a job that makes a snapshot, shared with a second account. A lambda in the other account reads it, makes a copy, and shares it back.
It’s run from a nomad batch job, assumes all RDS and S3 needs to be backed up.

Securing nomad jobs and vault policies

First, gate on checkin, and second, stop bad code.
gpg sign commits – sign with yubikey
or, duobot ( calls the duo api and sends you a push to your mobile.
They built some bots to automate all this, approver, duo-bot (mfa absed deploy) and bulldozer (auto-merge if status green) – (github)
Unit tests for jobs – naming schemes, folder structure, health checks
CI converges to what’s in the repo hourly

Experimenting in prod with immutable infrastructure – Tim Perrett

Slides here (slideshare login required):

Converged infrastructure – you know, scheduling workloads.
New table stakes – fast, observable, self service, auto cleanup.
Testing in prod is a reality.  Here, he avoids using any of the common “test in prod” memes, for which I salute him.
Emergent behaviors of large systems – conways law and hyrum’s law – and  microservice complexity means that local testing is still a fable. Scaling peaks, human error can’t be locally simulated.

On their site, user calls gets segmented at edge (opentrace, zipkin), routed to the right backend, and publish its telemetry for analysis. A front end envoy proxy to split to proxies that go to edges

Feature flags are inferior!
Data plane for Routing – envoy, linkerd, nginx+. Integration with the control plane is important.

Options: Envoy in container? Sidecar as proxy? Host based envoy?

Control Plane itsisomething or…  I lost some of it here.

Instrument with segment identifiers, make apps experiment aware, interesting data is usually wrong.

That was all the notes I took from Day 1.  I also spent time talking with people I knew, some vendors (Amazon, Hashi, Datadog, a new APM vendor called Instana…)

Day Two

Well, the scheduled keynoter dropped, so we got a quick sub:

The Ecological Impact Of Compute – Seth Vargo

Wholesale colos are hideously less energy efficient – once adjusted by volume, they use 49% vs cloud’s 5% of the energy picture.

2017-09-20 09.44.26
Due to overprovisioning, invalid IT procurement focus (bulk), unused “zombie” servers, lack of standard utilization metric
IT managers can’t identify owners for 15-30% of the servers but are reluctant to decomm them!
Electric costs are paid by someone else.
Priority of efficiency low compared to deploy, avail, security, reliability
Schedulers help with this.  And use the cloud you poor benighted momos.  (I added that last part)

Keynote by Kelsey Hightower! On Hashinetes!

Kelsey is obviously huge into Kubernetes but he loves the Hashistack too.  And sometimes he uses both nomad and kubernetes at the same time?  CRAZY EH???

He presents some info from circleci on why they use kubernetes and nomad both (kubernetes good for containers, nomad does non-containers)

Demo time!

  • dynamic credential provisioning via vault – vault makes a mysql username, the app uses it, then vault deletes it!
  • he tells his android phone “create nomad” and it deploys it in k8s
  • make your nomad worker pool in the cloud, not on K8S

Elasticsearch as a Service Using the Hashistack by Yorick Gersie

EBay classifieds runs a buttload of single tenant Elasicsearches.  So do we, which made me very interested in his talk.

They spin it up with terraform, nomad, etc.
– discover master nodes
– direct traffic to right cluster
– port conflicts
– data persistency
– control resources
– deal with hardware outages automatically
– separate auth
– enforce tls
– enforce allocation placement

Very quick notes, really need the video or slides:

  • Use LB, fabio, consul, nomad, docker
  • consul based discovery plugin for elasticsearch
  • nomad 0.5 has “sticky” ephemeral disks
  • use saltstack, trying to reduce amount of CM. just set up nomad, consul, fabio;
  • render job files per cluster; deploys/renews ssl certs
  • immutable infra still hard/slow to deal with

Jeff Mitchell – Vault and Sentinel

This was a bait and switch from Vault at scale which made me sad.

How Sentinel works with Vault.

ACL paths are put into a policy
A policy is attached to a token
at request, policies are combined into an ACL used to check access

ACLs are HCL – JSON but not very expressive.
ACLs are grants/restrictions on paths… which is restrictive

Sentinel extends ACLs with role governing policies and endpoint governing policies
Don’t use root tokens

Building a Privileged Access System In Vault – Lee Briggs – Apptio (jaxxstorm on github)

accessing dbs – root password, app passwords. tracking, sharing, rotating and distribution…
vault can make dynamic mysql creds!
they already had consul and use puppet, so used jsok/puppet-vault and used consul-HA backend
so they deployed vault onto the consul servers
initializing vault – GPG support built in and then init vaults in each DC with the API.
what about vault unsealing?
add your vault servers to a config fule, add your encrypted unseal key, it prompts for your GPG keyring password and unseals all the vaults
so unseal every morning

to config all the DCs
UKHomeOffice/vaultctl, also new terraform vault provider

for mysql config, they run puppet to install db and vault user and roles, then API call to the region’s vault to add as a backend.

to make logins easy, ldap auth with policies… but manual. apptio/breakglass like vault ssh in that it gets ad password and then does all the stuff to get to mysql, ssh, docker

if you’re using consul as a backend then turn on acls and block ports 8500/1
consul snapshot to take backups, test weekly bu restoring to different port vault, connect to the consule, unseal and verify

– pick 1 thing and vault it
– consul+vault gives you HA for free
– automation has tradeoffs (secret zero problem)
– engineers love the http api

Consul Infrastructure Recipes – Preetha Athan

OK so this went by fast and I didn’t take extensive notes because I knew most of it – but if you are using consul it is absolutely mandatory that you seek this out.  It’ll explain in quick form a dozen different things to do with consul from using consul-template to the one I didn’t know about…

Consul supports prepared queries with complex logic, you can use them e.g. for automated failover as part of service discovery.
So reviews.service.consul -> reviews.query.consul and it execute a query where it can have failover servers and all.

That’s my notes!  I also caught up with some folks from ScaleFT, who are implementing the Google BeyondCorp zero trust model as a product, we are very very interested in it.  Network perimeters and ssh keys?  Busted, like in 1990 busted.  Do something better.

Well I hope that’s enough for ya… I was definitely impressed, found a reason to maybe use terraform, found a reason to maybe use nomad, definitely want to use vault and more consul. I wish sentinel was usable by civilians.  Good crowd, good conference, make sure and watch the videos when they emerge!

Leave a comment

Filed under Conferences, DevOps

New DevOps Courses In The Can!

Well it was  busy week last week – James, Karthik, and I were all in lovely Carpenteria, CA  at the Learning HQ to film some new DevOps Foundations courses!

We have two out already – James and I did DevOps Foundations, which lays out the basics of DevOps from culture to containers. It’s a three hour course, and should suffice to orient someone in all the ways of DevOps and defines Continuous Delivery, Infrastructure Automation, and Reliability Engineering as its three practice areas. (There’s a course handout under the Exercise Files that has links and bibliography, as well.)  It’s DevOps 101 if you could use that!

And then we started to flesh out the major DevOps practice areas we defined in that course as 200-level courses.  These focus on concepts but illustrate with tool demos. So we filmed DevOps Foundations: Infrastructure Automation in March, which released the end of April.  It’s two hours, and covers infrastructure as code concepts and the basics of creating infrastructure from specs with e.g. CloudFormation, provisioning systems with e.g. Chef, and going immutable with Docker.

But now we have an irresistible urge to do more, so in a double shot that took about a year off my life, last week James and I recorded DevOps Foundations: Continuous Delivery, which goes over continuous integration and delivery and shows you how to build a delivery pipeline – we used  Jenkins/Nexus/Chef/go/abao/Robot Framework but again we focused on concepts and did just enough implementation to illustrate it.

2017-07-17 14.01.27

James went home mid-week and Karthik came out, and we also recorded DevOps Foundations: Lean and Agile!  Lean and Agile are integrally related to DevOps and especially to being successful at DevOps.  Our content manager actually asked us to do this one; we were kinda bulling ahead on our three main practice areas, but we said sure!  We cover some Agile and Lean basics, and then we take a tip from The Goal and The Phoenix Project, and the bulk of the course is a fictional implementation stitched together from real experiences we’ve both had doing these at various companies.  It was fun!  Here’s a look at behind the scenes.

2017-07-21 14.47.33

Both of these should drop in about 5 weeks, so keep an eye out.



Leave a comment

Filed under DevOps

AWS CLI Queries and jq

Anyone who’s worked with the AWS CLI/API knows what a joy it is. Who hasn’t gotten API-throttled?  Woot!  Well, anyway, at work we’re using Cloudhealth to enforce AWS tagging to keep costs under control; all servers must be tagged with an owner: and an expires: date or else they get stopped or, after some time, terminated.   Unfortunately Cloudhealth doesn’t understand Cloudformation stacks, so it leaves stacks around after having ripped the instances out of them.  We also have dozens of developers starting instances and CloudFormation stacks every day.  Sometimes they clean up after themselves, but often they don’t (and they like to set that expires: tag way in the future, which is a management problem not a technology problem).  Sometimes we hit AWS quotas in our dev environment and an irritated engineer goes and “cleans up” a bunch of stuff – often not in the way that they should (like by terminating instances and not stacks).

So, I was throwing together a quick bash script to find some of the resulting exceptions and orphans in our environment.  Unattached EBS volumes.  CloudFormation stacks where someone terminated the EC2 instance and figured they were done, instead of actually deleting the stack. Stuff like that. This got into some advanced AWS CLI-fu and use of jq, both of which are snazzy enough I thought I’d share.

Here’s my script, which I’ll explain. You will need the AWS CLI installed (on OSX El Capitan, “brew install awscli” or “pip install –ignoreinstalled six –upgrade –user awscli” – the –ignoreinstalled six works around an El Capitan problem).  You need “aws config” configured with your creds and region and such, and set output to json.  And you need to install jq, “brew install jq.”

#!/usr/bin/env bash
# This script finds problematic CloudFormation stacks and EC2 instances in the AWS account/region your credentials point at.
# It finds CF stacks with missing/terminated and stopped EC2 hosts.  It finds EC2 hosts with missing owner and expires tags.
# It finds unattached volumes. Should you delete them all?  Probably. Kill the EC2 instances first because it'll probably
# make more orphan CF stacks.


echo "Finding misconfigured AWS assets, stand by..."
for STACK in $(aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --max-items 1000 | jq -r '.StackSummaries[].StackName')
        INSTANCE=$(aws cloudformation describe-stack-resources --stack-name $STACK | jq -r '.StackResources[] | select (.ResourceType=="AWS::EC2::Instance")|.PhysicalResourceId')
        if [[ ! -z $INSTANCE  ]]; then
                STATUS=$(aws ec2 describe-instance-status --include-all-instances --instance-ids $INSTANCE 2> /dev/null | jq -r '.InstanceStatuses[].InstanceState.Name') 
                if [[ -z $STATUS  ]]; then
                        BADSTACKS="${BADSTACKS:+$BADSTACKS }$STACK"
                elif [[ ${STATUS} == "stopped" ]]; then

echo "CloudFormation stacks with missing EC2 instances: (aws cloudformation delete-stack --stack-name)"

echo "CloudFormation stacks with stopped EC2 instances: (aws cloudformation delete-stack --stack-name)"

echo "EC2 instances without owner tag: (aws ec2 terminate-instances --instance-ids)"
aws ec2 describe-instances --query "Reservations[].Instances[].{ID: InstanceId, Tag: Tags[].Key}" --output json | jq -c '.[]' | grep -vi owner | jq -r '.ID' | awk -v ORS=' ' '{ print $1  }' | sed 's/ $//'

echo "EC2 instances without expires tag: (aws ec2 terminate-instances --instance-ids)"
aws ec2 describe-instances --query "Reservations[].Instances[].{ID: InstanceId, Tag: Tags[].Key}" --output json | jq -c '.[]' | grep -vi expires | jq -r '.ID' | awk -v ORS=' ' '{ print $1   }' | sed 's/ $//'

echo "Unattached EBS volumes: (aws ec2 delete-volume --volume-id)"
aws ec2 describe-volumes --query 'Volumes[?State==`available`].{ID: VolumeId, State: State}' --output json | jq -c '.[]' | jq -r '.ID' | awk -v ORS=' ' '{ print $1  }' | sed 's/ $//'


The AWS cli, of course, lets you manipulate your AWS account from the command line.  jq is a command line JSON parser.

Let’s look at where I rummage through my CloudFormation stacks looking for missing servers.

aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE --max-items 1000 | jq -r '.StackSummaries[].StackName'

Every separate aws subsection works a little different.  aws cloudformation lets you filter on status, and CREATE_COMPLETE and UPDATE_COMPLETE are the “good” statuses – valid stacks not in flight right now.  The CLI likes to jack with you by limiting how many responses it gives back, which is super not useful, so we set “–max-items 1000” as an arbitrarily large number to get them all.  This gives us a big ol’ JSON output of all the cloudformation stacks.

    "StackSummaries": [
            "StackId": "arn:aws:cloudformation:us-east-1:12345689:stack/mystack/1e8f2ba0-4247-11e7-aad1-500c28601499", 
            "StackName": "mystack", 
            "CreationTime": "2017-05-26T19:11:28.557Z", 
            "StackStatus": "CREATE_COMPLETE", 
            "TemplateDescription": "USM Elastic Search Node"

Now we pipe it through jq.

jq -r '.StackSummaries[].StackName'

This says to just output in plain text (-r) the StackName of each stack. You use that dot notation to traverse down the JSON structure.  So now we have a big ol’ list of stacks.

For each stack, we have to go find any EC2 instances in it and check their status. So this time we use a select inside our jq call, to find only items whose resource type is “AWS::EC2::Instance”.

aws cloudformation describe-stack-resources --stack-name $STACK | jq -r '.StackResources[] | select (.ResourceType=="AWS::EC2::Instance")|.PhysicalResourceId')

And then for each of those instances, we get their status, which is in the InstanceState.Name field.

aws ec2 describe-instance-status --include-all-instances --instance-ids $INSTANCE 2> /dev/null | jq -r '.InstanceStatuses[].InstanceState.Name'

That works.  But there’s more than one way to do it!  The AWS CLI commands support a “–query” parameter – which lets you specify a JSON search string that happens on the AWS end, so you have to do less parsing on your end!

To find instances without the owner tag,

aws ec2 describe-instances --query "Reservations[].Instances[].{ID: InstanceId, Tag: Tags[].Key}" --output json | jq -c '.[]' | grep -vi owner | jq -r '.ID' | awk -v ORS=' ' '{ print $1  }' | sed 's/ $//'

What this does is look under Reservations.Instances and basically outputs me a new JSON with just the ID and tags in it.  “jq -c ‘.[]'” just crunches each one into a one-liner.   I grep out the ones without an owner, turn them into one line with awk, and strip the training space at the end from the awk with a sed (ah, UNIX string manipulation).

With this, you can choose what to put into the –query and what to do after in jq.  The –query is fast and cuts down your result set, so you run less risk of magically missing resources because AWS decided there were too many to tell you about.

You can do filters in the query – so for example, when I do the volumes, instead of doing what I did for the tags using grep, I can instead just do:

aws ec2 describe-volumes --query 'Volumes[?State==`available`].{ID: VolumeId, State: State}' --output json | jq -c '.[]'

Yes, those are backticks, don’t blame the messenger.  This is more precise when you can get it to work.  In the instances’ case, people aren’t good about using the same case (owner, Owner, OWNER) and also I just plain couldn’t figure out how to properly create the query, “Reservations[].Instances[?Tags[].Key==`owner`” and other variations didn’t work for me.  I’m no JSON query expert, so good enough!

Between the CLI queries and jq, you should be able to automate any common task you want to do with AWS!

1 Comment

Filed under DevOps

Tcpdump and Wireshark on OSX

I’m going to start sharing little techie tidbits that require me to go scour the Internet for exactly how to do them, in hopes of making you able to do it in a lot less time than it took me!

So I’m having trouble with connection times spiking to an Amazon Web Services ELB, so it’s time to break out the tcpdump to take packet traces and the wireshark (was ethereal long ago) to analyze it.  I’m on OSX El Capitan (10.11.6).

tcpdump comes on OSX (or if it doesn’t, something installed it without me knowing!).  Step one is figure out what network interface you want to dump.  This will list all your network interfaces.

networksetup -listallhardwareports

Then, run a packet trace on that interface.  I’m using en0 the primary wireless interface, so I run:

sudo tcpdump -i en0 -s 0 -B 524288 -w ~/Desktop/DumpFile01.pcap

I go to another window and hit the URL I’m having trouble with – you can use whatever, but I used ab (Apachebench) which comes with OSX.  Other popular URL-hitters you might install are curl, wget, and siege.

ab -n 10 http:///

Then come back and control-C out of the tcpdump capture.  Now I have a network dump of me hitting that URL (plus whatever other shenanigans my computer was up to at the time, so there’s probably a lot of noise in there from chat clients etc.).

Now to analyze it – wireshark.  I had to go a couple rounds with the installation.

If you want the UI you need to install it as:

brew install wireshark --with-qt

(If you just install wireshark without –with-qt you don’t get wireshark, you get a command line called tshark, and then  you need to reinstall…) For this, as with most things, you need Xcode or at least the Xcode command line tools (I always just install the tools).  You install them with:

xcode-select --install

But if you have an older version (<8.2.1) the wireshark build will fail.  To update the command line tools, you… Apparently you don’t any more.  The App Store doesn’t offer Command Line Tools updates and Apple has gotten more unclear and squirrelly about whether they’re even a thing.  So I just installed full XCode from the App Store, whatever, it’s just network and disk space and contributing to the heat death of the universe, but I’m not bitter, and then it builds.

Optionally if you want to capture from within wireshark on your local box instead of having to tcpdump separately also do

brew cask install wireshark-chmodbpf

But to analyze your tcpdump file just run


And load in the capture file. The quickest way to then sort into what you want is to find one part of a transaction of interest – like in my case by filtering on “http” or just looking around – and then right-clicking on one packed and saying “Follow… HTTP stream” and you get a whole transaction end to end.

Screen Shot 2017-05-26 at 2.43.56 PM

Screen Shot 2017-05-26 at 2.44.16 PM

And now you test out your TCP/IP network admin knowledge by rooting through and seeing if you can find what’s going wrong!


Filed under DevOps