Well, if nothing else I’m happy to have Google Cloud around to provide some competition to push Amazon Web Services. Immediately after Google announced dramatic price drops, Amazon has responded doing the same!
Now if they can only also shame them into dropping their whole crazy reserve instance scheme and go to progressive discounts like Google just did too, the world will be better.
We had a little get-together here in Austin today, sponsored by MomentumSI and hosted at Capital Factory (thanks to both!), to view the Google Cloud Platform newest product announcement webcast. About 24 local engineers showed up to watch.
You can view the whole thing yourself here, or just read my notes from the event.
Their thesis statement was that cloud, while cool, is still too hard for many people, hindering adoption or slowing innovation. So they’ve worked on making it easier.
Cost calculation is super complex (reserve, on demand, etc.). They talk about “other industry standard clouds” by which they mean Amazon Web Services. They note the drawbacks to reserved instances, which I am all totally in agreement on (see my earlier article Why Amazon Reserve Instances Torment Me for more on that). Specifically they note that reservations constrain your design choices, which is one of the great reasons to go to the cloud in the first place – Amen, brother!
Though cloud prices have been dropping 6-8% a year, hardware’s been dropping 20-30%. Why is Moore’s Law not translating into more sweet green in our pockets? It should, they contend. Thus, they are announcing on demand price drops:
Introducing sustained use discounts – no pre-plan or reserving ahead of time, instead prices automatically drop as VM usage is sustained over 25% of the month and then progressively from there. 100% use is a 53% discount over current (remember that includes the new 32% reduction, so another 21% from current for continued use). With linear machine cost scaling, makes it simple(r) to predict and calculate your costs.
Current cloud (hint: AWS) forces other tradeoffs: time to market vs scalability, flexibility (iaas) vs automatic management (paas), big data vs realtime data analysis.
But first, we interrupt our messaging to talk about other random new features based on customer feedback. To wit:
The features are nice but even nicer was that they implemented these based on customer feedback, which means they consider this a real product with real customers and not just a fun tech thing for their own ends (which to be fair 80% of Google’s offerings are, and it can be hard to tell the difference).
So on scaling… You need deployment! Troubleshooting! Use tools you know!
They have a new “gcloud” command line tool
“gcloud init” pulls down the app via git, you can just edit, git commit, git push
They have a build service integrated – it spins up a jenkins/maven and builds, deploys – you can see release status in the console.
There’s also a new unified logs viewer with basic searching – like Splunk junior, with one cool dev feature. Click on the code in the stack trace and you’re put directly into the code in the console’s source view. Fix and commit, it auto-builds, bam you’re fixed.
A new halfway state – “managed VMs.” It’s the normal PaaS, but in the config, you can tell it things to apt-get install onto the instances, so you can have more third party software than the PaaS previously allowed.
Also, you can “enable debugging” on an instance and then log in interactively.
They’ve upped BigQuery to have 100k rows/sec ingest.
Example Demo: smart monitoring of 60 events/hour from 400k glen canyon power meters (17bn events/mo), with about 128k records. They did a visualization that is updating in near real time showing all those meters geolocated and you can go click on them to get realtime data.
He showed the complex BigQuery “bigjoin” to filter by meter lat/long from sep table and then by quartile across whole population. “Doing this in NoSQL would be impossible or very slow.”
They will be doing a Google Cloud roadshow soon – see cloud.google.com/roadshow – it looks like Austin will be on the list of cities!
The good thing about getting a bunch of techies together to view this was the discussion afterwards. The general sentiment was that:
1. The cost drops are nice and the approach to reserve/sustained use instances is much better. The reserve instance scheme is one of the worst things about AWS and if this drives them to adopt the same model, hooray!
2. The other new features (managed VMs, gcloud) are definitely nice. They are focusing on dev friendliness in their discussion but it’s a lot less clear how to operate this. If you’re really trying to stitch together a bunch of micro-services there’s not a lot of great support for that. They talk about using their PaaS and say “of course, if you use our PaaS you don’t need to carry a pager! You’d only need to do that if you’re doing IaaS and maintaining your own OSes.” That is dangerously naive and really made the whole group skittish. Most people there have done “play” things in Google’s cloud but are reticent to put mission critical items there, and this section of the presentation didn’t do a lot to improve that.
3. The BigQuery/realtime demo was impressive and multiple people would like to kick the tires on it.
Overall – it was a little light, but it was a keynote; the new features/pricing are all good; this shows more Google commitment to their cloud as a product but actual concerns still linger about maturity and suitability for realistically complex revenue-generating production applications.
Filed under Cloud
Since I had some time and have been thinking about it lately, I’ve upgraded and expanded my definition of DevOps on theagileadmin. Healthy debate welcome!
The third annual DevOpsDays conference in Austin will be May 5-6 (Cinco de Mayo!) at the Marchesa, where it was held last year! As many of you know, the DevOpsDays conferences are a super popular format – half talks from practitioners, half openspaces, all fun – held in many cities around the world since the first one in Ghent launched the DevOps movement proper.
DevOpsDays Austin has been bigger and better every year since its inception and should have something good for everyone this year. Come out and join your comrades from the trenches who are trying to forge a new way of delivering and maintaining software!
Filed under Conferences, DevOps
So you’ve decided to start playing around with Amazon Web Services and are worried about doing so securely. Here’s the basics to do when you set up to ensure you’re on sound footing. In fact, I’m going to use the free tier of all these items for this walkthrough, so feel free and do it yourself if you’ve never taken the plunge into AWS!
Signing up for Amazon Web Services is as simple as going to aws.amazon.com and clicking the “Sign Up” button.
It will want a password – choose a strong one, obviously – and some credit card info for if you exceed the free tier. It’ll want a phone number for a robot to call you – they show you a PIN, the robot calls you, you give the robot the PIN, and you’re good to go.
Next, set up multifactor authentication (MFA) for your Amazon account. You should see an option like this to go directly there immediately post signup, or you can pick the “IAM” section out of the main Amazon console.
When you go to the IAM console you’ll see two options under Security Status to turn on – Root Account MFA and Password Policy. I won’t talk much about the password policy except to say “go turn it on and check all the boxes to ensure strong passwords.” To turn on MFA, you need some kind of MFA device. The Amazon docs to walk you through the process are here. Unless you have a Gemalto hardware token already your best best is to just download Google Authenticator (GA) onto your iPhone/Android from the relevant app store (other choices here).
Once you’ve installed Google Authenticator, click on “Manage MFA Device” and choose virtual; it’ll show you a QR code that GA can scan to do the setup. Then you enter two tokens in a row from GA and it’s hooked up to the account. Now, to log into the console you need both your password and a MFA token. (You can also use GA for Dropbox, Evernote, Gmail, WordPress, etc. and is a good safeguard against the inevitable password losses these companies sometimes have.) Of course once you do this you need to be careful – if you lose the phone or device you can’t get in!
Now make sure and save all your credentials in a password vault. I prefer Keepass Password Safe along with MiniKeePass on the iPhone. Besides the password, you should go to your Security Credentials (off a popdown form your name in the top right) and store your AWS Account ID, Canonical User ID (used for S3), and any access keys or X.509 certificates you make – but it’s better not to make these for the main account, just for IAM accounts. Proceed onward to hear more about that!
All right – now your main login is secure. But you want to take another step past this, which is setting up an Identity and Access Management (IAM) account. IAM used to be a lot less robustly supported in AWS but they’ve gotten it to be pretty pervasive across all their services now. Here’s the grid of what services support IAM and how. You can think of this as the cloud analogy to UNIX’s “secure your root account, but still you shouldn’t log in as it but as a more restricted user.”
First, you have to set up a group. On the IAM dashboard click on the big ol’ “Create A New Group Of Users” button in the yellow box at the top.
Then make a group, call it “Admins” for the sake of argument.
Choose the “Administrator Access” policy template. If you know what you’re doing, you can change this up extensively.
Add a username for yourself (and whatever other people or entities you want to have full admin access, ideally not a long list) to the group.
For each user it’ll give an Access Key ID and Secret Access Key – these are the private credentials for that user, they should take them and put them in whatever password vault they use.
If you want to use that account to log into the console – and for this first one, you probably do – then once this is complete you go into Users, select the user, and under Sign-In Credentials it will say Password: No. Click Manage Password and set a password for that user; they can then log into the custom IAM login URL shown on the front page of the IAM dashboard (it’ll put it in the file when you Download Credentials, too).
Just as with the main user, you can (and should) also set up MFA on IAM users. It’s the same process as with the main account so I won’t belabor it.
After you set up your IAM user and its MFA, you shouldn’t log in using your main AWS account credentials to do work – only if you need the enhanced access to mess with account credentials. Log out and then back in with your IAM account to proceed. If you want to take it a step farther and make an even less privileged account without admin rights, which you can use for everyday tasks like logging in and looking at state or just starting/stopping instances but not manipulating more sensitive functions, you can do that too.
Of course if you are looking to manage multiple people, or separate apps that have access to your account (many SaaS solutions that integrate with your Amazon account will ask for IAM credentials with specific access), you can set up more groups with less access and have those entities use those. In general I’d suggest using a group+user and not just a user (different SaaS monitoring services recommend different approaches, but I think a plain user is less flexible). You can also get fancy with roles (used for app access from your instances) and identity providers. Remember the principle of least privilege – give things only as much access as they need, so that if those credentials are compromised there’s a limit to what they can do. There’s an AWS IAM best practices guide with more tips.
Security folks know that nothing’s complete without the ability to audit it. Well, you can turn on logging of AWS security events using CloudTrail, the AWS logging service. This will basically dump IAM (and other Amazon API events) to a JSON file in S3. This is a whole can of whupass unto itself, but the short form is to follow this guide to set up your trail, making sure to say Yes to “Include global services?”.
You can also go into S3 and set your bucket (properties.. lifecycle) to expire (archive, delete, etc.) the logs after a certain time.
Then you can do something like set up SumoLogic to watch it and review/alert on your logs. If you want to try that, the short HOWTO is:
* | json “userIdentity.userName”, “eventTime”, “eventName” as username, time, action | sort by time
They also have an app specific to CloudTrail; you have to contact Sumo support to get it turned on though.
All of this, of course, is about access at the Amazon account and API level. For your actual instances, you’ll want to set up secure network access and then manage the SSH keys you use to log into them.
VPCs used to be a limited option, and mostly people just used security groups. Nowadays, VPCs are standard and an expected part of your setup. They’re like a private virtual network. When you create your account, it’ll actually create one default one for you automatically. You can see it by going to the VPC Console. This default VPC, though, is set up for convenience and back compatibility – instances you launch into it will get a public IP address, which may not be what you want, and the default security group allows all outbound traffic and all traffic from within the security group.
You should consider starting one instance this way and then using it as a bastion host to gateway into your other instances, which shouldn’t have public IPs unless you really want them to be publicly facing. It’s hard to prescribe other specifics here because it really depends on what you plan to do. At a bare minimum you need to add an inbound SSH rule from your location to the security group so you can log into your first instance when you start it below. (They have a neat new “My IP” choice that’ll detect where you’re coming from. Of course that won’t work when you drive to the Starbucks…) Consider removing the rule allowing all traffic from within a security group – even within a group it’s more secure to allow specific protocols instead of “everything from everywhere.”
Ideally, you’d set up a VPN to the VPC’s Internet Gateway – but this requires expensive hardware or setting up your own server and is way out of scope here.
Then, of course, you finally get to starting instances! Each instance will start with a default root ssh key. Things you want to do here are:
The final step to doing all this securely is to not be making manual changes. Via the CLI or API you can automate a lot of this, but even better is using CloudFormation, maybe in conjunction with OpsWorks or another CM tool, to define in a readable config how you want your system to behave (VPCs, security groups, etc.) and instantiate off that. Nothing’s more auditable than a system that’s built automatically from a spec! You can cheat a little and set up your VPCs and all the way you want and use their CloudFormer tool to generate a CloudFormation template from your running system. Then you can edit that and tear down/restart from scratch.
The more you automate, the tighter you can make the security controls without inconveniencing yourself. A trivial example is you could have a script that uses the CLI to change the security group to allow SSH from wherever you are right now, and then close it afterwards – so there’s no SSH access from a location unless you allow it! In the same vein, allowing “all access” within a security group or from one group to another is usually done out of laziness and flexibility for manual changes – if you automate such that if you add a new set of servers, they also configure their connectivity needs specifically, you’re more secure. For defense in depth you could automatically configure the onboard firewalls on the boxes to mimic the security groups, just read the security group settings and transform into similar iptables (or whatever) settings. Voila, a HIPS. Pump those logs into Sumo too.
You could add tripwire or OSSEC for change detection, but also if you run your servers from trusted images and recreate them frequently, you can very much reduce the risk of compromise.
That’s my quick HOWTO on how to get servers running in a mode that’s likely way more secure than the average enterprise server unless you work for a bank or something, inside a couple hours. MFA, key based auth, all the network separation you could want, separation of privileges…
Agile Austin asked me to help re-launch their blog, so I’ve contributed a piece on “What Is DevOps?” for them!
I thought I’d take a minute for a teambuilding tip from Management Corner and share a winning teambuilder from here in Austin that I’ve used a lot and has always been a solid success. We call it “Meat, Guns, and Booze.” Steal it and use it with your teams!
Coming up with good teambuilders for an engineering team is pretty difficult. You have a mix of introverts and extroverts. Going to a movie doesn’t really allow for much personal interaction. A group meal lets you talk to the 3-4 people next to you, which in large groups isn’t a large percentage, and even getting everyone seated together can be a challenge. Not everyone drinks and may not appreciate hanging out at a bar. Messing around in the office is fine but sometimes you want to get out of there so that people aren’t continually distracted by work. Not everyone is active enough for something like paintball (especially in Texas weather!). Well, back some years ago while I was at National Instruments we were mulling over what a good field trip/teambuilder could be given all these limitations, and happened across a winning combination that never fails to deliver!
Meat Frenzy
First, head out to Lockhart, about an hour from most places in Austin, for a lunch of barbeque. Austin BBQ is good, but for the truly top tier stuff you have to go out to other locations (we love Franklin’s/LA Barbecue but going and standing in line at 10 AM to get some isn’t really a team activity). You can have internal debates over whether you go to Black’s, Smitty’s, or Kreutz’; try them all out over time as everyone will passionately defend the one they think is best. For vegetarians, there’s good sides, but also there’s a cafe right across the street from Smitty’s Market with salads and whatnot. Smitty’s also has a “no utensils” rule; eating meat with your hands definitely switches the mood to informal and friendly quickly!
Pro Tip: Bring cash. Some of them – Smitty’s for sure IIRC – don’t take credit!
Goodsprings from Fallout: New Vegas?
Besides the great BBQ, many folks haven’t seen old style rural Texas – even folks living in Austin, but especially people from other countries. I remember once we were headed into town, and it has a traditional town square with raised boardwalks and all – and the two guys from Ukraine in the back seat started wriggling around like itchy bear cubs and talking rapidly to each other in Ukranian. “What’s up?” I asked from the driver’s seat. “It looks just like Fallout!” they breathlessly exclaimed. I had to laugh. It’s worth a brief stroll around the town square and courthouse. There’s a western-stuff store for the greenhorns, too.
My .30/.30
A Ukranian Engineer Learns About Texas
If it’s raining, you can fall back to Red’s in Austin, they have several locations. But they’re small and cramped for groups. Right near Lockhart is the Lone Star Gun Range. They have rifle and pistol ranges, outdoor but with shade. You just need a driver’s license (or passport for the foreigners) and you can rent anything from a revolver to an AK-47 for a reasonable fee – you mainly end up paying for ammunition. (You can bring your own guns and ammo, but have to buy their ammo to use in their rental guns.) They give a quick safety briefing and off you go. As long as you have one or two more gun-savvy people to help folks who are having trouble figuring out how to reload or whatnot, you’re good to go; I’ve been there many times with groups of 10-20 composed of mostly noobs with no problems. And don’t worry – everyone gets into it. I’ve had people who were initially intimidated but once they get hands on for a minute they always take to it with gusto.
Even in Texas, gun owners usually don’t go out to the range all the time, so it’s a treat even for those who are more into it, and a novelty for those who aren’t.
Pro Tip: Don’t drive a super-nice car – the road to the place is not paved, it’s shell. If you have a sweet hand-waxed Porsche you should ask someone else to drive. Also, it’s outside in Texas in (whatever season) – bring sunscreen, hats, and a cooler full of water. There is an air-conditioned store with a bathroom and all if someone gets overwhelmed by the heat.
The Barrett
Sad Note: They used to have a Barrett .50 out there at Lone Star but they sold it. The rounds were super expensive but no one wanted to fire more than 1 or 2 anyway! But they have revolvers, automatics, rifles from .22 to ARs to AKs…
After shooting, you’re usually hot, and so stopping in at a watering hole on your way back to Austin is a good thing to do. Anywhere that’s convenient for you is good – I’ve done the Whip In as that’s uber Austiny, has an outdoor patio with picnic tables for good social mobility, and has a lot besides just beer (food, wide non-alcoholic drink variety). You got out of the office (and out of town), had the team nourish itself on food, then have an exciting activity, and now they get to interact with the ice already well broken. People don’t drink a lot because it’s already been a long day, as opposed to a “let’s go drink” teambuilder where people are more tempted to overindulge. But they have plenty to talk about. Non-drinkers – or if your entire company doesn’t like sponsoring events with alcohol – can just drink sodas and have an appetizer or two to unwind.
And yes, to all the wags out there who inevitably respond with “but don’t mix up the order!”, you do want to do it in this order – there’s the wisdom of no drinking before gun handling, but also because this sequence of events is carefully constructed to provide a cadence of icebreaking and interaction. You “break bread” together, which is a universal signal of togetherness, and sit by some people to eat, by others in the car, interact with others while doing a stimulating activity, unwind with others after, etc. And since it’s multipart, if someone really does have to come late or leave early there are multiple natural breakpoints. We also like doing three-round K1 racing as teambuilders but that’s more disruptive if people come and go.
I’ve used this teambuilder with multiple teams, multiple companies, and it’s always a hit. It’s especially a hit with international teams – folks from Malaysia, Ukraine, etc. see all of this as a special treat; it’s very common to talk to them a year later about their trip to Austin and they’ll immediately volunteer that this was their favorite part of their visit hands down. It’s relatively inexpensive and just takes a long afternoon (we usually leave at 11 and start wrapping up at 5). It’s maybe $1000 for a full day of fun for a dozen people.
Well of course the luridly titled “Meat, Guns, and Booze” specifically evokes Texas and is tuned to our location. But consider the same kind of event cadence and think about what’s distinctive in your area. You just want the team to share a meal, then participate in an activity that everyone can easily participate in, and then have an opportunity to relax and talk with each other afterwards.
Filed under General
I started thinking about this recently because there was an Agile Austin QA SIG meeting that I sadly couldn’t attend entitled “How does a QA manager fit into an Agile organization?” which wondered about how to fit members of other disciplines (in this case QA) in with agile teams. Over the last couple years I’ve tried this, and seen it tried, in several ways with DevOps, QA, Product Management, and other disciplines, and I thought I’d elaborate on the pros and cons of some of these approaches.
There are two things I’ve learned from this process that are pretty universal in terms of their truth.
1. Conway’s Law is true. To summarize, it states that a product will tend to reflect the structure of the organization that produces it. The corollary is that if your organization has divisions which are of no practical value to the product’s consumer, you will be creating striations within your product that impact client satisfaction. Hence basic ITSM and Agile doctrines on creating teams around owning a service/product.
2. People want to form teams and stay with them. This should be obvious from basic psychology/sociology, but if you set up an organization that is too flexible it strongly degrades the morale of your workers. In my previous role we conducted frequent engineer satisfaction surveys and the most prominent truth drawn from them is the more frequently people are asked to change roles, reorg, move to different teams, the less happy they are. Even people that want to move around to new challenges frequently are happier if they are moving to those new challenges with a team they’ve had an opportunity to move through Tuckman’s stages of group development with.
I have seen enough real-world quantified proof of both of these assertions that I will treat them as assumptions going forward.
We tried out all four of these models within the same organization of high performing engineers and thus had a great opportunity to compare their results.
When we started, we had the traditional model of separate teams which would hand off work to each other. “Dev,” “Ops,” “QA,” “Product” were all under separate management up through several levels and operated as independent teams; individual affinities with specific products were emergent and simply matters of convenience (e.g. “Oh, he knows a lot about that BI stuff, let him handle that request”) and not a matter of being dedicated to specific product(s).
Our first step away from the pure separate team model was to take those separate teams and embed specific members from them into service oriented teams, while still having them report to a manager or director representing their discipline. In some cases, the disciplinary teams would reserve some number of staff for tool development or other cross-cutting concerns. So QA, for example, had several engineers assigned to each product team, even though they were regarded as part of the permanent QA org primarily.
We (very loosely) considered our approach to be decentralized and microservice based; Martin Fowler is doing a good article in installments on Microservice Architecture if you want more on that topic.
With our operations staff we went one step farther and simply permanently assigned them to product teams and removed the separate layer of management entirely. Dev and “DevOps” engineers reported to the same engineering manager and were a permanent part of a given product team. Any common tooling needed was created by a separate “platform” engineering team which was similarly integrated.
Due to the need to surge effort at times, we also had some organizations that were project, not product, based. Engineers would be pulled either from existing teams or entire teams would additionally be pulled into a short term (1, 2, 3 month) effort to try to make significant headway across multiple products, and then dissolve afterwards.
Hmm, this looks like it’s getting big (and I need to do some diagrams). I’ll break it up into separate articles for each type of org and its pros and cons, and then a conclusion.
the agile admin 