DevOpsDays Austin 2019 Highlights

devops_mascot_texas_color_swapWe held our eighth DevOpsDays Austin last month! DevOpsDays Austin 2019 was held at the UT Austin stadium for two days full of talks, openspaces, and so on. All the videos of the sessions are up on YouTube in the DevOps Austin channel that holds other years’ videos as well.

Here’s my top 10 countdown list of great things about this year’s DevOpsDays Austin!

2019-05-03 09.49.41

Platinum Sponsor Suite

10. We brought the sponsor room back, and added platinum suites in the stadium luxury boxes so sponsors that wanted to hold sessions could do so. There were very well attended sessions in these suites!

9. We had two content tracks and a new “Conversations” talk format – a short 20 minute talk followed by a linked openspace for interactive demos and discussions and command line stuff that doesn’t do well in a talk session. We only had space for a handful of them but they were very highly rated and we’re considering shifting significantly towards them next year.

8. We made the happy hour more modest and onsite, but with DevOps Trivia from Patrick Debois!  We had a bunch of teams compete and it was a wild and woolly time. We even used Patrick’s zender.tv online trivia thing to let people outside the venue compete.

2019-05-03 17.58.58

The remnants of the cupcakes

7. Our fine venue, food, and drink team and vendors… We ripped into some mini cupcakes at snack time!!!

6. The openspaces.  I actually got to attend some this year instead of just running around working.  And they were all brilliant.

5. Our organizers! We bestowed the title of MVP organizer on two organizers this year – Daria Ilic for her great job with communication and Dan Zentgraf for doing a yeoman job with the sponsors.

Special thanks to all the DevOpsDays Austin 2019 organizers: James Wickett (Speakers), Peco Karayanev (Speakers), Karthik Gaekwad (Swag), Daria Ilic (Marketing, Volunteers), Dan Zentgraf (Sponsors), Tom Hall (Sponsors), Boyd Hemphill (Volunteers), Scott Baldwin (Web site), Lee Thompson (AV), Carl Perry (AV), Ian Richardson (Attendees), Chris Casey (Signage and Slides), Richard Boyd (Venue, Food, Happy Hour), Asif Ahmad (Venue, Food, Happy Hour), Bailey Moore (Venue, Food, Happy Hour), and thanks to Laura from ConferenceOps for doing all our finances.

4. I let the other organizers talk me into buying the Jumbotron!  I am naturally thrifty so had resisted given the significant price tag in previous years, but we had a glut of sponsors and everyone really wanted it so I finally gave in. Karthik even changed his Slack name to JUMBOTRON to petition for it. It remains so until this very day. You  have to respect the dedication. So behold – the DevOpsDays Austin Jumbotron! (Yes, that’s real, not Photoshopped.)

2019-05-02 09.46.00

3. Check out our cool organizer swag I got each organizer this year as a thank you gift – custom Vans with the DevOpsDays Austin mascot on them!  (They’re only $80, if a little work intensive to design on their site, feel free and steal the idea!) People always love our DevOpsDays Austin shirts so I wanted to give the organizers a really distinctive way to show their pride in the event.

vans

2019-05-02 09.48.152. A very special thank you to DevOpsDays Austin from Mandy Whaley and the Cisco DevNet crew, who have been sponsors and speakers and attendees for many years.  I wasn’t expecting this – they actually used their sponsor shout-out time to present us onstage with a heartfelt card that they read to the audience.

We appreciate everything that Mandy and the team bring to the event and the card was super touching.2019-05-02-09.49.56.jpg

2019-05-02 09.50.08-1

1. What could be better than that, though, you ask? How can such a kind shout-out be number 2 on the list?

Well, we had a little problem, and that problem was a spare $25,000 from letting in the gold sponsors above our initial sponsor room cap because they really, really wanted in and we felt bad for them. DevOpsDays Austin (like all DoDs) is a non-profit, so while we keep a war chest to pay for next year’s venue and stuff, the rest has to go. Previous years we did some modest donations to the Capitol Area Food Bank; last year we actually had enough spare money so that we let each organizer do a $1000 donation to a charity of their choice. But this was quite a larger chunk, so what to do?

Some of the organizers brought up a great opportunity they knew about and had given to themselves. Here in Austin there’s a really unique program going on, the Community First! Village – a planned community that provides affordable, permanent housing and a supportive community for men and women coming out of chronic homelessness.

mobile-loaves-fishes-community-first-village-microhome-300x200

Community First! Village Micro-Home

And it turns out $25,000 is how much is needed to build a micro-home in their next phase of expansion, to house a formerly homeless person in their community. These are little 180-200 square foot homes with electricity but no plumbing that are the foundation of their village. The whole organizer team got super excited about this opportunity.

So that’s what we did – we sponsored one of these homes to be built. We’re pleased to have the ability to help Austin in a permanent way out of the conference!

I’m going to do a separate blog post on this because it’s an awesome program that many companies in Austin have been getting behind, and it’s remarkably successful in helping our large homeless population. But thanks so much to all the sponsors and attendees that made this possible.

2019-06-08 10.21.02

DoD Austin Organizer (and Family) Tour of the Community First! Village

We had a great time at DevOpsDays Austin this year and hope many of you did too. Next, we’ll publish a full retrospective that we hope some of you and other DevOpsDays organizers will find interesting.

Leave a comment

Filed under Conferences, DevOps

Cloud Native security

It’s been a whirlwind few months, and I’ve been terrible about posting to the blog! Had a great time at RSA this year with fellow agile admins James and Ernest, and got to meet many of the folks I really look up to in the security industry. I attended devsecops days as well, which was eye opening to see security folks want to shift more left and work with developers and operations.

Also, I had a chance to present about the different things going on with respect to security in the cloudnative world. Here’s a copy of the slides.

I have more thoughts to add on this, but I’ll come back to it later today!

Leave a comment

Filed under DevOps

Want to be part of the DevSecOps Handbook?

The word is out, at RSA this week Shannon Lietz (@devsecops), James Wickett (@wickett), John Willis (@botchagalupe), and myself (Ernest Mueller, @ernestmueller) did a panel on our upcoming book, the DevSecOps Handbook.  We’re still writing it, and we want to make you a part of it!

Like the DevOps Handbook, also from IT Revolution Press, the heart of the book is case studies from practitioners like you.  Have you done something DevSecOpsey – adapted the culture of infosec/appsec to work better with your product teams, added security testing to your CI pipeline, added instrumentation and feedback loops for your security work, or other security-as-code kind of work?  Well, we want to hear from you!

We are interested in successes and failures, in both advanced implementation and people taking their first step – others will benefit from your experience in any of these cases.  You can be hardcore security dipping your toes into devops, hardcore dev or ops dipping into security, or someone getting started on the whole ball of wax. Don’t worry, we’re not asking you to write anything, we can interview you and do all the heavy lifting. Not sure if your company will sign off?  We can anonymize it, or if it’s been published publicly as conference proceedings or whatnot then journalism rules apply, we’ll just cite prior work.

To contact us, email book@devsecops.org or go to devsecops.org and fill out the form there. Or if you already know one of us, ping your favorite!

werereadytobelieveyou

We’re ready to believe you!

Leave a comment

Filed under DevOps, Security

SRE: The Biggest Lie Since Kanban

There is a lot of discussion lately about how SRE fits into or competes with or whatever-s with DevOps.  I’m scheduled to speak on a “SRE vs DevOps Smackdown” panel today here at Innotech Austin, and at the exact same time I see Bridget tweeting Liz Fong-Jones’ slides from Velocity on using SRE to implement DevOps. And the more I think about it, and see what people are doing, the more I’m getting worried.

The Big Lie

Just to get the easily provoked to put up their pitchforks, I don’t dislike SRE and I don’t dislike Kanban.  The reason I call Kanban a “Big Lie” is because really doing Kanban correctly and getting the value out of it requires even more discipline that doing something like Scrum.  But it looks so close to doing nothing new that many lazy teams out there say “they’re doing Kanban” and by that they mean they’re doing nothing, but they’ve turned on the Kanban view in JIRA for your convenience.  They have no predictability, they’re not managing WIP, they’re not identifying bottlenecks – they just have a visible board now and that’s it. I strongly believe from my experience that most teams “doing Kanban” are really doing mostly nothing.  There’s articles on this blog about how I make my teams I’m teaching Agile do Scrum first if they want to get to Kanban to build up the required discipline.  And I’m not just a crank, David Hawks from Agile Velocity just told our management team the same thing yesterday, which brought this back to mind for me and spurred this article.

Because I’m starting to see the same thing with SRE.  It’s not surprising – there was and is plenty of “DevOps-washing” of existing teams out there.  Rename your ops team DevOps, done. Well, at least DevOps was able to say “it’s a methodology not a job description or group name stop it” to force deeper thought – it’s why my team at work is the “Engineering Operations” team not “DevOps”, Lee Thompson insisted on that when he set it up! But SRE – yeah, it’s a team just like your own ops team, from an “org chart” viewpoint it looks the same. So doing SRE can – and in many shops does – mean doing nothing new. You just call your existing ops team SRE and figure you’re done.

A brief personal history lesson – my last job before DevOps hit was running the Web Systems team at National Instruments, an ops team.  That’s where we agile admins met, Peco and James were both ops engineers on that team! (Karthik was a dev we worked with.) We had smart people and did ops all right.  We had automation, monitoring, we had “definition of done” standards for new services. You wouldn’t have to squint too hard to just call that team a SRE team and call it a day. But, I wouldn’t wish that job on my worst enemy. It was brutal trying to do ops for just 4-5 dev teams, and that’s with business support, some shared goals, and so on. Our quality of life was terrible, we weren’t empowered, and no matter how hard we tried, success was always right out of our grasp. When we actually started a team using DevOps thinking at NI after that, the difference was night and day, and we actually began to enjoy our jobs as ops engineers. I would hate for anyone to deceive themselves into thinking they’re getting the goodness they should be able to get from a DevOps/”real” SRE approach while still just doing it the way we were doing it.

I have a friend at a local legal software firm, who told me they’re going through and just renaming all the QA folks to SWET (Software Engineer In Test), whether they can code or not, and all the Ops folks to SREs in this manner. One might be charitable and say they’re leaning forward and they intend to loop around and back that up with retraining or something, but… will they? Probably not, it’s just a rename to the hot new term without any of the changes to help those engineers succeed more in their jobs!

SRE isn’t “an implementation of DevOps” if you just apply it as a name for a hopped-up ops team.  Properly understood, it can be an implementation of one of the three parts of DevOps, Infrastructure As Code, Continuous Integration/Deployment, and Site Reliability Engineering. But note that reliability engineering doesn’t start with deploy to production; so much of it is Michael Nygard-esque techniques to write your app reliably in the first place; reliability engineering, in usual DevOps fashion, requires dev and ops work both way back in the dev cycle and out in production to work right. It doesn’t need to be a different team.  If it is, and that team doesn’t get to decide if it takes over ops for a given app, and it’s not allowed to spend 50% of its time on reducing toil and you’re not comping SREs like you do dev engineers – it’s not SRE and you’re a liar for calling it SRE. If you don’t keep DevOps principles in mind, you’re just going to get your old ops team with its old problems again.

That’s why SRE is a Big Lie – because it enables people to say they’re doing a thing that could help their organization succeed, and their dev and ops engineers to have a better career and life while doing so – but not really do it.  Yes, there have been Big Lies before, which is why I cite Kanban as another example – but even if the new criminal is pretty much like the old criminal, you still put their picture up on the post office wall.

Frankly, anyone pushing SRE that doesn’t put warning labels on it is contributing to the problem.  “Well but it mentions in chapter 20 of the second book,” said someone responding to the first version of this article on Twitter.  Not good enough. If something you’re selling is profoundly misused it’s your responsibility to be more up front about the issues.

The Little Problems

Now there are legitimate issues to have even with the “real SRE” model, at least the way that it’s usually being described.  The Google books kinda try to have it both ways, describing it as an engineering practice (how I describe it above and in the SRE course I did for LinkedIn) and describing it as “a team that works this way.”  Even among those not SRE-washing classical ops, the generally understood model is that SRE is a org/job title for a production operations team.

There’s an issue here, the problem of specialization.  If you are Google scale, well, then you’re going to have to specialize and a separate ops team makes sense.  But – first of all, you are not Google scale.  In my opinion, if you are under 100 engineers, you are committing an error by having a separate ops team. You need your product teams to own their products. Second of all – I don’t want to make an enemy of all the lovely Google engineers out there, but is your experience with Google services that they evolve quickly and get better once they go to wide release?  It’s not mine.  They rot.  Have you used Google Hangouts lately without it ending up with cursing and moving off to someone’s Zoom? That kind of specialization still has its downsides in terms of hindering your feedback loops that let you improve (the Second Way). Is SRE just Google-ese for “sustaining?”

I get that the Google folks say they still get feedback and innovation using the SRE model, I’m sure they do and they work hard at that, but that doesn’t change the fact that running a separate ops team is making a deliberate tradeoff between innovation and efficiency. There is no way in that you get as much feedback or improve as quickly with a separate team, you can compensate for it, but you’re still saying “look… Not as important.” Which is fine if that’s your situation, I worked at many companies with 200 abandoned apps in production and you had to do something.  But “not getting there in the first place” is better.

Some of the draw of the model, and why Google is highly aligned with it, is Kubernetes itself. k8s is very complex to run drives people back a little bit to the old priest-in-the-tabernacle model of “someone maintains the infrastructure and you write the app and then you have them deploy it,” but now there’s some standards (like deploying as a container) that make that OK – I guess? But if you think reliability, and observability, are the primary responsibility of an ops team that is not involved in constructing the application, you either have deep and profound company standards that allow seamless plugging of the one into the other or you’re fooling yourself. 90% of you are fooling yourself.

At this conference I heard “Service meshes!  They get you observability so your devs don’t have to think about it.” Do you not see how dangerous that mindset is?

SRE, as interpreted as “a separate newfangled ops team,” may work for some but you need to be realistic about the issues and tradeoffs you’re making.  Consider whether product teams supporting their product, maybe with aid from a platform team making tooling and an enabling/consulting/center of excellence team that can give expert advice?  DevOps helped us see how the “throw it over the wall from dev to ops” model was profoundly harming our industry.  Throwing over the wall from dev to SRE doesn’t improve that, it’s profoundly regressive. Doing SRE “right” to compensate for this, like doing Kanban right, requires more skill and discipline, not less – be realistic about whether you have Google levels of skill and discipline in your org, eh?

Conclusion

SRE (and Kanban) aren’t bad, they have their pros and cons, but they are easy to “pretend to do” in some minimal, cargo cult-ey way that gets you little of the benefits. And if you think spinning up an ops team and calling it SRE is “an implementation of DevOps” you’ve swallowed the worst poison pill the DevOps talk circuit can deal to you.

15 Comments

Filed under DevOps

3 Features I love in Kubernetes 1.11

Originally published in the cloudnative blog on July 3rd

Kubernetes 1.11 was released last week, and I spent some time looking at the features and fixes released. It’s the 2nd Kubernetes release this year, and this one comes with a lot of cool features to try out. You can take a look at the release notes here, or if you want to get down in the weeds, check out the changelog.

I’m most excited about the “Dynamic Kubelet Configuration” feature! This feature existed previously but has graduated to a “beta” feature. It means that’s it’s more stable than before, and the feature is well recognized. The feature essentially allows you to change the configuration of Kubelet on a running cluster in a more accessible manner using configmaps. The configmap is saved as a part of the Node object which is monitored by Kubelet. Any changes to it and Kubelet will download the reference and stop. If you’re using something like systemd to watch Kubelet, it’ll automagically restart Kubelet, which will start with the new configuration. This feature is super exciting because it gives admins who manage all of the nodes a little break. In the past, any updates to the config had to be rolled individually to each node, which could be a time-consuming process.

I like that Custom Resource Definitions (CRD) are a lot more usable now with versioning. In the past, you were limited to a single version of a CRD; any changes, and you had to create a new one and manually convert everything that used the old CRD to the new one. All a bit painful! With versioning, the path to using updated custom resources is more straightforward than before.

Finally, CoreDNS was promoted to General Availability! In the early Kubernetes years, there some confusion on what DNS provider to use, and there were a few options. For someone who was looking at the ecosystem from the outside, it was hard to tell what DNS solution to pick. I touched on this in my Kubernetes: CNCF Ecosystem course, and how the CNCF was able to steer the community to a better default! It took some time, but in the end, having CoreDNS as a default DNS server will help Kubernetes be more reliable, and make DNS debugging simpler for those of us dealing with the inner workings of K8s.

There are a lot more things released, so check out the release announcement if you haven’t already!

There are also a few tiny things that were released that have me excited:

First, this PR allows for Base64 decoding in a kubectl get command using go-templates. Super useful to have a one-liner to decode what something might be in a secret.

Second, from a monitoring perspective, Kubelet will expose a new endpoint, /metrics/probes. This presents a new Prometheus metric that contains the liveness and readiness probe results for the Kubelet. It will allow you to build better health checks and get a better picture of the state of your cluster.

Third, RBAC decisions are in the audit logs as audit events! Since I’ve worked on authn and authz systems in the past, I get irrationally excited about stuff like this. In the past, we’d have to go hunting through logs to find why an RBAC call passed/failed, whereas now we can quickly look at the audit events stream.

That’s my (biased) list! What about you? What feature or bugfix has you excited? Let me know in the comments below, or tweet at me @iteration1!

Leave a comment

Filed under DevOps

Cloud-native helloworld

wood_3200402_1920-1
Originally published  on cloudnative labs on June 28th, 2018

Speaking and writing come pretty naturally to me, but setting a title is always the hardest part. It’s true while writing code as well- writing 1000 lines of code comes naturally, but when I have to create and name a new file, it’s a different story…

But, I digress- Hi! I’m Karthik Gaekwad, and I’m the newest member of the Developer Relations team here at Cloud-native labs. If you live in Austin, we’ve probably already crossed paths at one of the many meetups I attend or run including CloudAustinAustin DevopsDocker Austin, OWASP, etc; or perhaps at Devopsdays Austin, for which I’ve been one of the core organizers since its inception in 2012. I’m also an author on Lynda.com, and have authored a few courses on Kubernetes, and Agile devops methodologies.

I’m joining the Cloud-native labs team from the Oracle Container Engine team- which is Oracle’s managed Kubernetes service running on Oracle Cloud Infrastructure. Naturally, I’ll be focusing my efforts on Kubernetes, microservices and Cloud Native architectures and applications.

There are many things I’m excited about with the new job, but I’m most excited to learn and teach! The one constant theme that I’ve noticed with Kubernetes over the last few years since it got hot is the word “How?”. As a user of Kubernetes, I’ve frequented in the Kubernetes doc searching for answers, and as a Lynda author, I’ve received many messages of thanks from viewers that they now knew how to use Kubernetes. The cloud-native ecosystem is one of the fastest growing ecosystems I’ve seen, and it’s hard to keep up with the changes, new releases, and new projects that support the ecosystem. As a result, I’m excited to spend more time keeping pace with all the new happenings and spend time researching best practices for microservices and cloud-native apps, welcome new users to the world of K8s, and bridge the gap between the cloud-native platforms we have on OCI today.

I’ll be spending a lot of time researching, speaking, blogging and answering questions! Feel free to reach out to me on TwitterLinkedin or comment on here as well- I’m here for you!

-Karthik

Leave a comment

Filed under DevOps

DevOpsDays Austin 2018 Videos Posted

Well, we were “unplugged,” but we managed to smuggle videos out anyway for your pleasure… Watch ’em, like ’em, comment to the speakers that you appreciate them giving to the global tech community!  Especially since this year they weren’t pre-selected, voting on talks was done at the event, so these folks prepared a talk but weren’t for sure to give it, which takes guts!

Leave a comment

Filed under Conferences, DevOps