Monthly Archives: June 2022

DevOps War Story – The VMWare Realization

Hey all! There are some stories that were foundational in my path to DevOps that I think illustrate larger issues pretty well. I use them a lot when I do talks or whatever but I figured I might as well put some of them down here to inspire readers.

I was working at National Instruments in the IT department back in the early 2000s. I ran the Web Systems team, alongside a bunch of technology focused teams – the UNIX Admins, the Windows Admins, Notes Admins, Network Admins, DBAs, and various others.

It took 6 weeks to get a new server for the Web. We’d say what we wanted, but then the UNIX admins would spec it out, and then order it from Dell, wait 2 weeks for fulfillment, then it’d come, then the network team would rack and jack it in the server room, the UNIX admins would put the OS on it, various other teams would get their pound of flesh, and then eventually it would get sent to us so we could put our software on it and then let the app teams use it. Not exactly quick turnaround.

Then came VMWare, the shining new star! Their sales rep showed up and said “You know, you can have a new server in 15 minutes.” We went out to a live sales demo and were like “this is very cool.” I asked the sales team, “so I could make the root read only for security reasons and write everything to external storage right?” They thought a minute, never having considered that. “Yes? That’s actually a really good idea.” A coworker and I fist-bumped each other.

Anyway, so we bought it, yet another technology vertical team was formed to own it, and after some degree of enterprise folderol it was in place.

So now when I needed a new server, guess how long it took me?

Four weeks. All we had removed was the 2 weeks of Dell fulfillment. Not 15 minutes.

When the procurement was slow anyway, it was just kind of accepted that 2 weeks turned into 6 weeks in the process. “Well, someone has to carefully assign an IP address, or else everything would be chaos!” But then when it was 15 minutes turning into 4 weeks purely based on our own internal friction (the actual work involved being much less than a day end to end) – suddenly it became clear how much of our own problem we were generating.

But it was hard to fix. There is inevitably niche protection in an environment like that. “Hey UNIX admins, you put cfengine on the boxes to manage them? Can we use that to load on our software too?” “No.” “Hey, can we get our own Web DBA so Web initiatives aren’t always bottom-feeding off the big ERP initiatives?” “No.” Tossing a ball over walls from silo to silo inevitably resulted in complex handoff processes and wait times and resource contention.

So then when we moved to the R&D department and started working up some SaaS products, and we were faced with the option of using our own integral IT department – we said “Absolutely not. There’s this new Amazon Web Services thing and we’re going to do it all ourselves.” “But what about all the value those specialists bring?”

Well, it’s not the specialists’ fault – but the one day worth of specialty benefit they brought was stapled to 4 weeks of sitting on hands, and no one felt it was their role to fix that endemic process problem in the org. I don’t care if you’re offering me Linus Torvalds to fix my Linux problems, if you offer me “a day of Linus and then 4 weeks in solitary,” I’ll do without, thank you very much.

This was 2009, it was before “DevOps,” but we immediately realized that having a product team responsible end to end for the product – code, systems, security, etc. – allowed us to deliver high quality in a fraction of the time that we were used to. It was intoxicating. That’s actually where we Agile Admins met and why we still hang out together today, because we all went through the experience of the gig actually becoming fun again!

Leave a comment

Filed under DevOps

DevOpsDays Austin Hits $100,000 In Charitable Donations

DevOpsDays Austin believes in supporting the local community that supports us, the techies of Austin. As we’ve grown and we’ve been able to end some of our years with extra money from our sponsors. We are careful to be thrifty with the event and rely on our volunteers to do a lot, and all DevOpsDays events run on a nonprofit basis; one of the few core rules of all DevOpsDays is that the events are not for individual or corporate profit.

The Austin technical community is a cross-section of, and part of, our community. We have a diverse set of individuals from many backgrounds and neighborhoods all around the area. As technologists we are largely blessed with decent salaries and technology companies have lots of money, and many of the keenest needs in the area need relatively little funding to make a real difference. We believe it’s our responsibility as a part of the community to use part of what we make to give back to support the most vulnerable among us. Therefore we’ve made it a point to use our excess funds to give back to charities that directly help those most in need.

We started DevOpsDays Austin in 2012 from nothing, and relied on a free venue (courtesy of my employer at the time, National Instruments). We made enough to make a down payment on a venue in 2014, and by 2015 we were confident enough of our finances that we considered our first charitable donation.

The Charity Page In Our 2022 Yearbook

The natural choice was the Central Texas Food Bank (at the time, known as the Capital Area Food Bank), a well known long time Austin charity that combats hunger in the area. We gave $5000, and we also did a charity drive at the event, handing out leftover t-shirts and swag from the previous two years to those who made a donation of their own, which sent another $2600 to the food bank.

In 2016 we moved to a new, larger, much more expensive venue (the Darrell K. Royal Texas Memorial Stadium out at UT Austin) so we could let more people attend DevOpsDays. That put us at the limit of our finances for a couple years, especially with our 2017 “Monsters of DevOps” blowout conference with many international speakers and great fun events. But by 2018 we had that venue dialed in as well and had $20,000 left over that we decided to donate to charity. That year, instead of sending all the donation to one charity, we let each of our 20 organizers send $1000 to a charity of their choice. This ended up driving our helpful accountant Laura at ConferenceOps batty trying to get proper tax receipts from everyone, however, and we promised her we wouldn’t do that again.

Then in 2019 we had a great event, sponsors were in a right frenzy to get in and we had to add more sponsor booths to accommodate them – which was a lot of work, but left us also with a lot of leftover money ($25,000, off a conference with only a $200k total budget). We weren’t sure how we could be the most effective with a gift like this, and one of our organizers said “Did you know… There’s a new project here in Austin that builds miniature houses for the homeless in a beautiful community?” And that’s how we were introduced to the Community First! Village, a quickly growing and very effective outgrowth of Mobile Loaves and Fishes to house the homeless. And it turns out $25,000 is exactly how much it takes to build one of those homes. Our organizers enthusiastically approved the donation and we went out and did a great tour of the site, and many of us have returned as volunteers since.

And then the hard times came. During the pandemic, DevOpsDays Austin was on ice. In fact, we had planned on moving to an even more expensive hotel venue and had a down payment in place when the lockdown came, and we had to get a lawyer into play to get our deposit back.

But the needs of the community weren’t on hold. We have many Black and brown technologists in our community, and the high profile brutality directed at them was completely unacceptable. Long time friend of DevOpsDays Austin, John Wills, started a fundraiser for Black Lives Matter around making a quilt with all his DevOpsDays shirts (many of which were from Austin) in 2020, and we felt compelled to donate $2000 of the more than $12,000 he raised.

DevOpsDays Shirt Quilt

Then we were in the long night of lockdown. We weren’t doing anything from a DevOpsDays Austin perspective in 2021, though there was a virtual DevOpsDays Texas conference to fill in some of the gap. But as jobs and aid dried up, hunger became a critical need again in Austin. Fellow organizer Boyd Hemphill encouraged people to help out and volunteer during a virtual meetup, and his words made my conscience burn till I brought it to my fellow DevOpsDays Austin organizers to see if we could dip into our reserves and help. They all enthusiastically approved a $10,000 donation to our old friends the Central Texas Food Bank again.

Two donations without any revenue, that’s good enough till we can have an event again, right? Well, you’d think, and then Russia went and invaded Ukraine.

While we’re an Austin organization and we’ve always given to help out in Austin directly, we have many Ukrainians as part of our local tech community. I worked with many of them hand in hand while I was running teams at Bazaarvoice, as we had a great relationship with the Ukrainian consulting company Softserve. We brought many of them here to Austin to work with us, we went out together, I had toasted them with “Slava Ukraini!” Many of our organizers had similar experiences. And since we don’t like bullies around here, that riled us up. After a discussion along the lines of “well, we started from nothing once, we can do it again if we have to,” we donated $10,000 to Ukrainian relief organizations Razom for Ukraine and Nova Ukraine.

And that brings us up to date with the past, but we finally managed to have DevOpsDays Austin again! In May, we got a great venue (the University of Texas Alumni Center, at half price courtesy of Bill our venue lead being an UT alum) and planned for a slightly smaller than usual (350 masked attendees, to hedge against super-spreading) conference on our 10 year anniversary – the DevOpsDays Austin 10 Year Class Reunion.

Since it was our 10th anniversary, we did a yearbook. And when I put the charitable donation page together for the yearbook, I realized we’d given $72,000 to charity over the years. 10 years and an even $100,000 sounded mighty nice.

The conference went great, and all those sponsors have been saving up their marketing money wondering what to do with it. After some laborious running of numbers I realized we could free up the $28,000 donation to get to that bar and leave enough for us to make a venue down payment the next year.

As we contemplated this year’s donation our Texas state government decided to openly attack the LGBT+ citizens of our state. Many of those in our technical community we meet with every month in user groups are gay, lesbian, transgender, binary, and so on, and this is a direct attack on many of our coworkers, colleagues, and friends. And not just them, but their children.

As a result we gave $14,000 to The Trevor Project, a national service that provides suicide prevention hotlines for LGBT+ youth, $14,000 to Out Austin, a local place for youth of all sexual orientations and gender identities. I’ll write a separate post about those organizations and why them, and more importantly how you can help.

But in the end we’ve been very happy that we’ve been able to use our position as techies in the tech hotspot of Austin to consistently give back. We’d challenge other conferences, tech companies, and individual technologists to do so as well – especially to reputable charities that directly help those who need it.

I hope that DevOpsDays Austin can continue to give back in this way over the next ten years too!

Leave a comment

Filed under Conferences, DevOps

DevOpsDays Austin is Back For 2022!

Your DevOpsDays Austin 2022 Organizers!

Well, we had to skip 2 years in a row due to the pandemic, but we were finally able to have DevOpsDays Austin in person again in 2022! It’s our tenth anniversary of DoD Austin, we had the first one at National Instruments back in 2012, one of the early DevOpsDays in the US.

We had to move to a new venue, and used the beautiful University of Texas Etter-Harbin Alumni Center (our site lead Bill is a UT alum which makes it half price!). The Etter-Harbin Center is right across from the stadium where we had DoDA in the years leading up to the hiatus. It has plenty of great outdoor spaces, which we used for lunches and happy hours, as well as a great main ballroom with views of the outside. It worked great for our target of 350 attendees, and we think we could make it work for more in the future.

We were thinking and came across the perfect theme – it’s our tenth anniversary, and we’re just back from the pandemic, and DevOps is also just a little more than ten years old and at a weird inflection point that has people asking “is DevOps dead? Where does DevOps go from here?” So we decided that since we were also in the Alumni Center, the obvious theme was our 10 Year Class Reunion!

We don’t take our themes lightly at DevOpsDays Austin. We settled on a new theme for our talks – instead of the normal RFC for whatever technical and culture topics, we required all talks to be a retrospective format – reflecting on what you’ve learned over the years of DevOps and what you think the future holds. We had lots of great speakers, many of whom are long time parts of the DoD Austin community, both locals like Rob Hirschfeld, Christa Meck, Ross Dickey, and Victor Trac, as well as folks from other parts of the Earth like Patrick Debois, Damon Edwards, J. Paul Reed, Pete Cheslock, and Michael Cote, who all frequently come to Austin to share with us.

And we printed a yearbook, with pics of speakers from all the events, our tshirts over the years, and more! Very snazzy, and we had people sign each others’ yearbooks to add some fun to the hallway track. In fact, you can view the yearbook online and buy a hardcopy here if you want!

The DevOpsDays 2022 Yearbook

We did require COVID protocols – masking inside and (honor system) vaccine or test, and while it is a bummer to not see each others’ faces, it also resulted in only one person I know of getting COVID the week after, so well worth it.

We didn’t have to worry about sponsor interest! We sold out quickly. Here’s the ones I got snaps of!

Everything went great, and it was super to finally get back together and interact with our local DevOps community. J. Paul Reed led a great session where a retro was done on DevOps in general!

And one of the best things is that we managed to carry on our tradition of giving our excess proceeds to charity! I’ll do a separate post on that, but the short form is that we contributed $28,000 to LGBT-supporting charities, half to The Trevor Project and half to Out Youth here in Austin, bringing us to $100,000 given to charity over our 10 years in existence! Stay tuned for more details on that…

1 Comment

Filed under Conferences, DevOps