Category Archives: DevOps

Pertaining to agile system administration concepts and techniques.

DevOpsDays Austin 2022 Retrospective

DevOpsDays Austin was back in person in 2022 for our 10 year anniversary! So how’d it go? We are committed to transparency and metrics here so enjoy our retrospective.

Financially it went well; we sold 317 tickets and had 296 people show up (an almost unheard of 93% show rate), we were able to donate $28,000 to support the LGBT+ members of our community via Out Youth and The Trevor Project, bringing us to $100,000 donated to charity total over the history of the event, and still ended up with a $1000 increase in our bank account.

We also send out retrospective surveys to our attendees, speakers, sponsors, volunteers, and even our fellow organizers to find out how we’re doing and get an idea of what we should do better on (or keep doing) in the future!

Here’s the deeper details if you want them: DevOpsDays 2022 Retrospective

To sum up, however, it’s looking good! Our last surveys were from 2019, from our last pre-pandemic DevOpsDays Austin, so we have a previous number to compare to.

Our attendee NPS was up to 77 (44 responses) from 62 (50 responses). And the things people loved were, basically, the personal interaction. Community, people, discussions, and openspaces were the most cited positives by far. We knew people had been missing that for a couple years now so our retrospective format and event plan were specifically designed to promote small group interactions.

The gripes were more varied. Primary was the food, which is fair enough. While we don’t intend to change from a boxed lunch format – it leaves so much more time for the actual conference, so we left fancy catered lunches behind long ago – we were forced to use the venue caterer and they ran out of food especially veggie options, and we had asked for breakfast tacos both mornings and on day two the breakfast was what I can only call “leftover meat bits.” So room for improvement, with the understanding that boxed lunches are here to stay, but we’ll definitely see what we can do about better options and especially making sure there’s not shortages of anyone’s dietary needs.

The other leading concern we’re just plain going to ignore, and here’s why. It was “the retro format – but what about technical talks, what about content for newbies?”

At DevOpsDays Austin we explicitly reject the assumption that all events must be the same generic thing every time. We specifically change our format every year. We’ve had the Monsters of DevOps where we went for flying in big keynoters (including all the authors of the DevOps Handbook) and doing everything up huge; we’ve had DevOps Unplugged where talks were voted in on site and there were no sponsor tables. This year we had a “class reunion” format with talks only being 20 minute “retrospectives” from what the speaker’s learned over their time in DevOps (some speakers were experienced, others were new voices). We very, very clearly talked about this format on our website, social media, and emails to attendees and sponsors. In the end, people just don’t read, and there’s nothing really to do about that. And we won’t be doing the retro format two years in a row, we’ll keep mixing it up!

Organizer feedback was good (we have 20 organizers), up slightly, with everyone enjoying working with the group, and some concerns about unclear roles and roles already taken. That’s always a challenge – we have a lot of organizers but not all of them are up to actually leading something. We have people volunteer to own roles and then encourage them to reach out to the others/the others to chip in when we need something, but that doesn’t always go well, which is frustrating for everyone. In the end, most roles need someone who can commit to consistent participation over the planning period (there’s a couple specialty roles like making signage that can be backloaded, but not many). But we want to be inclusive and not tell people “no, take the year off if you can’t be putting in a couple hours a week including making the organizer calls, and truly own something.” We’ve wrestled with this for 10 years, no clear answer is in sight.

Speakers love speaking at the event. NPS was 92, slightly up from 90 in 2019. They love the audience, how supportive and welcoming they are, and how low stress and chill the experience is. There’s always some AV issues as a fly in the ointment – we do AV checks but not everyone shows up for them.

Volunteers have a good experience too. NPS was 88, slightly down from 94 but still good; we try to make sure that the load isn’t too much on any given volunteer so they can also enjoy the event. Posting the openspace topics is always a challenge each year; we tweet out photos and then desperately type them into the sessionize, but a bunch of attendees are social media impaired I guess and it’s hard to get the schedule to everyone, but there’s not a lot of options given that openspaces are predicated on doing the agenda immediately previous; I’m not sure more time would help short of printing out copies or having live monitors everywhere.

Sponsor feedback was down from 60 to 50 NPS. They do like the audience and authentic content. The main problem was the new venue and unclear flow meant that the platinum sponsor rooms were more out of the way than we’d planned (we gave them tables in the gold area as well once this became clear). And then the general sponsor gripes about it not being a good lead gen event. We always tell sponsors this is a good participation event, not a good lead gen event, no badge scanners, no sponsor list, etc., but a previously mentioned people don’t read, plus the teams being sent out aren’t the people buying the sponsorships and often just assume they’ll be getting a standard conference experience. We sell out every year so I’ll worry about it when that stops.

There’s one other thing worth mentioning, which is that we did require masking at the event and asked people to either be vaccinated or test prior to the event.

One the one hand, a couple sponsors and attendees griped about the masking.

On the other hand, despite other events resulting in superspreading (Kubecon EU, RSA, even some DevOpsDays events):

So, with all due respect, we are very happy with our choice and that we had a safe event. No one likes wearing masks. If you don’t like it enough to not come – don’t come. Hopefully it won’t be necessary next year.

Everything was pretty good! There was one issue, though – in all the survey sub-questions, there was a drop in the perceived friendliness of the organizer team, so we’re going to make some changes there – stay tuned to hear what!

Leave a comment

Filed under Conferences, DevOps

DevOpsDays Austin Donates to Out Youth and The Trevor Project To Support LGBTQ+ Youth

We were psyched to be back and meeting in person for DevOpsDays Austin 2022 this year! Charitable donation is a part of our Austin tech culture and very important to us, and since it was our 10th anniversary we aimed to hit a total of $100,000 in charitable donations since we started the event in 2012. And we did it! We’re happy to have $28,000 to give to local charities after our event this year.

That left us with the question of who to donate to. We like to choose things that fill the greatest need in our community at the time, and strongly bias towards supporting Austin area charities. Our state government decided to help us make our decision by starting a pogrom of discrimination against LGBTQ+ Texans.

Many of our colleagues in the Austin technology community are gay, lesbian, transgender, nonbinary, or identify with other nontraditional gender identities and sexual preferences, or have family members that are. I myself have a transgender son who I’ve loved and supported through his transition, and now he’s a happy, healthy adult. We find the attempts of the Texas government to institutionalize hostile behavior towards them deeply unacceptable and want to find ways to support them.

We looked initially at charities like Equality Texas that are working to ensure the rights of LGBTQ+ Texans, but as we discussed we wanted our funds to go directly to the benefit of people in need, and not all go to the lawyers fighting the long fight.

Since $28,000 is a lot, we decided to split it up into two $14,000 donations. After some research we nominated two recipients, Out Youth and The Trevor Project, and brought them to the DoDA organizer team for a vote, which they enthusiastically approved.

We selected Out Youth for our first donation as a deeply local Austin organization directly supporting LGBT youth.

For 32 years, Out Youth has supported Central Texas LGBTQIA+ youth and young adults by providing safe places where they are loved, acknowledged, and accepted for exactly who they are. Their life-saving and life-changing programs and services ensure these promising young people develop into happy, healthy, successful adults. Out Youth hosts a variety of programs to keep their youth mentally and physically safe such as drop-in times at their youth community center, by offering free individual counseling and group therapy sessions, and through their in-school-support services. To find out more information about ways to get involved and about their services, please visit outyouth.org today.

It’s funny that our relationships with these charities usually start with us showing up with a big surprise donation and then after that getting deeply involved with the organization; we plan to go tour their house and spread the word about volunteer opportunities with them to the Austin technology community.

Not only do I have a transgender son, but also fellow Agile Admin and conference organizer James lost a brother to suicide. So while The Trevor Project isn’t Austin based per se, it does help many Austinites, and its mission of suicide prevention among LGBTQ+ youth is deeply and personally meaningful to us both. Therefore we chose them for our second donation this year.

The Trevor Project has worked to save young lives by providing support through our free and confidential crisis programs on platforms where young people spend their time — online and on the phone: TrevorLifeline, TrevorChat and TrevorText. We also run TrevorSpace, the world’s largest safe space social networking site for LGBTQ youth, and operate innovative education, research, and advocacy programs.

  • The Trevor Project’s research has found that having at least one accepting adult in an LGBTQ young person’s life reduces their risk of suicide by 40%.
  • Transgender and nonbinary youth who reported having pronouns respected by all of the people they lived with attempted suicide at half the rate of those who did not have their pronouns respected by anyone with whom they lived.
  • “You are lovable” – this is one of the most common phrases The Trevor Project’s crisis counselors share with youth in crisis.
  • According to Trevor’s research 42% of LGBTQ youth seriously considered attempting suicide in the past year, including more than half of transgender and nonbinary youth.

You can sign up to become a volunteer counselor on their site; there’s extensive training and it requires a year commitment.

In closing, we appreciate the work Out Youth and The Trevor Project are doing and hope that others will look into finding ways to support them as well.

To the LBGTQ+ technologists in Austin, you are welcome, and both DevOpsDays Austin and other user groups we run like CloudAustin have published codes of conduct that don’t allow any hostile behavior towards you at our events, and we look forward to interacting with you there. Happy Pride Month!

Leave a comment

Filed under Conferences, DevOps

DevOps War Story – The VMWare Realization

Hey all! There are some stories that were foundational in my path to DevOps that I think illustrate larger issues pretty well. I use them a lot when I do talks or whatever but I figured I might as well put some of them down here to inspire readers.

I was working at National Instruments in the IT department back in the early 2000s. I ran the Web Systems team, alongside a bunch of technology focused teams – the UNIX Admins, the Windows Admins, Notes Admins, Network Admins, DBAs, and various others.

It took 6 weeks to get a new server for the Web. We’d say what we wanted, but then the UNIX admins would spec it out, and then order it from Dell, wait 2 weeks for fulfillment, then it’d come, then the network team would rack and jack it in the server room, the UNIX admins would put the OS on it, various other teams would get their pound of flesh, and then eventually it would get sent to us so we could put our software on it and then let the app teams use it. Not exactly quick turnaround.

Then came VMWare, the shining new star! Their sales rep showed up and said “You know, you can have a new server in 15 minutes.” We went out to a live sales demo and were like “this is very cool.” I asked the sales team, “so I could make the root read only for security reasons and write everything to external storage right?” They thought a minute, never having considered that. “Yes? That’s actually a really good idea.” A coworker and I fist-bumped each other.

Anyway, so we bought it, yet another technology vertical team was formed to own it, and after some degree of enterprise folderol it was in place.

So now when I needed a new server, guess how long it took me?

Four weeks. All we had removed was the 2 weeks of Dell fulfillment. Not 15 minutes.

When the procurement was slow anyway, it was just kind of accepted that 2 weeks turned into 6 weeks in the process. “Well, someone has to carefully assign an IP address, or else everything would be chaos!” But then when it was 15 minutes turning into 4 weeks purely based on our own internal friction (the actual work involved being much less than a day end to end) – suddenly it became clear how much of our own problem we were generating.

But it was hard to fix. There is inevitably niche protection in an environment like that. “Hey UNIX admins, you put cfengine on the boxes to manage them? Can we use that to load on our software too?” “No.” “Hey, can we get our own Web DBA so Web initiatives aren’t always bottom-feeding off the big ERP initiatives?” “No.” Tossing a ball over walls from silo to silo inevitably resulted in complex handoff processes and wait times and resource contention.

So then when we moved to the R&D department and started working up some SaaS products, and we were faced with the option of using our own integral IT department – we said “Absolutely not. There’s this new Amazon Web Services thing and we’re going to do it all ourselves.” “But what about all the value those specialists bring?”

Well, it’s not the specialists’ fault – but the one day worth of specialty benefit they brought was stapled to 4 weeks of sitting on hands, and no one felt it was their role to fix that endemic process problem in the org. I don’t care if you’re offering me Linus Torvalds to fix my Linux problems, if you offer me “a day of Linus and then 4 weeks in solitary,” I’ll do without, thank you very much.

This was 2009, it was before “DevOps,” but we immediately realized that having a product team responsible end to end for the product – code, systems, security, etc. – allowed us to deliver high quality in a fraction of the time that we were used to. It was intoxicating. That’s actually where we Agile Admins met and why we still hang out together today, because we all went through the experience of the gig actually becoming fun again!

Leave a comment

Filed under DevOps

DevOpsDays Austin Hits $100,000 In Charitable Donations

DevOpsDays Austin believes in supporting the local community that supports us, the techies of Austin. As we’ve grown and we’ve been able to end some of our years with extra money from our sponsors. We are careful to be thrifty with the event and rely on our volunteers to do a lot, and all DevOpsDays events run on a nonprofit basis; one of the few core rules of all DevOpsDays is that the events are not for individual or corporate profit.

The Austin technical community is a cross-section of, and part of, our community. We have a diverse set of individuals from many backgrounds and neighborhoods all around the area. As technologists we are largely blessed with decent salaries and technology companies have lots of money, and many of the keenest needs in the area need relatively little funding to make a real difference. We believe it’s our responsibility as a part of the community to use part of what we make to give back to support the most vulnerable among us. Therefore we’ve made it a point to use our excess funds to give back to charities that directly help those most in need.

We started DevOpsDays Austin in 2012 from nothing, and relied on a free venue (courtesy of my employer at the time, National Instruments). We made enough to make a down payment on a venue in 2014, and by 2015 we were confident enough of our finances that we considered our first charitable donation.

The Charity Page In Our 2022 Yearbook

The natural choice was the Central Texas Food Bank (at the time, known as the Capital Area Food Bank), a well known long time Austin charity that combats hunger in the area. We gave $5000, and we also did a charity drive at the event, handing out leftover t-shirts and swag from the previous two years to those who made a donation of their own, which sent another $2600 to the food bank.

In 2016 we moved to a new, larger, much more expensive venue (the Darrell K. Royal Texas Memorial Stadium out at UT Austin) so we could let more people attend DevOpsDays. That put us at the limit of our finances for a couple years, especially with our 2017 “Monsters of DevOps” blowout conference with many international speakers and great fun events. But by 2018 we had that venue dialed in as well and had $20,000 left over that we decided to donate to charity. That year, instead of sending all the donation to one charity, we let each of our 20 organizers send $1000 to a charity of their choice. This ended up driving our helpful accountant Laura at ConferenceOps batty trying to get proper tax receipts from everyone, however, and we promised her we wouldn’t do that again.

Then in 2019 we had a great event, sponsors were in a right frenzy to get in and we had to add more sponsor booths to accommodate them – which was a lot of work, but left us also with a lot of leftover money ($25,000, off a conference with only a $200k total budget). We weren’t sure how we could be the most effective with a gift like this, and one of our organizers said “Did you know… There’s a new project here in Austin that builds miniature houses for the homeless in a beautiful community?” And that’s how we were introduced to the Community First! Village, a quickly growing and very effective outgrowth of Mobile Loaves and Fishes to house the homeless. And it turns out $25,000 is exactly how much it takes to build one of those homes. Our organizers enthusiastically approved the donation and we went out and did a great tour of the site, and many of us have returned as volunteers since.

And then the hard times came. During the pandemic, DevOpsDays Austin was on ice. In fact, we had planned on moving to an even more expensive hotel venue and had a down payment in place when the lockdown came, and we had to get a lawyer into play to get our deposit back.

But the needs of the community weren’t on hold. We have many Black and brown technologists in our community, and the high profile brutality directed at them was completely unacceptable. Long time friend of DevOpsDays Austin, John Wills, started a fundraiser for Black Lives Matter around making a quilt with all his DevOpsDays shirts (many of which were from Austin) in 2020, and we felt compelled to donate $2000 of the more than $12,000 he raised.

DevOpsDays Shirt Quilt

Then we were in the long night of lockdown. We weren’t doing anything from a DevOpsDays Austin perspective in 2021, though there was a virtual DevOpsDays Texas conference to fill in some of the gap. But as jobs and aid dried up, hunger became a critical need again in Austin. Fellow organizer Boyd Hemphill encouraged people to help out and volunteer during a virtual meetup, and his words made my conscience burn till I brought it to my fellow DevOpsDays Austin organizers to see if we could dip into our reserves and help. They all enthusiastically approved a $10,000 donation to our old friends the Central Texas Food Bank again.

Two donations without any revenue, that’s good enough till we can have an event again, right? Well, you’d think, and then Russia went and invaded Ukraine.

While we’re an Austin organization and we’ve always given to help out in Austin directly, we have many Ukrainians as part of our local tech community. I worked with many of them hand in hand while I was running teams at Bazaarvoice, as we had a great relationship with the Ukrainian consulting company Softserve. We brought many of them here to Austin to work with us, we went out together, I had toasted them with “Slava Ukraini!” Many of our organizers had similar experiences. And since we don’t like bullies around here, that riled us up. After a discussion along the lines of “well, we started from nothing once, we can do it again if we have to,” we donated $10,000 to Ukrainian relief organizations Razom for Ukraine and Nova Ukraine.

And that brings us up to date with the past, but we finally managed to have DevOpsDays Austin again! In May, we got a great venue (the University of Texas Alumni Center, at half price courtesy of Bill our venue lead being an UT alum) and planned for a slightly smaller than usual (350 masked attendees, to hedge against super-spreading) conference on our 10 year anniversary – the DevOpsDays Austin 10 Year Class Reunion.

Since it was our 10th anniversary, we did a yearbook. And when I put the charitable donation page together for the yearbook, I realized we’d given $72,000 to charity over the years. 10 years and an even $100,000 sounded mighty nice.

The conference went great, and all those sponsors have been saving up their marketing money wondering what to do with it. After some laborious running of numbers I realized we could free up the $28,000 donation to get to that bar and leave enough for us to make a venue down payment the next year.

As we contemplated this year’s donation our Texas state government decided to openly attack the LGBT+ citizens of our state. Many of those in our technical community we meet with every month in user groups are gay, lesbian, transgender, binary, and so on, and this is a direct attack on many of our coworkers, colleagues, and friends. And not just them, but their children.

As a result we gave $14,000 to The Trevor Project, a national service that provides suicide prevention hotlines for LGBT+ youth, $14,000 to Out Austin, a local place for youth of all sexual orientations and gender identities. I’ll write a separate post about those organizations and why them, and more importantly how you can help.

But in the end we’ve been very happy that we’ve been able to use our position as techies in the tech hotspot of Austin to consistently give back. We’d challenge other conferences, tech companies, and individual technologists to do so as well – especially to reputable charities that directly help those who need it.

I hope that DevOpsDays Austin can continue to give back in this way over the next ten years too!

1 Comment

Filed under Conferences, DevOps

DevOpsDays Austin is Back For 2022!

Your DevOpsDays Austin 2022 Organizers!

Well, we had to skip 2 years in a row due to the pandemic, but we were finally able to have DevOpsDays Austin in person again in 2022! It’s our tenth anniversary of DoD Austin, we had the first one at National Instruments back in 2012, one of the early DevOpsDays in the US.

We had to move to a new venue, and used the beautiful University of Texas Etter-Harbin Alumni Center (our site lead Bill is a UT alum which makes it half price!). The Etter-Harbin Center is right across from the stadium where we had DoDA in the years leading up to the hiatus. It has plenty of great outdoor spaces, which we used for lunches and happy hours, as well as a great main ballroom with views of the outside. It worked great for our target of 350 attendees, and we think we could make it work for more in the future.

We were thinking and came across the perfect theme – it’s our tenth anniversary, and we’re just back from the pandemic, and DevOps is also just a little more than ten years old and at a weird inflection point that has people asking “is DevOps dead? Where does DevOps go from here?” So we decided that since we were also in the Alumni Center, the obvious theme was our 10 Year Class Reunion!

We don’t take our themes lightly at DevOpsDays Austin. We settled on a new theme for our talks – instead of the normal RFC for whatever technical and culture topics, we required all talks to be a retrospective format – reflecting on what you’ve learned over the years of DevOps and what you think the future holds. We had lots of great speakers, many of whom are long time parts of the DoD Austin community, both locals like Rob Hirschfeld, Christa Meck, Ross Dickey, and Victor Trac, as well as folks from other parts of the Earth like Patrick Debois, Damon Edwards, J. Paul Reed, Pete Cheslock, and Michael Cote, who all frequently come to Austin to share with us.

And we printed a yearbook, with pics of speakers from all the events, our tshirts over the years, and more! Very snazzy, and we had people sign each others’ yearbooks to add some fun to the hallway track. In fact, you can view the yearbook online and buy a hardcopy here if you want!

The DevOpsDays 2022 Yearbook

We did require COVID protocols – masking inside and (honor system) vaccine or test, and while it is a bummer to not see each others’ faces, it also resulted in only one person I know of getting COVID the week after, so well worth it.

We didn’t have to worry about sponsor interest! We sold out quickly. Here’s the ones I got snaps of!

Everything went great, and it was super to finally get back together and interact with our local DevOps community. J. Paul Reed led a great session where a retro was done on DevOps in general!

And one of the best things is that we managed to carry on our tradition of giving our excess proceeds to charity! I’ll do a separate post on that, but the short form is that we contributed $28,000 to LGBT-supporting charities, half to The Trevor Project and half to Out Youth here in Austin, bringing us to $100,000 given to charity over our 10 years in existence! Stay tuned for more details on that…

2 Comments

Filed under Conferences, DevOps

All Day DevOps Is Coming Up

And James, Karthik, and I are hosting the Modern Infrastructure track again this year! ADDO is a free, 24 hour, multi-track online conference with a lot of great speakers. More info follows…

What: All Day DevOps, Live Online

When: November 12, 2020 (24 hours)

Where: From your desktop, laptop, or mobile device

Free: Click here to register

On November 12th, we will be supporting the 5th Anniversary Celebration for All Day DevOps. This is a 24 hour event with 6 simultaneous tracks, delivering 180+ sessions, live online. Session tracks include CI/CD Continuous Everything, Cloud Native Infrastructure and Monitoring, DevSecOps, Cultural Transformation, Site Reliability Engineering and Government.

Check out the amazing speaker lineup. Registration is free. Full details are located at AllDayDevOps.com.

Virtual Viewing Parties: Hosting a virtual viewing party is free for anyone in the community you supply the group and the connection while All Day DevOps provides the content. Here are some guidelines that will provide detailed information and tips to assist you with your party planning.

Leave a comment

Filed under Conferences, DevOps

Change Management – Share Your Thoughts!

Hey loyal admineers! (I just made that up.) I wanted to toss a question out to our readers. I’m working on a Change Management course for LinkedIn Learning right now to join my other courses, and I was hoping to hear some good new techniques people are using to do it that a) ensure compliance but b) are not super heavy and lame.

My current approach is to mention the ITIL, COBIT, ISO 27000-1, etc. approaches but then come in with a practical approach inspired by Visible Ops and leavened with DevOps innovation.

Chime in in the comments – how do you do change management? What compliance regimes do you have to fulfill? Are you using one of the ITSM frameworks? Are you using a tool (ServiceNow aka “ServiceNo” followed by JIRA were the two most popular when I asked at the latest CloudAustin meeting)? Are you using any techniques that you think are excellent and would like others to hear about? I’d love to hear from you!

Leave a comment

Filed under DevOps

Pragmatic Pipeline Security

Check out agile admin James Wickett’s talk from DeliveryConf last month on adding security into your continuous software delivery pipeline!

Leave a comment

Filed under Conferences, DevOps, Security

Record and Replay Browser Testing, Take 2

Recently I reported, and I quote, “Bah!” from trying a bunch of record and playback cross browser testing options. To recap, we’re a startup, our devs are writing UI tests but we don’t have cross-browser testing, so I tried to find something where I could record and replay our pretty simple Angular UI flow and get some cross-browser testing on it without needing code. And I didn’t find anything that worked. But I got a bit more traction after that first run at it, so here’s part 2.

EndTest

The EndTest support folks looked into it and fixed my test so it worked. Some were alternate locator schemes.  Some were advanced options I am not sure how I would have figured out (to close those pesky multiselect dropdowns, you can’t just click on the overlay backdrop that comes up, you have to offset by some pixels…).  I am not a front end guy, I just fake it, so this is a little daunting, but their support seems able to help with tests so it’s doable.

I now have a working test, somewhat generated from the recording and somewhat generated by programming. The trial doesn’t allow other browsers by default but they turned them on for me.  With some light fiddling I got them all running, and the only issue was IE not running because it didn’t like that pixel-offset workaround from the above and EndTest fixed it by changing my test to hit “enter” instead – fair enough.

I then also made a test for the second part of our flow that has a PDF that needs validation, you can do it with a screenshot comparison.

So after help from their support, I have a working test suite – it’s a stretch to say it’s pure record and replay; it’s record, edit the locators a bunch, and replay, but since my last iteration on this was “none of these solutions work and do crossbrowser”, that’s a big win.

Sauce Labs

Last time I had been trying to use the Selenium IDE for record/replay and integrate with Sauce for the crossbrowser testing. They got back to me and extended my trial minutes and gave me some tips on how to get some of the tests working.  They don’t support the Selenium IDE however so they don’t really help with the tests per se.

I had a weird experience, there were intermittent (but frequent) timeouts from running tests against Sauce with selenium-side-runner, just at some point 0-4 minutes into the test it’d hang and time out and give me a “ETIMEDOUT connect ETIMEDOUT 66.85.49.22:443″.  

And then I got into the lovely rich set of test things that don’t work the same across the browsers.  Oh, in Edge “the element isn’t clickable because it’s obscured.”  In Firefox on Mac “that radio button isn’t clickable because the inner part of the radio button obscures it.” All different elements.  The problem is, fiddling with the locators to get something that browser likes breaks it in the other browsers.  But the app works in those browsers, just not the tests.

Anyway, Sauce support said “we can’t support Selenium IDE questions” so I guess that’s it.  (I don’t know that those timeouts when running tests on their service count as selenium IDE questions but whatever). I had hope when I found Selenium IDE that a record and replay with Sauce was feasible but it seems like it’s a starting point to seed your own Selenium code at best.

mabl

A sales rep from mabl wouldn’t let go of me till I tried their solution. So I did and it has a lot of promise.  It tries a bunch of different methods to find a locator automatically and self-heals the tests, which is great.  On the one hand the tests are slow; a 4 minute selenium test is a 6-8 minute mabl test. On the other hand it works!

It worked the first time in fact (Well… second, but it was because I went into a click frenzy trying to get the mabl trainer window and our Hubspot popup out of my way). I now have a working mabl test, though it is having one-minute timeouts and confusion on that same “click on the backdrop to get out of the multiselect dropdown” issue that’s a pain in all the other solutions as well, it does it but after a long timeout and it doesn’t self heal it for the next test run so it works but is slow. I hate multiselect dropdowns. Anyway, after a discussion with mabl support it turns out that it tries a bunch of ways to find a locator and then self heals if it found a better one, and then tries a bunch of ways to click/exercise that locator but doesn’t self heal from that hence the long timeout in my test.  I gave them the feedback “self heal that too, yo!”

OK so it’s working, how about cross-browser?  I add Firefox – it works.  IE and Edge are only on the “enterprise plan” but I ask them to add them to the trial, they do, I run them, and…  They work!  Safari… Well, a problem there, but mabl looks at it and thinks it may be them. I’m super impressed, of all the stuff I’ve tried only GhostInspector actually recorded and replayed without significant recoding and that was only Chrome/Firefox.  They’re still working on a fix as of “press time.” They also do PDF testing, which we need..

So it works great and looks good!  And they have a lot of cool dev integration and stuff to get into, it uses a branching model for the tests, you can run the tests locally…  Great looking ecosystem. You can’t choose like OS platform though, the Chrome and Firefox are just on Linux. At our current level of detail that’s fine.

Next step is discussing pricing (Web site just says “contact us”)… And unfortunately it’s way, way out of reach of a 10-person startup. The cross-browser and PDF testing are part of the “Enterprise plan” but even if I sweet talk them into putting those features in their lowest cost plan it’s still cost prohibitive for us while we’re in seed round.  I mean, it’s totally worth it because it works out of the box without fiddling around, if I were still at AT&T Cybersecurity I’d make someone spring for it for sure, but it’s an extremely significant pricing step above these other solutions and not at the value point for where we are right now.  Dang.

New Bottom Line

OK so my new bottom line is this.

  1. If you just want quickie Chrome/Firefox on Linux record and replay, GhostInspector works out of the box. Nice but I want more browsers, doesn’t quite fit my needs.
  2. If you want record and replay on various OS/browser combinations and are willing to do some re-coding and testing of it to make it work, EndTest does it. Fits my need with a little pain, and is affordable.
  3. If you want cross-browser record and replay without hand coding, without full platform choices but with a bunch of cool dev friendly extras, mabl does a great job – for a lot more money than the other options.  Best for my needs, but most expensive.
  4. The other options basically don’t work on Angular (Crossbrowsertesting), or possibly work but with intensive time investment that makes it not really record and replay IMO (Sauce + Selenium IDE).
  5. Though if you don’t need a bunch of CI-driven executions per day and don’t care about all the platforms you can probably just use the Selenium IDE to do 3 major browsers locally installed on whatever laptop you have yourself for free (Safari/Chrome/Firefox on Mac or Chrome/Firefox/Whatever Microsoft Is Pushing Today on Windows). Free but DIY.

So for us being on a seed round budget, EndTest is probably the best compromise of functionality and price at this point, YMMV of course.

1 Comment

Filed under DevOps

Trying To Record And Replay Browser Tests… FML

I’m working for a startup right now and we don’t have a huge excess of development staff.  Our devs have been implementing UI testing in Cypress, but we also need some wide cross-browser testing of our front end Angular apps – we’d already found a couple blocker bugs on Edge and IE largely by accident.  The devs are all busy devving, so I figured I’d take that on. I said, “Well, there’s products where you can just click to record a UI session and replay it in other browsers without writing a bunch of code, let’s try that out.”  Most everyone has a free trial nowadays so I could see which ones were best. Then the pain began.

Sauce Labs – Part 1

I had used Sauce in a previous life, when we had a bunch of Robot Framework/Selenium tests and I liked it.  So I went there first.  Unfortunately, they have no record/replay capability, verified by their support, so I moved on.

But I came back later because I had found that there was a Selenium IDE that’s a record and playback tester that you can integrate with Sauce by using selenium-side-runner.

Selenium IDE is very cool, its killer feature is that as it records it copies various ways to address an item on the screen – css, xpath, full xpath – and when it replays if the first one doesn’t work it tries the next one and if that works tells you “hey you should update this test.” That’s great because UI testing is shitty and unreliable at best, and once you have Angular generating ever-changing ids for elements it is even worse.  The only bad thing is you have to go add assertions in manually afterwards.

Screen Shot 2020-01-09 at 10.58.47 AM

So in fairly short order, I managed to get a reproducible Selenium IDE script that exercised our Angular app and works.  The app’s just like 7 screens of form fill, it’s not crazy.

Well, then I tried to save it as a “.side” project and feed it through Sauce by using selenium-side-runner, which is just:

npm install -g selenium-side-runner

selenium-side-runner –server <sauce-url> -c “browserName=’chrome’ version=’latest’ platform=’macOS 10.14′” ‘Paul Precision.side’

You get that sauce URL that has credentials embedded under User Settings/Driver Creation in their UI.

Unfortunately once I push it to sauce (starting on the same OS/browser, which you go get the tokens for from their Platform Configurator) – problems. The player is great, it shows the video (even live while testing) synced to the step taking place (unfortunately since I’m piping it in, it’s not showing the steps in the test syntax, but in raw Selenium execution syntax).

Screen Shot 2020-01-09 at 12.37.42 PM

I fixed most of them by going and changing selectors away from CSS to xpath, then sitting there iterating with chrome dev tools and the IDE trying new ways to use an item that works in chrome then works in selenium IDE and then… Doesn’t work on Sauce. I have gotten it 90% working but the last 10% is blocking me.

CrossBrowserTesting

Next I tried SmartBear’s CrossBrowserTesting.com. An all-in-browser recorder that worked great!  And then the replays didn’t work.  I messed with it a while and contacted their support, who said “Oh yeah it doesn’t do angular, it’s for static pages.”  So on to the next one. Who uses static pages, this is 2020?

The interface is nice enough, editable steps next to a running video (though not synced up).

Actually looking at it closer I bet I could do the same “edit all the locators” deal and try to get it to work but… My 7 day trial is over (a week shorter than the other options) so I guess I can’t try.  It didn’t do the nice multi-locator guessing Selenium IDE did but it does seem to have several options in a dropdown while I edit the tests, and the recorder is integrated into the offering so that’s nice – the UI was good overall. Unfortunately the super short trial and the presales support saying “Angular? Go away!” prevented me from really seeing if it can work for us.

Screen Shot 2020-01-09 at 12.38.18 PM.png

GhostInspector

Demoralized, I head to Twitter, and someone recommends GhostInspector.  You record it with a Chrome plugin and then replay in the browser – video, but then it shows the editable steps next to a screenshot showing % change from the last screenshot (the steps aren’t synced to the video, which would be better) .  You can do assertions while you’re recording.  And the replay works the first time and every time – Hallelujah!

Screen Shot 2020-01-09 at 12.54.11 PM

And then I look to set up cross-browser and discover they only support Chrome and Firefox, and to even do that in an automated manner you have to duplicate the entire test suite.  I was so disappointed, it worked perfectly otherwise.

Seriously y’all if you add more browsers I’ll pay you immediately for this.

EndTest

Determined to make this happen I find EndTest and, after verifying they support a full OS/browser matrix, try them. They also use a browser plugin recorder like GhostInspector.

I’ll be honest, the UX is terrible.  Besides the 1990s colored icons, everything is always a click away – you have to watch the replay video separately from looking at the logs from looking at the steps from editing the steps. Everyone, the magic combination is editable steps to the left, running video and logs to the right, highlight the step you’re on as it plays. Anything else harms your usability.  And also while editing steps you can’t add a step in just anywhere, you have to add it at the end of your 100 steps and then drag it up page by page…  And often when you do that you just get “error saving test” messages for no reason. Argh.

Screen Shot 2020-01-09 at 1.08.46 PM

But… The recording is quick and then it is semi working.  Tempting.  Now I start the iterative edit-replay-debug cycle.  It is slow. You get to give your steps a name but those names don’t show up in the test output, because why would they.  After an afternoon of fiddling, I’m halfway through a 7 screen flow. Their support was nicely proactive and reached out to me about an error (I was looking for text with a $ in it and you can’t do that, but you can define a variable and then use that…)

It’s at this point I also find the Selenium IDE and bring Sauce back into the mix.

Keep Trying – Sauce and EndTest

Next, what I was doing was fiddling with the steps in the Selenium IDE, then pumping those changes both into Sauce via CLI and manually editing them into EndTest’s UI, desperately hoping to get one to pass (they don’t act the same under the same inputs for whatever reason).

Locator by locator I grind through making the test work.  I have a lot of trouble where we use multiple option mat-selects, because they “stay open” while you select items and I can’t get them to close.  I try sending ESCAPE keys but can’t get that to work, I try double clicking on other things…  One of our devs figured out the magical thing to click on was the overlay backdrop (css=.cdk-overlay-backdrop) to close the damn multiselect box.

This takes several grueling days.  I ask support folks for help but don’t really get any useful traction.  Finally, I get a magic combination in the Selenium IDE that also works in Sauce!  I try the same ones in EndTest and they don’t work.

Screen Shot 2020-01-10 at 11.15.10 AMScreen Shot 2020-01-10 at 11.15.18 AM

It’s super frustrating.  The same locator doesn’t work in all 3 tools, often forcing me to choose a less portable option – instead of something resilient to change like “xpath=//span[contains(.,’Visual Line of Sight’)]” – which works in some cases – I end up having to use something like like  xpath=//mat-option[@id=’mat-option-87′]/mat-pseudo-checkbox (and sadly in angular material those IDs randomize unpredictably). Like, there will literally be two identical-except-for-the-text-and-ids-in-them widgets one after the another and one kind of locator works on the first one and not on the second. No idea why.

Sauce Labs – Part 2

OK, so of all the options the only one that actually works for me and will allegedly do crossbrowser testing is an unsupported combo of Selenium IDE and Sauce run off the command line.  A couple sources I found over the course of this:

Not optimal, but at this point I’m a week in and taking what I can get.  Let’s try an actual crossbrowser matrix now.  Bonus hacky Bash script:

#!/usr/bin/env bash

tests=("Paul Precision.side")

platforms=("browserName='chrome' version='latest' platform='macOS 10.14'"
        "browserName='chrome' version='latest' platform='Windows 10'"
        "browserName='chrome' version='latest' platform='Linux'"
        "browserName='MicrosoftEdge' version='latest' platform='Windows 10'"
        "browserName='safari' version='latest' platform='macOS 10.14'"
        "browserName='firefox' version='latest' platform='macOS 10.14'"
        "browserName='internet explorer' version='latest' platform='Windows 10'")

for test in "${tests[@]}"
do
        for platform in "${platforms[@]}"
        do
        echo Running "${test}" "${platform}" 
        echo
        selenium-side-runner --server https://<secrets>@ondemand.saucelabs.com:443/wd/hub -c "${platform}" "${test}"
done
done

Chrome on MacOS – works.  Chrome on Windows – works.  Chrome on Linux – for some reason can’t find a selector early on.  Edge on Windows – weird proxy 400 error, won’t even load the page.  Pretty sure that’s not my fault.  Safari on MacOS – can’t click on the first things it needs to click on.  Firefox on MacOS – same error?  Really?  Now IE… Out of minutes (despite the UI telling me .6 automated hours remain).

I have tried all these os/browser combos manually and they work.

So my conclusion is all these suck and I guess I just need to pay manual QA people to click on our app.  Great.  Or for Cypress to get off their butts and add cross-browser support, which they say “is coming” for three years now.

We’re a startup and time is money, so in the end cross-browser testing is not worth the hassle in all these solutions.  But it is important and I’d love someone to make a solution that actually works for it.

P.S. Please do not suggest another solution unless it has a) UI record and replay capability and b) is cross browser (Chrome/Firefox/Safari/IE/Edge on Windows/MacOS/Linux). I know there’s a million browser automation testing tools out there, that’s not what I need.

Update

I put some more time into this and got some working options – see Record and Replay Browser Testing, Take 2!

Leave a comment

Filed under DevOps