Monthly Archives: June 2010

by | June 11, 2010 · 1:07 pm

Automated Testing Basics

Everyone thinks they know how to test… Most people don’t.  And most people think they know what automated testing is, it’s just that they never have time for it.

Into this gap steps an interesting white paper, The Automated Testing Handbook, from Software Test Professionals, an online SW test community.  You have to register for a free community account to download it, but I did and it’s A great 100-page conceptual introduction to automated testing. We’re trying to figure out testing right now – it’s for our first SaaS product so that’s a challenge,  and also because of the devops angle we’re trying to figure out “what does ‘unit test’ mean regarding an OS build or a Tomcat server before it gets apps deployed?'”  So I was interested in reading a basic theory paper to give me some grounding; I like doing that when engaging in a new field rather than just hopping from specific to specific.

Some takeaways from the paper:

  • Automated testing isn’t just capture/replay, that’s unmaintainable.
  • Automated testing isn’t writing a program to test a program, that’s unscalable.
  • What you want is a framework to automate the process around the testing.  Some specific tests can/should be automated and others can’t/shouldn’t.  But a framework helps you with both, and lets you make more reusable tests.

Your tests should be version controlled, change controlled, etc. just like software (testware?).

Great quote from p.47:

A thorough test exercises more than just the application itself: it ultimately tests the entire environment, including all of the supporting hardware and surrounding software.

We are having issues with that right now and our larval SaaS implementation – our traditional software R&D folks historically can just do their own narrow scope tests, and say “we’re done.”  Now that we’re running this as a big system in the cloud, we need to run integration tests including the real target environment, and we’re trying to convince them that’s something they need to spend effort in.

They go on to mention major test automation approaches, including capture/replay, data-driven, and table-driven – their pros and cons and even implementation tips.

  1. Capture/replay is basically recording your manual test so you can do it again a lot.  Which makes it helpful for mature and stable apps and also for load testing.
  2. Data driven just means capture/replay with varied inputs.  It’s a little more complicated but much better than capture/replay, where you either don’t adequately test various input scenarios or you have a billion recorded scripts.
  3. Table driven is where the actions themselves are defined as data as well – one step short of hardcoding a test app.  But (assuming you’re using a tool that can work on multiple apps) portable across apps…

I guess writing pure code to test is an unspoken #4, but unspoken because she (the author) doesn’t think that’s a real good option.

There’s a bunch of other stuff in there too, it’s a great introduction to the hows and whys and gotchas of test automation.  Give it a read!

Leave a comment

Filed under DevOps

Tagged as ,

by | June 9, 2010 · 9:31 am

How to hire an Agile Admin

The Kitchen Soap blog has some great interview questions for hiring a WebOps position. Check it out, it is worth the read.

In my experience with hiring, a real simple one is to ask (while holding their resume), “Can you tell me about yourself?” Sure, I can read what it says, but letting them verbalize usually is a good indicator. In one of the last sets of interviews I did I asked a candidate this question and I got a gruff response, “It is all right there, what do you need to know?” Good communication skills? No, see you later.

One other question I like is, “What are two character flaws you have?” Usually someone prepares for one in advance with something like, “I am an over-committed worker…” or other statement that is meant to actually show a positive side about them. Asking for two lets you watch for quick thinking and (again) communication skills. In our industry technical is a must, but people can be trained. If you are bad at communicating or just a jerk, then no amount of training can help.

Anyone else have some good interview Q’s?

2 Comments

Filed under General

Tagged as , ,

by | June 8, 2010 · 11:58 pm

vim tip of the day

One thing that every admin (agile or not, although we hope agile) needs to do is use vi.  Or vim for the slightly more civilized, which I am encouraging adoption therof by calling this the vim tip of the day in lieu of the vi tip of the day.

If you are reading this, and are thinking, “what is vim?” then you might want to skip this.  If you read this and are thinking, “I dream in regex and I just can’t wait” then you might be a little let down.  But for those of us that are left, here is a handy little vim tip for you.

Often I find myself looking running

sudo vim /etc/hosts

which is fine and dandy. But more often than not, I forget to sudo. When that happens and you are just about to save your work you are greeted with a “Can’t open file for writing” message. Dang! At this point you probably copy out the changes you made, exit the file, and reopen the file using sudo. All the while you are wondering, “surely there is a better way.”

Well there is.

:w !sudo tee % >/dev/null

Type this into vim and it will save your work just as if you were running vim using sudo.

I am not sure if this will be a regular feature, but I am going to try and cook up some other vim tips and share them with you.

Leave a comment

Filed under General

Tagged as , , , ,

by | June 8, 2010 · 1:18 pm

Velocity and DevOpsDays!

A double threat is coming your way.  Velocity 2010, the Web performance and operations conference, is June 22-24 in Santa Clara, CA.  As one of the very few conventions targeted at our discipline, we’ve been attending since the first one in 2008.  And this time, there’s dessert – the day after it ends, a new DevOps unconference, DevOpsDays 2010, will be held nearby in Mountain View!

OpsCamp Austin kicked ass, and I’m sure this will be even better.  So come double up on Ops knowledge and meet other right-thinking individuals.

If you want to read all my musings from the previous Velocity conferences, you can do that too!

Leave a comment

Filed under DevOps

Tagged as ,

by | June 4, 2010 · 10:18 am

CloudCamp Austin Is Soon!

Mark your calendars; Thursday of next week (June 10) is CloudCamp here in Austin!  It’s in North Austin at Pervasive’s offices (Riata Trace) from 5:30-10:00 PM.  Get details and sign up here.

Leave a comment

Filed under Cloud

Tagged as , ,

by | June 3, 2010 · 10:10 am

Busting the Myths of Agile Development: What People are Really Doing

I just watched a good Webcast from an IBM agile expert about the state of Agile in the industry, and it had some interesting bits that touch upon agile operations.

Webcast – (registration required, sadly)

It’s by Scott Ambler, IBM’s practice leader for agile development.  They surveyed programmers using Dr. Dobbs’ “IT State of the Union” survey, which has wide reach across the world and types of programmers; the data in this presentation comes from that and other surveys.  All their surveys and results and even detailed response data (they’ve done a lot over time) are online for your perusal.

In the webcast, he talks some about “core” agile development extending to “disciplined” agile development that extends to address the full system lifecycle, is both risk and value driven, and has governance and standards.  “Core” begs the question of “where do requirements and architecture come from?” and “how do I get into production?”   Some agile folks who consider themselves purists say these aren’t needed and you should just start coding, he calls this view “phenomenally naive.”

He does mention some times where things like enterprise architecture were slow and introduced huge delays into the process because it took months to do reviews and signoffs.  He calls this “dysfunctional” but really isn’t it the way things are unless there’s a pattern to change it? I think project management and enterprise architecture are suffering from the same problem we operations folks are, which is that we’re just now figuring out “what does it mean to incorporate agile concepts into what we do?”

The meat of the preso is over agile team metrics and busting myths about “it’s only for small colocated teams…”  Here’s my summary notes.

  • Agile teams have a success rate higher than traditional teams.  Agile and iterative about tie and come in with much higher (2-4x) result on quality, functionality, cost efficiency, and timeliness than traditional or ad hoc processes.
  • Agile is for not just coding, but project selection/initiation, transition, ops, and maintenance.  More and more folks are doing that, but some get stuck with doing agile only in the coding and not the surrounding part of the lifecycle.
  • Agile is being used by co-located teams in only 45% of the time; all the rest are distributed in some manner.  But that does affect success some  – “far located” teams have 20% lower success than fully co-located.  Don’t distribute if you don’t have to.
  • Most orgs are using agile with small teams, the vast majority are size 10 or less.  But they see success with even the very large teams.
  • We’re seeing people successful with agile under complaince frameworks liek Sarbox, and governance frameworks like ISO – even ITIL is mentioned.
  • Agile isn’t just for simple projects – the real mix in the wild is actually weighted at medium to very complex projects.
  • Though agile is great for greenfield projects, there’s very large percentages of teams using it on legacy environments.  COTS development is the rarest.
  • 32% of successful agile teams are working with enterprise architecture and operations teams.  Should be more, but that’s a significant inroad.  He says those teams are also most successful when behaving agile (or at least lean).
  • Biggest problems with agile adoption are a waterfall culture (especially one where the overall governance everyone has to plug into is tuned to waterfall) and stakeholder involvement.  Testers say “We need a detailed spec before we can start testing…”  DBAs say “Developers can’t code until we have a complete data model…”  Management resistance is actually the lowest obstacle (14% of respondents)!

A lot of nice stats.  The two biggest takeaways are “agile isn’t just for certain kinds of projects, it’s being used for more than that and is successful in many different areas” and “agile is for the entire lifecycle not just coding.”  As advocates of agile systems, I think that’s a good sign that the larger agile community is wandering our way as we’re building up our conception of DevOps and wandering their way ourselves!

Leave a comment

Filed under DevOps

Tagged as , , ,

by | June 3, 2010 · 9:54 am

DNS Rebinding

Recently I was able to give a talk at Austin OWASP about DNS Rebinding.  I will be uploading slides and example code on this blog soon, but first an overview of the topic.

The most important portion of this topic is the same origin policy of the browsers.  It disallows a user from visiting a site, and then executing JavaScript against their local network.  Or, at least that is the idea.

In computing, the same origin policy is an important security concept for a number of browser-side programming languages, such as JavaScript. The policy permits scripts running on pages originating from the same site to access each other’s methods and properties with no specific restrictions, but prevents access to most methods and properties across pages on different sites.

This mechanism bears a particular significance for modern web applications that extensively depend on HTTP cookies to maintain authenticated user sessions, as servers act based on the HTTP cookie information to reveal sensitive information or take state-changing actions. A strict separation between content provided by unrelated sites must be maintained on client side to prevent the loss of data confidentiality or integrity.  Excerpt from Wikipedia

DNS Rebinding overrides same origin policy so that the client believes it is talking to the same host when it really isn’t.  The browser accesses sortabadsite.com and at first is getting legitimate responses from it.  Shortly after the first requests (initial page load) are made, all communication is dropped and the browser will make a call back to DNS.  At this point the IP address for the domain is swapped (maybe with 127.0.0.1) and the client is now running XHR (XML HTTP Requests) against the localhost. There are some interesting vectors that this can go and will be explored in future posts.

Check back at this blog for a video demo, slide deck and future plans for new code.  Right now I am working on writing a DNS Rebinder application in Ruby that includes DNS, a firewall and a web server (or hooks into them).  If you are interested, let me know.  Gmail:  wickett

I would be amiss if I didnt mention RSnake’s work on DNS Rebinding over ha.ckers.org.  Check it out!

Leave a comment

Filed under Security

Tagged as , , ,