Category Archives: Cloud

Cloud computing and all its permutations.

by | June 18, 2013 · 4:54 pm

Velocity 2013 Day 1 Liveblog – Using Amazon Web Services for MySQL at Scale

Next up is Using Amazon Web Services for MySQL at Scale. I missed the first bit, on RDS vs EC2, because I tried to get into Choose Your Weapon: A Survey For Different Visualizations Of Performance Data but it was packed.

AWS Scaling Options

Aside: use boto

vertical scaling – tune, add hw.

table level partitioning – smaller indexes, etc. and can drop partitions instead of deleting

functional partitioning (move apps out)

need more reads? add replicas, cache tier, tune ORM

replication lag? see above, plus multiple schemas for parallel rep (5.6/tungsten). take some stuff out of db (timestamp updates, queues, nintrans reads), pre-warm caches, relax durability

writes? above plus sharding

sharding by row range requires frequent rebalancing

hash/modulus based- better distro but harder to rebalance; prebuilt shards

lookup table based

Availability

In EC2 you have regions and AZs. AZs are supposed to be “separate” but have some history of going down with each other.

A given region is about 99.2% up historically.

RDS has multi-AZ replica failover

Pure EC2 options:

  • master/replicas – async replication. but, data drift, fragile (need rapid rebuild). MySQL MHA for failover. haproxy (see palomino blog)
  • tungsten – replaced replication and cluster manager. good stuff.
  • galera – galera/xtradb/mariadb synchronous replication

Performance

io/storage: provisioned IOPS.  Also, SSD for ephemeral power replicas

rds has better net perf, the block replication affects speed

instance types – gp, cpu op, memory op, storage op.  Tend to use memory op, EBS op.  cluster and dedicated also available.

EC2 storage – ephemeral, epehemeral SSD (superfast!), EBS slightly slower, EBS PIOPS faster/consistent/expensive/lower fail

Mitigating Failures

Local failures should not be a problem.  AZs, run books, game days, monitoring.

Regional failures – if you have good replication and fast DNS flipping…

You may do master/master but active/active is a myth.

Backups – snap frequently, put some to S3/glacier for long term. Maybe copy them out of Amazon time to time to make your auditors happy.

Cost

Remember, you spend money every minute.  There’s some tools out there to help with this (and Netflix released Ice today to this end).

Leave a comment

Filed under Cloud, Conferences, DevOps

Tagged as , , , , ,

by | January 14, 2013 · 10:51 pm

Awesome Austin Events!

Besides the “big one,” DevOpsDays Austin 2013, there’s a bunch of great events going on in Austin for techies.

The Agile Austin DevOps SIG meets every last Wednesday over lunch at Bazaarvoice; lunch is provided.  This month’s meeting on January 30 is Breaking the Barriers.

The Austin Cloud User Group meets every third Tuesday in the evening at Pervasive; dinner is provided. This month’s meeting is January 15, sponsored by Canonical, and there is a talk on Openstack Quantum, the network virtualization platform.

South by Southwest Interactive is here of course on March 8-12.

Hip security event BSides Austin is on March 21-22.

Data Day Austin is on January 29.

Texas Linux Fest will be May 31 – June 1.

It’s never been a better time to be a techie in Austin!

3 Comments

Filed under Cloud, Conferences, DevOps

Tagged as , , , , , ,

by | January 25, 2012 · 4:33 pm

Awesome Austin Tech Meetups

Austin is such a great place to be a techie.

  • The Austin Cloud User Group (I help run it) meets every third Tuesday evening, and we’ve ben having 50+ people come in to check out some awesome stuff.  Next meeting Feb 21 on Puppet, hosted by Pervasive.
  • The Agile Austin DevOps SIG meets fourth Wednesdays, we had our meeting today and had about 20 attendees, hosted by CA/Hyperformix. I also help run that one.
  • The Austin Big Data User Group is back meeting – next one is tomorrow night! Hosted by Bazaarvoice.
  • The Austin OWASP chapter is one of the biggest and most active in the country, and also meets monthly, hosted by National Instruments. Fellow Agile Admin James Wickett helps run that group.
  • The Cloud Security Alliance, Austin chapter is just getting started but has a lot of momentum and we’re coordinating with them from the ACUG and OWASP sides. Their first meeting is tonight, come out!

There are others but those are my favorites and therefore the coolest by definition.

There’s also cool events coming up you should keep an eye out for.

  • DevOpsDays Austin, Apr 2-3, hosted by National Instruments, and this’ll be big! Patrick Debois and the whole crew of DevOps illuminati will be here. Now taking sponsors and speakers! Register now!
  • AppSec USA 2012, Oct 23-26 – Austin OWASP kicks so much ass with LASCON that the annual OWASP convention is coming here to Austin this year!
  • South by Southwest Interactive, March 9-13 – quickly becoming theWeb conference in the flyover states :-). Lots of stuff happens during it, like:
    • Austin Cloud/DevOps party courtesy GeekAustin (ACUG is a community sponsor). March 10.
    • CloudCamp – Dave Nielsen will be bringing a CloudCamp to Austin again this year during SXSWi. Details TBD, sounding like Mar 11 maybe.
  • The Cloud Security Alliance and ACUG are hoping to put together an Austin cloud conference, too. Maybe early 2013.

Leave a comment

Filed under Cloud, Conferences, DevOps

Tagged as , , , , , , ,

by | January 19, 2012 · 3:33 pm

Why Does Cloud Load Balancing Suck?

Back in the old world of real infrastructure, we used Netscalers or F5’s and we were happy.  Now in the cloud, you have several options all of which seem to have problems.

1. Open source.  But once you want SSL, and redundancy, and HTTP compression, you get people saying with a straight face “nginx (for HTTP compression) –> Varnish cache (for caching) –> HTTP level load balancer (HAProxy, or nginx, or the Varnish built-in) –> webservers.” (Quoted from Server Fault).  Like four levels, often with the same software twice in it. And don’t forget some kind of heartbeat between the two front-ends. Oh look I’ve spent $150/mo on just machines to run my load balancing. And I really want to load balance/failover between all my tiers not just the front end.  It’s a lot of software parts to go wrong.

2. Zeus.  For some reason none of the other LB vendors have gotten off their happy asses and delivered a good software load balancer you can use in Amazon.  I got tired of talking to our Netscaler reps about it after the first couple years.  They’re more interested in selling their hardware to the cloud data centers than helping real people load balance their apps. Zeus is the only one – and it’s really quite expensive

3. Amazon ELBs.  These just have a lot of problems under the hood.  We’ve been engaged with Amazon ELB product management on them – large files serve out super slow; users get hits refused due to throttling/changes during ELB scaling – basically if you want 100% of your hits to come through you can’t use them.

4. Geo-IP load balancing, through Dyn or whoever. They claim to have the failover problem fixed, but it still only works for the front end tier of what is a multitier architecture. I certainly don’t want to have to advertise every internal IP in external DNS to make load balancing work.

And really the frustrating part is there seems to have been no headway on any of this stuff in a decade. Same old open source options, same old techniques.  Can someone come up with a way to load balance on the cloud that a) doesn’t lose any hits, b) is one thing not 4 things, and c) is useful for front and back end balancing?  Seems like a necessary part of oh say every system ever, why is it still so hard?

31 Comments

Filed under Cloud, DevOps

Tagged as ,

by | December 16, 2011 · 10:14 am

Cloud Security Is No Oxymoron

Fellow Agile Admin, James Wickett, just wrote an article for Control Engineering about cloud security. Cloud security is kinda funny, it’s the biggest FUD attractor and “concern” of folks who don’t really know how their on premise security works either.  Anyway, read the article! We’re putting some real security work into our cloud products at NI; can’t speak for others but…

In related news, the excellent Cloud Security Alliance is starting an Austin chapter! Go check it out. The CSA got a start by being an organization that actually issues effective guidance on cloud security instead of being another vendor-haven or FUD collector.

Also see my older LASCON 2010 presentation on “Why The Cloud Is More Secure Than Your Existing Systems” (now a year later, the trade rags are stealing that headline for their own spam…  Woot!)

Leave a comment

Filed under Cloud, Security

Tagged as ,

by | November 17, 2011 · 2:46 pm

Turbo-Charging Your Product Development

Bruce Rayner from Enterprise Efficiency did this nice writeup on how we leveraged the IT expertise within our company to jump start our SaaS product development here at NI. Our whole team except for our manager worked at some point within NI IT, but are now doing product development in R&D. I know to smaller companies it seems like not so much of a big deal, but in larger companies it can be hard to maintain a culture where you really can decide what you want to do, establish its priority, and then put the right kinda and type of resource on it despite organizational boundaries. Organizational agility FTW!

Leave a comment

Filed under Cloud

by | September 15, 2011 · 7:00 am

How We Do Cloud and DevOps: The Motion Picture

Our good friend Damon Edwards from dev2ops came by our Austin office and recorded a video of Peco and I explaining how we do what we do! Peco never blogs, so this is a rare opportunity to hear him talk about these topics, and he’s full of great sound bytes. 🙂

I apologize in advance for how much I say “right.”

Leave a comment

Filed under Cloud, DevOps

Tagged as , , , , , , ,

by | September 13, 2011 · 9:55 am

What Is Cloud Computing?

My recent post on how sick I am of people being confused by the basic concept of cloud computing quickly brought out the comments on “what cloud is” and “what cloud is not.” And the truth is, it’s a little messy, there’s not a clear definition, especially across “the three aaSes“. So now let’s have a post for the advanced students. Chip in with your thoughts!

Here’s my Grand Unified Theory of Cloud Computing. Rather than being a legalistic definition that will always be wrong for some instances of cloud, it attempts to convey the history and related concepts that inform the cloud.

The Grand Unified Theory of Cloud Computing

( ISP -> colo -> MSP ) + virtualization + HPC + (AJAX + SOAP -> REST APIs) = IaaS
(( web site -> web app -> ASP ) + virtualization + fast ubiquitous Internet + [ RIA browsers & mobile ] = SaaS
( IDEs & 4GLs ) + ( EAI -> SOA ) +  SaaS + IaaS = PaaS
[ IaaS | PaaS | SaaS ] + [ devops | open source | noSQL ] = cloud

* Note, I don’t agree with all those Wikipedia definitions, they are only linked to clue in people unsure about a given term

Sure, that’s where the cloud comes from, but “what is the cloud?” Well, here’s my thoughts, the Seven Pillars of Cloud Computing.  Having more of these makes something “more cloudy” and having fewer makes something “less cloudy.” Arguments over whether some specific offering “is cloud” or not, however, is for people without sufficiently challenging jobs.

The Seven Pillars of Cloud Computing

“The Cloud” may be characterized as:

  • An outsourced managed service
  • providing hosted computing or functionality
  • delivered over the Internet
  • offering extreme scalability
  • by using dynamically provisioned, multitenant, virtualized systems, storage, and applications
  • controlled via REST APIs
  • and billed in a utility manner.

You can remove one or more of these pillars to form most of the things people sell you as “private cloud,” for example, losing specific cloud benefits in exchange for other concerns.

Now there’s also the new vs old argument. There’s the technohipsters that say “Cloud is nothing new, I was doing that back in the ’90’s.” And some of that is true, but only in the most uninteresting way. The old and the new have, via alchemy, begun to help users realize benefits beyond what they did before.

Benefits of Cloud – What and How

Not New:

  • Virtualization
  • Outsourcing
  • Integration
  • Intertubes
 Pretty New:

  • Multitenancy
  • Massively scalable
  • Elastic self provisioning
  • Pay as you go
 Resulting Benefits:

  • agility
  • economy of scale
  • low initial investment
  • scalable cost
  • resilience
  • improved service delivery
  • universal access

Okay, Clouderati – what do you think?

2 Comments

Filed under Cloud

Tagged as , , , , ,

by | September 8, 2011 · 4:41 pm

Mystifying Cloud Computing

I have now received my 200th email entitled “Demystifying Cloud Computing.” This one is from InfoWorld, but I have gotten them from just about every media outlet there is. This has got to stop.

People, it is not a mystery any more!  If you are still “mystified” by cloud computing, you probably need to consider an alternate line of work that does not generate new ideas at the aggressive rate of one every decade.

Let’s get this over with.

Q: What is cloud computing?

A: It is other people taking care of shit for you on the Web.
Maybe it’s running a data center, maybe it’s storing your files, maybe it’s running your ERP system or email system.  It’s, like, stuff you would do, but you are paying someone else to do it better instead, on demand.
Maybe there’s “scaling,” or “utility billing,” or “REST APIs” involved, maybe not. Ask a grownup.

There, consider yourself demystified.  You may now go get some of the green paper out of your parents’ wallet and mail it to me.

6 Comments

Filed under Cloud

Tagged as ,

by | April 21, 2011 · 3:26 pm

Moving Your Amazon EC2 AMIs To Another Region

Here’s what I found out the hard way during the Amazon outage when I wanted to migrate my systems to a different region.

You can’t use your AMIs or other assets in a region different from the ones they were created in. You can’t use your security groups or keypairs or EBSes, you have to migrate or recreate all of them, yay. Some methods to do this follow.

Manual method:

Automated:

Paying Money:

Of course what the answer should be is “click a button in the Amazon console or invoke the Amazon API to say “move ami-xxxxx to region y” done.

In the end none of these were working during the outage because they all rely on the ability to bring up an instance/EBS in the source region. We then rebuilt images from scratch in us-west but looks like east is back online now just as we’re finishing with that. So plan ahead!

3 Comments

Filed under Cloud

Tagged as , , , , ,