Tag Archives: Security

AWS re:Invent Keynote Day 2 Takeaways

TL;DR – performance improvements and two huge announcements, Docker-based EC2 Container Service and cloud-CEP-like AWS Lambda.

I was in a meeting for the first 45 minutes but I hear I didn’t miss much. Happy customer use cases.

The first big theme of this morning’s keynote is “Containers” – often just shorthand for “docker.”  I went to a previous event here in town with even large enterprises and government – State of Texas, Microsoft, Dell, Red Hat – all freaking out about Docker. Docker is similar to VMWare or cloud in that it is a new technology that requires new monitoring and management just for it. (Heck, Eric, the CopperEgg founder, is now running a startup around docker container management, StackEngine.)

  1. Keynote from pristine.io about how they implemented. Docker, the new low overhead containerization technology, is a heavily cited part of the power (they actually used Flux7 as the expert consultants, they’re based here in Austin!).
  2. Keynote from Werner Vogels on the new “Amazon EC2 Container Service,” announced to cheers and applause. It allows launching and terminating containers to sets of instances on EC2. Their PM did a demo where they had a big farm of r3 servers and then they deploy a redis cluster and rabbitmq across them, and then front end components on a farm of c3s, and then audio processing across all of them. If you’re new to this it’s basically VMs within VMs but without noticeable overhead.
EC2 Container Service

EC2 Container Service

  1. Next they had the actual docker cofounder and CEO Ben Golub. He mentioned that docker is only 18 months old and its huge success and ecosystem this early in is “surreal.”

Next… Leapfrogging PaaS?

  1. Werner is back to announce AWS Lambda available now in preview – event-driven computing service for dynamic applications. No instance running/management required, events go in and “cloud functions” run on them.  Holy shit, this replaces a large number of servers running semi-trivial apps. 20 cents per million requests, plus some complex stuff for seconds of execution – free for 3.2M seconds/1M requests.

    Amazon Lambda

    Amazon Lambda

  2. Netflix chief product guy came on to show how they’re using lambda as a higher level abstraction and have eliminated a bunch of servers – no system monitoring/management, no inefficient polling, no gaps/opacity. They’re using it to encode video, run backups, run security and compliance checks against instances, and for operational monitoring and dashboards. Replacing procedural control systems with event-driven services.
  3. AWS core innovations… New c4 instance, Haswell based (crazy fast processor, 36 vCPUs). Diane Bryant, SVP/GM Data Center Group from Intel, came on to go into the CPU specifically. Larger and faster EBS volumes, up to 20,000 IOPS. Enhanced and consistent networking speeds.

And this has been your cloud update! Also see Ben Kepes in Forbes for a similar summary.

The container engine is cool – it’ll certainly remove a lot of instance gerrymandering and instance reservation pain if nothing else. But Lambda is the potential disruptor here.  It’s taking the idea of “bring your own algorithm” from MapReduce and saying “hmmm you can probably replace your trivial web app just with this” – it’s halfway between a PaaS and a SaaS, none of the Beanstalk complexity, just “here take this function and run it on stuff when it comes in.” If a library of common lambas becomes available, so much computing work done for trivial purposes becomes obsoleted.  Who hasn’t seen a Web service to “upload a file here, then zip it or something, then store it…” OK, no servers needed any more. Very interesting.

Leave a comment

Filed under Cloud, Conferences

AWS re:Invent Keynote Day 1 Takeaways

Sadly I couldn’t attend this year, but heck that’s what the Internet is for.  Here’s the interesting bits from the AWS re:Invent Day 1 keynote (livestreamed here). Loads of interesting stuff.

  1. AWS is growing revenue >40% YOY, far outstripping other large IT companies – EC2 use grew 99% YOY and S3 usage 137%, they have 1M active customers now. (Microsoft cloud services report 128% YOY growth as well.)
  2. New product announcement for Aurora – new commercial-grade database engine – fully MySQL compatible but 5x the performance, available through Amazon RDS, 1/10 the cost of the commercial DB engines (starts at 29 cents an hour, ~$210/mo). Can do 6M inserts/second and 30M selects/second. Highly durable (11 9’s), crash recovery in seconds with no data loss. Nice!
  3. SLDC stuff!
    1. CodeDeploy (was internal tool called Apollo), a new code-deployment system that lets you do rolling updates, rollbacks, and tracks deployment health. This works for all languages and is free. They use it internally for 95 deploys/hour on their own stuff.
    2. In early 2015 will come some more software lifecycle management services – first is CodePipeline for continuous integration and deployment (also used internally)
    3. Second is CodeCommit as a managed code repository that can colocate with where you’re going to deploy and has no size limits of repos or files. These “integrate with” github, jenkins, chef, etc. though it’s not clear how they don’t cannibalize them.
  4. Security stuff! Big push to be able to say “we easily surpass the security you can do on premise.”
    1. FISMA, ITAR, FIPS, FedRAMP, HIPAA, ISO 9001
    2. Current encryption approach is either “let Amazon manage keys” or use their CloudHSM hosted key thing, both of which are still a pain. As a result they’re launching AWS Key Management Service as a HA service that manages keys, provides one-click encryption and transparent key rotation.
    3. AWS Config is a new-gen agile CMDB with full visibility into all your AWS resources. You can query it and see relationships and show scope of a config change. Streams all config changes out to you.
    4. A new-gen service catalog called AWS Service Catalog available early 2015. Create and share product portfolios, let internal people launch them, tracking and compliance.
  5. Enterprise Cloud Adoption Patterns
    1. Often the first wave of moving into the cloud for enterprises is moving dev and test environments to run in AWS for flexibility and spin up/down for cost savings and  brand new apps, custom written for the cloud
    2. Second wave is web sites and digital transformation (media, corp sites, ecomm) and analytics, since mass processing and sharing is cheap in the cloud – data warehouses (like pfizer’s). And mobile app back ends – phone, tablet, gps, more.
    3. Third wave is business critical applications.  Macmillan and Hoya run their SAP in AWS. Conde Nast runs HR and Legal there.
    4. New wave – you’re starting to see entire datacenter migration and consolidation as DCs come up for lease (Hess, Conde Nast, NewsCorp). SunCorp. Time Inc., GPT, Nippon Express moving “all in” to AWS – many ISVs as well. The CIA moved to AWS and now Intuit is doing so now as well.
    5. Intuit moved their “TurboTax AnswerXchange” app there to deal with tax time peaks last year and the scales fell from their eyes when they did so – 6x cost cut, setup 1/5 of the time, faster development. They started doing more and realized the global datacenters, ease of integration with acquisitions, and dev recruiting benefits. They have 33 services on AWS now, and have moved mint.com there. They have decided to move everything else there now. Funny how once companies start looking at how much they accomplish instead of just the monthly cost the “cloud is more expensive at scale” argument gets dropped like a flaming bag of poo.
  6. Hybrid cloud
    1. Various stuff like directory service (AD in the cloud) and identity federation and storage gateway and SystemCenter and vCenter integration already exist to power mixed shops
    2. Johnson & Johnson went on for a while about their use of AWS.  They are planning a 25,000 seat deployment of Workspaces (virtual desktop offering, like Citrix).

Whew, that’s the quick notes version.  Aurora is obviously of interest – a lot of the fretting over whether to use mySQL or RDS I’ve seen will get settled by this – it was just ‘well, run the same thing yourself or have them do it…” and now it’s “have them run something insanely better”. But the SDLC tools are also interesting – they made noise about how these “work with!” ansible, jenkins, git, etc. but that seems mildly disingenuous, without any more looking into it yet they sound more like direct competition for them. But the config and service catalog could be great extensions – yay for simple composable services, not huge painful “BSM/ITMOM suites”.

Feel free and share your thoughts on the announcements in the comments section!

3 Comments

Filed under Cloud, Conferences

LASCON Interview: Jason Chan

 IMG_1513Jason Chan (@chanjbs) is an Engineering Director of the Cloud Security team at Netflix.

Tell me about your current gig!

I work on the Cloud Security team at Netflix, we’re responsible for the security of the streaming service at Netflix.  We work with some other teams on platform and mobile security.

What are the biggest threats/challenges you face there?

Protecting the personal data of our members of course.  Also we have content we want to protect – on the client side via DRM, but mainly the pipeline of how we receive the content from our studio partners. Also, due to the size of the infrastructure, its integrity – we don’t want to be a botnet or have things injected to our content that can our clients.

How does your team’s approach differ from other security teams out there?

We embody the corporate culture more, perhaps, than other security teams do. Our culture is a big differentiator between us and different companies.  So it’s very important that people we hire match the culture. Some folks are more comfortable with strong processes and policies with black and white decisions, but here we can’t just say now, we have to help the business get things done safely.

You build a security team and you have certain expertise on it.  It’s up to the company how you use that expertise. They don’t necessarily know where all the risk is, so we have to provide objective guidance and then mutually come to the right decision of what to do in a given situation.

Tell us about how you foster your focus on creating tools over process mandates?

We start with recruiting, to understand that policy and process isn’t the solution.  Adrian [Cockroft] says process is usually organizational scar tissue. By doing it with tools and automation makes it more objective and less threatening to people. Turning things into metrics makes it less of an argument. There’s a weird dynamic in the culture that’s a form of peer pressure, where everyone’s trying to do the right thing and no one wants to be the one to negatively impact that.  As a result people are willing to say “Yes we will” – like, you can opt out of Chaos Monkey, but people don’t because they don’t want to be “that guy.”

We’re starting to look at availability in a much  more refined way.  It’s not just “how long were you down.”  We’re establishing metrics over real impact – how many streams did we miss?  How many start clicks went unfulfilled.  We can then assign rough values to each operation (it’s not perfect, but based on shared understanding) and then we can establish real impact and make tradeoffs. (It’s more story point-ish instead of hard ROI). But you can get what you need to do now vs what can wait.

Your work  – how much is reactive versus roadmapped tool development?

It’s probably 50/50 on our team.  We have some big work going on now that’s complex and has been roadmapped for a while.  We need to have bandwidth as things pop up though, so we can’t commit everyone 100%. We have a roadmap we’ve committed to that we need to build, and we keep some resource free so that we can use our agile board to manage it. I try to build the culture of “let’s solve a problem once,” and share knowledge, so when it recurs we can handle it faster/better.  I feel like we can be pretty responsive with the agile model, our two week sprints and quarterly planning give us flexibility. We get more cross-training too, when we do the mid-sprint statuses and sprint meetings. We use our JIRA board to manage our work and it’s been very successful for us.

What’s it like working at Netflix?

It’s great, I love it.  It’s different because you’re given freedom to do the right thing, use your expertise, and be responsible for your decisions. Each individual engineer gets to have a lot of impact on a pretty large company.  You get to work on challenging problems and work with good colleagues.

How do you conduct collaboration within your team and with other teams?

Inside the team, we instituted once a week or every other week “deep dives” lunch and learn presentation of what you’re working on for other team members. Cross-team collaboration is a challenge; we have so many tools internally no one knows what they all are!

You are blazing trails with your approach – where do you think the rest of the security field is going?

I don’t know if our approach will catch on, but I’ve spent a lot of my last year recruiting, and I see that the professionalization of the industry in general is improving.  It’s being taught in school, there’s greater awareness of it. It’s going to be seen as less black magic, “I must be a hacker in my basement first” kind of job.

Development skills are mandatory for security here, and I see a move away from pure operators to people with CS degrees and developers and an acceleration in innovation. We’ve filed three patents on the things we’ve built. Security isn’t’ a solved problem and there’s a lot left to be done!

We’re working right now on a distributed scanning system that’s very AWS friendly, code named Monterey. We hope to be open sourcing it next year.  How do you inventory and assess an environment that’s always changing? It’s a very asynchronous problem. We thought about it for a while and we’re very happy with the result – it’s really not much code, once you think the problem through properly your solution can be elegant.

1 Comment

Filed under Cloud, Conferences, Security

LASCON Interview: Nick Galbreath

IMG_1509Nick Galbreath (@ngalbreath) is VP of Engineering with client9, LLC.

What are you doing nowadays since leaving Etsy?

I am managing a small DevOps team for a company whose engineering team is based in Moscow, from Tokyo, Japan. Some other executives and our biggest customer is from there. And, I love Japan!

I know you from Velocity and the other DevOps conferences. Why are you here at a security conference?

I’ve been active at Black Hat, DEFCON, etc. as well as DevOps conferences. I’ve found that if your company is in operational chaos you don’t need security.  Once you have a good operational component and it’s not in chaos – standardized infrastructure, automation – you get up to the level where you can be effective at security.  I used the same approach at Etsy – I started there working on security, stopped, worked in infrastructure until that was basically squared away, and only then started working on security again. You have to work your way up Maslow’s hierarchy.

It’s the same with development. My background is originally development and when you’re programming in C/C++ your main effort is stability, but all those NPEs and other bugs are also security issues.  I don’t know any company doing well at security and not well at development, I’m not sure you can do it. Nail the basics and then the advanced topics are achievable.

What’s your opinion on how much the security space has left developers behind?

Look at the real core issues behind security. Dev teams have trouble with writing secure code, ops folks have problems with patching – at security conferences you don’t see anything for solving those problems.  Working on offense/breaking and blocking tools is lucrative but inhibits us from going after the root causes.

For many security pros, working in a team instead of solo is a different skill set. “We don’t want to bother the developers with this” – siloed approaches are killing us.

What do you see as the most interesting thing going on in the security landscape right now?

What has happened in the last 3-4 months, as much as I hate to say it, with all the leaking of documents – we’ve been lazy about encryption and privacy and other foundational elements and we assumed it worked, now we’re doing some healthy review to do a next generation of those. It brought that discussion to the forefront. The certificate authority problems, and the NSA stuff – we need to spend some time and think about this.  The next generation of SSL and certificate transparency are very interesting.

In terms of pure language work… Improvement of cryptography. Also, we’re making more business level APIs for common problems like PHP5’d password hashing APIs.  If your’e building a Web app and need auth you’re starting from zero most of the time and now you’re starting to see things put into the languages that solve these problems.

Out in the larger DevOpsey world, what are the things to watch, what is your team excited about?

Stuff that we’re excited about is traditional devops stuff like really treating our infrastructure like code.  No button clicking, infrastructure completely specified in config files in source control, code reviews, and then the file pushed to production to allocate/deallocate hardware and deploy software.  That’s a big change.

How do we disseminate best practices/prevent worst practices through those who aren’t the technical “1%?”

Well, best practices are harder

People went into server programming because they don’t like doing user interface stuff. But the joke’s on us, there is still a user interface, it’s configuration files, installers, etc. which are nontrivial. We should either be bundling audit software or server-side config healthchecks to provide warnings. “Why do you have SSL v2 enabled?” “Why are your .htaccess files visible by default?” [Ed: Where the hell did apache chkconfig go?]

People in ops can write these but retroactively folks won’t use them… But the future can have them.  If you at least get warned that your Apache config is using suboptimal security configs it’s your deliberate negligence to not do it right.

Maybe take the module approach (Apache wouldn’t want it in their core I’m sure) – if you want to work on it give me a call!

What message do you want to send to other security folks?

For security people, the message is, “It’s really important you start bringing your  non-security friends to these security conferences.” Devs and ops and business and QA. They’ll find it interesting and get involved. It’s really important.

Last year, we had a dozen people from my company come out to AppSec. But except for me and our security team, they’re not back this year. There just wasn’t enough content to hold the interest of the devs. What can we do about that?

Really!  Interesting.  Maybe we need more of a proper dev track, with more things like Karthik’s talk.

A project I’ve wanted to do for a very long time – most people in business and development don’t have  real idea of how much damage can be done, it’s why we have Red Teams. If someone’s really good at SQLi, etc. do a talk showing how much damage can be done.

Also – if you work at any company, you depend on an immense set of open source software and they don’t have a security person or anything.  Get involved in their process, try to help them and make it better and it’ll improve quality of everyone’s systems. We could do a hackathon during the convention to improve some existing projects.

Leave a comment

Filed under Conferences, DevOps, Security

LASCON 2013 Report – Second Afternoon

I’m afraid I only got to one session in the afternoon, but I have some good interviews coming your way in exchange!

User Authentication For Winners!

I didn’t get to attend but I know that Karthik’s talk on writing a user auth system was good, here are the slides. When we were at NI he had to write the login/password/reset system for our product and we were aghast that there was no project out there to use, you just had to roll your own in an area where there are so many lurking security flaws.  He talks about his journey and you should read it!

AWS CloudHSM And Why It Can Revolutionize Cloud

Oleg Gryb (@oleggryb), security architect at Intuit, and Todd Cignettei, Sr. Product Manager with AWS Security.

Oleg says: There are commonly held concerns about cloud security – key management, legal liability, data sovereignty and access, unknown security policies and processes…

CloudHSM makes objects in partitions not accessible by the cloud provider. It provides multiple layers of security.

[Ed. What is HSM?  I didn’t know and he didn’t say.  Here’s what Wikipedia says.]

Luckily, Todd gets up and tells us about the HSM, or Hardware Security Module. It’s a purpose built appliance designed to protect key material and perform secure cryptographic operations. The SafeNet Luna SA HSM has different roles – appliance administrator, security officer. It’s all super certified and if tampered with blows up the keys.

AWS is providing dedicated access to SafeNet Luna SA HSM appliances. They are physically in AWS datacenters and in your VPC. You control the keys; they manage the hardware but they can’t see your goodies. And you do your crypto operations there. Here’s the AWS page on CloudHSM.

They are already integrated with various software and APIs like Java JCA/JCE.

It’s being used to encrypt digital content, DRM, securing financial transactions (root of trust for PKI), db encryption, digital signatures for real estate transactions, mobile payments.

Back to Oleg. With the HSM, there’s some manual steps you need to do, Initialize the HSM, configure a server and generate server side certs, generate a client cert on each client, scp the public portion to the server to register it.

Normal client cert generation requires an IP, which in the cloud is lame. You can isntead use a generic client name and use the same one on all systems.

You put their LunaProvider,jar in your Java CLASSPATH and add the provider to java/security and you’re good to go.

Making a Luna HA array is very important of course. If you get two you can group them up.

Suggested architecture – they ahve to run in a VPC. “You want to put on Internet? Is crazy idea! Never!”

Crypto doesn’t solve your problem, it just moves it to another place. How do you get the secrets onto your instances? When your instance starts, you don’t want those creds in S3 or the AMI…

So at instance bootstrap, send a request to a server in an internal DC with IP, instance ID, public and local hostanmes, reservation ID, instance type… Validate using the API including instance start time, validate role, etc. and then pass it back. Check for dupes.  This isn’t perfect but what are ya gonna do?  You can assign a policy to a role and have an instance profile it uses.

He has written a Python tool to help automate this, you can get it at http://sf.net/p/lunamech.

1 Comment

Filed under Conferences, Security

LASCON 2013 Report – Second Morning

Everyone shuffles in slowly on the second morning of the con. I spent the pre-keynote hour with other attendees sitting around looking tired and comparing notes on gout symptoms.  (PSA: if the ball of your foot starts hurting really bad one day, it’s gout, take a handful of Advil and go to your doctor immediately.)

  • Impact Security
  • NetIQ
  • SWAMP

You can also see a bunch of great pictures from the event courtesy Catherine Clark!

Blindspots

The keynote this morning is from Robert “RSnake” Hansen, now of White Hat. It’s about blind spots we all have in security.  Don’t take this as an attack, be self reflective.

Blindspot #1 – Network & Host Security

Internetworked computers is a very complex system and few of us 100% understand every step and part of it.

How many people do network segregation, have their firewall on an admin network, use something more secure than a default Linux install for their webservers, harden their kernel, log off-host and log beyond standard logs? These are all cheap and useful.

Like STS, it was only considered very tightly and the privacy considerations weren’t identified.

Blindspot #2 – Travel and OPSEC

Security used to be more of a game. Now the internet has become militarized. Don’t travel with your laptop. Because – secret reasons I’ll tell you if you ask. (?)

[Ed. Apparently I’m not security 3l33t enough to know what this is about, he really didn’t say.]

Blindspot #3 – Adversaries

You seed to be able to see things from “both sides” and know your adversary (personally ideally). Some of them want to talk! Don’t send them to jail, talk and learn. Yes, you can.

Blindspot #4 – Target Fixation

Vulnerabilities aren’t created equal. Severities vary. DREAD calculations vary widely. Don’t trust a scanner’s DREAD. Gut check but then do it on paper because your gut is often not correct. Often we have “really bad!” vulnerabilities we obsess about that aren’t really that severe.

Download Fierce to do DNS enumeration, do bing IP search, nmap/masscan/unicornscan for open ports.

Blindspot #5 – Compliance vs Security

These aren’t very closely related.  Compliance gets you little badges and placated customers. Security actually protects your systems and data. Some people exercise willful negligence when they choose compliance over security. Compliance also pulls spend to areas that don’t help security. Compliance doesn’t care about what hackers do and it doesn’t evolve quickly.

Blindspot #6 – The Consumer

Consumers don’t really understand the most rudimentary basics of how the Internet works and really don’t understand the security risks of anything they do. They’re not bad or stupid but they can’t be expected to make well informed decisions. So don’t make security opt in.

We the security industry are not pro-consumer – we’re pro-business. Therefore we may be the first ones against the wall when the revolution comes. Give them their privacy now.

So pick one, work on it, we’ll be less blind!

Big Data, Little Security?

By Manoj Tripathi from PROS in Houston.

Big Data is still emerging and doesn’t have the mature security controls that older data platforms have.

Big data is a solution to needs for high volume, high velocity, and/or rich variety of data.  Often distributed, resilient, and not hardware constrained (but sometimes is).

Hadoop is really a framework, with HDFS, Zookeeper, mapreduce, pig/hive, hbase (or cassandra?). He’ll talk a lot about this framework because it’s so ubiquitous.

NoSQL – Cassandra (eventually consistent, highly available, partition tolerant), MongoDB (consistent, partition tolerant).

Security is an afterthought in Big Data.  It can be hard to identify sensitive data (schemaless). He says there’s provenance issues and enhanced insider attacks but I don’t know… Well, if you consider “Big Data” as just large mineable data separate from the actual technology, then sure, aggregate data insights are more valuable to steal… His provenance concern is that data is coming from less secured items like phones/sensors but that’s a bit of a strawman, the data sources for random smaller RDBMSes aren’t all high security either…

Due to the distributed architecture of hadoop etc. there’s a large attack surface. Plus Hadoop has multiple communication protocols, auth mechanisms, endpoint types… Most default settings in Hadoop on all of these are “no security” and you can easily bypass most security mechanisms, spoof, accidentally delete data… Anonymous access, username in URL, no perm checking, service level auth disabled, etc.

Hadoop added Kerberos support, this helps a lot. You can encrypt data in transit, use SSL on the admin dashboards.

But – it’s hard to configure, and enterprises might not like “another” auth infrastructure. It also has preconditions like no root access to some machines and no communication over untrusted networks. And it has a lot of insecure-by-default choices itself (symmetric keys, http SPNEGO has to be turned on in browsers, Oozie user is a super-user with auth disabled by default). No encryption at rest Kerberos RPC is unencrypted. Etc, etc, etc.

To Cassandra.  Same deal. CLI has no auth by default. Insecure protocols.

NoSQL vulns – injections just like with SQL. Sensitive data is copied to various places, you can add new attributes to column families.

Practical Steps To Secure It

Cassandra – write your own authorization/authentication plugin.  [Ed. Really?] But this has keyspace and column family granularity only. 1.2 has internal auth. Enable node-node and client-node encryption. If you do this at least it’s not naiively vulnerable. Also, use disk support for encryption.

Hadoop – basically wait for Project Rhino. Encryption, key mgmt, token based unified auth, cell level auth in hbase. Do threat modeling. Eliminate sensitive data, use field level encryption for sensitive fields, use OS or file level encryption mechanisms. Basically, run it in a secured environment or you’re in trouble.  Apache Knox can enforece a single point of access for auth to Hadoop services but has scalability/reliability issues. Can turn on kerberos stuff if you have to…

Also. commercial hadoop/cassandra have more options.

Leave a comment

Filed under Conferences, Security