the agile admin

I (@wickett) work as the Cloud Ops Team Lead at National Instruments where we have several Software as a Service products that we have built on different cloud providers (AWS, Azure, Google) and have implemented with a host of other supporting SaaS tools (cloudkick, ZenDesk, AlertSite and PagerDuty plus several others).  When building out our products we decided to eat our own dog food and use SaaS solutions as much as possible.  Great, but where am I going with all this?

No matter what tools you are using to monitor or log on your systems, you need a reliable way to get actionable events to your Ops Team.  If you have implemented several types of monitors, you generally are setting up an email address for them to send alerts to.  Then you write scripts to forward those to on call devices or forwarding rules to turn them into SMS–not exactly state of the art.  Try mixing in a global on call rotation and trying to configure all of your monitoring tools to account for that and it becomes a big problem.

Enter PagerDuty–by far this is the best SaaS product we use in our day-to-day Operations team.  PagerDuty is an alerting tool that is simple, easy-to-use and integrates into your other systems.   Why is PagerDuty so awesome?  Well, I am glad you asked.

1. User defined escalation.  Once an alert gets sent into PagerDuty, it is processed through our escalation pathway.  It determines who is the first level of support and begins to alert that person.  Here is the cool part, I can choose to be alerted however I want and other ops team members can choose however they want.  For me, I get an email after 1 minute, SMS alerts at 3 minute intervals for the next 9 minutes, then phone calls every 5 minutes for the next 20 minutes.  Lets say you are a hard sleeper then you might want to skip the SMS and move straight to phone calls.  If I don’t acknowledge the alert in 30 minutes, it will get escalated to the next ops team member.
2. Alert Acknowledgement.  From any one of the alert mechanisms above, there is an in-kind way to acknowledge and resolve the alert.  I can reply to the SMS message with an ACK code or when I get a call from the PagerDuty version of Siri I can select a response right on my keypad at that time.  No time is lost logging into PagerDuty to acknowledge the alert and the ops team can just get busy responding.
3. Equality of alerts.  This is a subtle one.  We have a policy on our team that all alerts are equal and need to be handled with the same care and diligence.  Anything that makes it to PagerDuty is treated with the same level of importance and is escalated through the same channel–no “you can just ignore those alerts from that system over there” syndrome on my team.  All alerts are escalated and all must be handled.
4. API integration.  PagerDuty lets you integrate with tools you already use (e.g. nagios, zenoss, cloudkick, splunk) and those tools can open and close alerts as they are detected and/or resolved.
5. Email integration. Even if you have created some code or monitor that you want to alert from that doesn’t integrate with the API then all you need to be able to do is send an email.  Once that email is received, PagerDuty will treat it as an alert.
6. Global 24/7 tools.  PagerDuty works in lots of countries and has scheduling that allows for follow-the-sun ops teams to thrive.  I am on call every day for 8 hours and after my shift ends one of my other global ops team members is on call (@einsamsoldat and @hafizramly) for the next 8 hours–at which point I am bumped to the second tier escalation path.  Most tools miss this and for our team this is a huge benefit.

I had initially thought I would just write a quick paragraph or two about why using PagerDuty for alerting is great for your devops team.  But, I am such a big fan that I couldn’t resist coming up with more reasons why we love PagerDuty on our team.  The biggest reason of all is that as an Ops team manager, I can sleep at night knowing that all alerts will get handled and I won’t woken up with a phone call from a VP or marketing person telling me that our cloud products are down–well at least not because of a failure to handle alerts.

I would encourage you to stop using subpar alerting mechanisms in monitoring and logging tools that don’t treat alerting as a first class citizen.  Those tools were’t created to be awesome at alerting they were meant to be detective in nature.  PagerDuty is made for alerting and defintely deserves a spot in your devops toolchain.

Last night I gave a 5 minute lightning presentation on Vagrant for the Austin Cloud User Group’s December meeting which was aptly titled “The 12 Clouds of Christmas.”  These ’12 clouds’ fleshed out into 12 lightning talks on different clouds and implementation thereof.  The format was great and thought that it allowed everyone to get exposure to new tech.

Below are the slides from the demo.  Slides 9 and 10 are where I showed the actual setup (Vagrantfile, rvm), virtual box console and ran vagrant commands.  Squint real hard and tilt your head to the right and maybe you can envision the actual demo portion of the talk…  Or if your imagination fails you, you can watch some random vagrant demos on youtube.

I had a chance to talk with Dan Cornell from the Open Web Application Security Project (OWASP) and the Denim Group.  Dan has over 13 years of experience in development and is one of the founders at the Denim Group, a company that does security consulting day in and day out for all types of clients. Dan wrote Sprajax, a security analyzer for AJAX applications, and is the chair of the OWASP Global Membership Committee.

While a lot of us work with security, it is usually in addition to other dev or ops responsibilities and isn’t our sole focus.  However, Dan Cornell and his company are in the weeds of app security daily.  It is a pleasure to have him on the agile admin blog and we are chatting with him on his thoughts on the Rugged Software Manifesto. We ran across this manifesto at the Lonestar Application Security Conference (LASCON) and heard it referred to by both OWASP board members and Department of Homeland Security types so we thought we should look it up, and it seemed interesting.  So here’s Dan’s thoughts on the subject.

@wickett: Hi Dan, welcome to the agile admin blog.  Our first question for you is, what do we mean by Rugged Software?

@danielcornell:  I think of software that is going to run as intended, regardless of the conditions it encounters when it is deployed.  A big part of this is security, but that isn’t the only thing.  You also have to take into account reliability, survivability and a variety of other ‘-ity’ properties.  The Internet, and networks in general, are dangerous places where you need to be conscious of both malicious actors as well as other, unintentional adverse conditions such as service outages, congestion, untrained users making mistakes and so on.  Rugged Software is software that is designed and built to deal with these conditions – foreseen or not.

@wickett: When we have talked about Rugged Software with developers, we get a fair amount of pushback and hear statements like, “Rugged Software isn’t really real and that it is just a PR play.”  Some developers and engineers we;ve talked to say that it lacks depth, clarity and process, but I wonder if these engineers have even seen the original Agile Manifesto.  In my mind, I look back at the original Agile Manifesto and I am equally under-whelmed by it as I imagine they are by the Rugged Software Manifesto.  Do you see any parallels between the original Agile Manifesto and the Rugged Software Manifesto?

@danielcornell: The two biggest challenges I see for the Rugged Software movement are to provide a business justification for being Rugged to management and to answer the “so what?” question for developers.  Dealing with the first will go a long way toward solving the second.

If you ask most executives and managers if they’d like their software to be Rugged they’ll likely give an answer somewhat along the lines of: “Sure I’d love to be ‘Rugged.’  And ‘Agile’ and ‘Pragmatic’ and whatever other buzzwords the industry has come up with lately.  All that stuff is great but what I really want is to be on time and on budget and I want more features than our competitors.”  Thinking about security and quality and performance and anything else comes later.

So this is an area where the Rugged movement needs to do a better job of explaining why Rugged makes good business sense and you run into the same challenges that information security folks have had making a business justification for security for years.  Allowing an organization to avoid pain when the auditors or compliance folks come around is of value – but it often isn’t enough.  Reducing downtime and disruption in the future because you have avoided a breach or outage is of value – but it often isn’t enough.  David Rice has done some great thinking in this area in his book Geekonomics but, as an industry, I think we are still looking for the universal truth to be revealed that will make everyone sit up and take notice and I think we’re going to be waiting for a long time.

That said, some organizations have made the decision that this is an area that merits focus and when executives and managers make Rugged (or security or whatever) a priority then it is much easier to get the troops to fall in line.  I’m reminded of a secure coding training class I ran recently for an ISV where the VP in charge of engineering stood up at the beginning of the class and said “This is a priority for us and the material in this class represents the base level of knowledge everyone is expected to have.”  Then he sat in the entire two day class and at the end told his people “I hope you paid attention because you are going to be held accountable for knowing and implementing this material.”  Wow!  This was one of the best classes I taught in a long time; everyone showed up on time, paid attention, asked questions and so on.  I’d love to think it is because I’m such a funny and engaging guy but what really made the difference is that the leadership demonstrated with actions that security is a priority.  A similar model is always going to be the most effective way to drive Rugged into an organization.  Bottom-up, grass roots efforts are valuable because they raise awareness and help to identify like-minded potential champions, but to drive adoption at scale management needs to feel that this is a priority.

Speaking of priorities, I think this is an area where the Agile Manifesto style can provide some guidance.  The Rugged Manifesto is a great call to arms, but it is very self-referential: “I am Rugged… and more importantly my code is Rugged”  And so on.  If you look at the Agile Manifesto it calls out trade offs: “Individuals and interactions over processes and tools,” for example.  It accepts that processes and tools have merit, but chooses to value individuals and interactions more.  Acknowledging that decisions must be made and providing guidance on why one is superior to another is a stronger basis for a conversation and this is going to be key.  Right now those trade-offs are presented as “Prefer code without vulnerabilities over code with vulnerabilities”  Duh!  Who wouldn’t want that?  But the real question is what are they going to have to give up to get it.

@wickett: Over the next five years, do you see the Rugged Software Manifesto becoming a standard?  Will we have Rugged development practices and be operating on a new paradigm for development or will it get lumped under Agile?

@danielcornell: Well the Rugged Software Manifesto is just that – a Manifesto.  It isn’t a standard.  And I don’t think we are ever going to see a standard for software development because “development” is too big a thing to be done in a standard way.  You can’t expect folks building banking applications for East Coast financials to follow the same process as a couple of folks in a garage trying to build the next Facebook.  What I hope is developed and adopted, however, are so-called “best practices” such as threat modeling, static analysis for quality and security, security code review, penetration testing and so on.

Also, Agile and Rugged are two separate movements.  Agile is about creating customer-centric software in a flexible and timely manner and Rugged is about making software that doesn’t break.   Their goals are not exclusive, but they’re also different.

@wickett: When thinking about Rugged Software, what are practical steps that individual developers can adopt into their current practices whether they use Agile, Waterfall, or other development methodology?

@danielcornell: Well first I think it is important to note that individual developers can have a positive impact on their own code, but this is going to be largely ineffective if the rest of the development team doesn’t share the same goals.  The “weakest link in the chain” analogy comes to mind.  That said…

Threat Modeling is a practice that can be adopted into any development methodology.  There are lots of resources out there that explain how to use it to improve the security of complicated systems and it can also be used for the greater purpose of Rugged software by asking questions such as “what if this data flow stop flowing” or “what if this data store becomes unavailable”  This is best done by the team as a whole but it is also a practice that an individual developer can use on their own portions of a system.

Penetration testing and resiliency testing are other practices that teams can implement to help test the Ruggedness of their applications and systems.

@wickett: What open source tools do you think should fit into the tool chain for a rugged developer?  What about for a rugged operations/sysadmin?

@danielcornell: Static analysis tools like FindBugs, PMD and FxCop can be really valuable.  For dynamic testing a web proxy like WebScarab can be really useful to testing how software is going to respond to unexpected inputs.

On the operations/sysadmin thing a critical Rugged practice is to proactively monitor systems in order to anticipate problems.  From the standpoint, you should be instrumenting systems and not just tracking failures.  When an HTTP server goes down it is often too late to prevent an outage and that isn’t very Rugged.  Instead you also need to be tracking disk and processor usage, traffic and response time trends and so on in order to be able to anticipate problems.  Fortunately there are a lot of open source tools available for DIY folks as well as SaaS offerings that can make this sort of instrumentation less of a hill to climb that it was in the past.

@wickett: We appreciate having time with Dan and hearing from an industry leader on Rugged Software.  To keep up Dan on the interweb, you can follow him on twitter or check out his blog.

In late 2007 Bruce Schneier, the internationally renowned security technologist and author, wrote an article for IEEE Security & Privacy. The ominously named article: The Death of the Security Industry predicted the future of the security industry or lack thereof.  In it he predicts that we would treat security as merely a utility like we use water and power today.  The future is one where “large IT departments don’t really want to deal with network security. They want to fly airplanes, produce pharmaceuticals, manage financial accounts, or just focus on their core business.”

Schneier closes with, “[a]s IT fades into the background and becomes just another utility, users will simply expect it to work. The details of how it works won’t matter.”

Looking back 3 years and having the luxury of hindsight, it is understandable to see why he thought the security industry would become a utility.  In part, it has become true.  Utility billing is the rage for infrastructure (hello cloud computing) and more and more people are viewing the network as a commodity.  Bandwidth has increased in performance and decreased in cost.  Continually people are outsourcing pieces of their infrastructure and non-critical IT services to vendors or to offshore employees.

But there are three reasons why I disagree with the The Death of the Security Industry and I believe we are actually going to see a renaissance of the security industry over the next decade.

1. Data is valuable. We can’t think of IT as merely the computers and network resources we use.  We need to put the ‘I’ back in IT and remember why we play this game in the first place.  Information.  Protecting the information (data) will be crucial over the long haul.  Organizations do not care about new firewalls or identity management as a primary goal, however they do care about their data.  Data is king.  Organizations that succeed will be ones that master navigating a new marketplace that values sharing while keeping their competitive edge by safe-guarding and protecting their critical data.

2. Security is a timeless profession. When God gave Adam and Eve the boot from the Garden of Eden, what did he  do next?   He used a security guard to keep them out of the Garden for good.  Security has been practiced as long as people have been people.  As long as you have something worth protecting (see ‘data is valuable’ in point 1) you will need resources to protect it.   Our valuable data is being transferred, accessed and modified on computing devices and will need to be protected.  If people can’t trust that their data is safe then they will not be our customers.  The CIA security triad (Confidentiality, Integrity, and Availability) needs to remain in tact for consumers to trust organizations with their data and if that data has any value to the organization, it will be need to be protected.

3. Stuxnet. This could be called the dawn of a new age of hacking.  Gone are the days of teenagers running port scans from their garages. Be ready to start seeing hackers using sophisticated techniques that simultaneously attack multiple vectors to gain access on their targets.  I am not going to spread FUD (Fear Uncertainty and Doubt) around, but I believe that Stuxnet is just the beginning.

In addition to how Stuxnet was executed, it is just as interesting to see what was attacked.  This next decade will prove to be a change in the type of targets attacked.  In the 80′s it was all about hacking phones and more physical targets, the 90′s were the days of the port-scanning and Microsoft Windows hacking, the last decade has primarily focused on web and application data.  With Stuxnet, we are seeing the revitalization of hacking where it is returning to its roots of hacking targets that are physical in nature such as SCADA systems that control a building’s temperature systems.  The magazine 2600 has been publishing a series on SCADA hacking over the last 18 months.  What makes it even more interesting is that almost every device you buy these days has a web interface on it, so never fear, the last 10 years spent hacking websites will come in real handy when looking at hacking control systems.

In closing, I think we are a long way off from seeing the death of the security industry.  As our data becomes more valuable, the more we will need to secure.  Data is on the rise and with it comes the need for security.  Additionally as more and more of our world is controlled with computers, the targets become more and more interesting.  Be ready for the rise of the security industry.

Let me know what you think on twitter: @wickett

Recently I have been reading on OPSEC (operations security).  OPSEC, among many things, is a process for security critical information and reducing risk.  The 5 steps in the OPSEC process read as follows:

1. Identify Critical Information
2. Analyze the Threat
3. Analyze the Vulnerabilities
4. Assess the Risk
5. Apply the countermeasures

It really isn’t rocket science, but it is the sheer simplicity of the process that is alluring.  It has traditionally been applied in the military and has been used as a meta-discipline in security.  It assumes that other parties are watching, sort of like the aircraft watchers that park near the military base to see what is flying in and out, or the Domino’s near the Pentagon that reportedly sees a spike in deliveries to the Pentagon before a big military strike.  Observers are gathering critical information on your organization in new ways that you weren’t able to predict.  This is where OPSEC comes in.

Since there is no way to predict what data will be leaking from your organization in the future and it is equally impossible to enumerate all possible future risk scenarios, then it becomes necessary to perform this assessment regularly.  Instead of using an annual review process with huge overhead and little impact (I am looking at you, Sarbanes-Oxley compliance auditors), you can create a process to continue to identify risks in an ever-changing organization while lessening risk.  This is why you have a security team, right?  Lessening the risk to the organization is the main reason to have a security team.  Achieving PCI or HIPPA compliance is not.

Using OPSEC as a security process poses huge benefits when aligned with Agile software development principles.  The following weekly assessment cycle is promoted by SANS in their security training course.  See if you can see Agile in it.

The weekly OPSEC assessment cycle:

1. Identify Critical Information
2. Assess threats and threat sources including: employees, contractors, competitors, prospects…
3. Assess vulnerabilities of critical information to the threat
4. Conduct risk vs. benefit analysis
5. Implement appropriate countermeasures
6. Do it again next week.

A weekly OPSEC process is a different paradigm from the annual compliance ritual.  The key of the security is just that: lessen risk to the organization.   Iterating through the OPSEC assessment cycle weeklymeans that you are taking frequent and concrete steps to facilitate that end.

There are a couple military concepts that have bled into technology and in particular into IT security, a DMZ being one of them. A Demilitarized Zone (DMZ) is a concept where there is established control for what comes in and what goes out between two parties and in military terms you can establish a “line of control” by using a DMZ. In a human-based DMZ, controllers of the DMZ make ingress (incoming) and egress (outgoing) decisions based on an approved list–no one is allowed to pass unless they are on the list and have proper identification and approval.

In the technology world, the same thing is done based on traffic between computers. Decisions to allow or disallow the traffic can be made based on where the traffic came from (origination), where it is going (destination) or even dimensions of the traffic like size, length, or even time of day. The basic idea being that all traffic is analyzed and is either allowed or disallowed based on determined rules and just like in a military DMZ, there is a line of control where only approved traffic is allowed to ingress or egress. In many instances a DMZ will protect you from malicious activity like hackers and viruses but it also protects you from other configuration and developer errors and can guarantee that your production systems are not talking to test or development tiers.

Lets look at a basic web tiered architecture. In a corporation that hosts its own website they will more than likely have the following four components: incoming internet traffic, a web server, a database server and an internal network. To create a DMZ or multiple DMZ instances to handle their web traffic, they would want to make sure that someone from the internet could only talk to the web server, they would also want to verify that only the web server can only talk to the database server and they would want to make sure that their internal network is inaccessible by the the web server, database server and the internet traffic.

Using firewalls, you would need to set up at least the below three firewalls to adequately control the DMZ instances:

1. A firewall between the external internet and the web server
2. A firewall in front of the internal network
3. A firewall between your web servers and database server

Of course these firewalls need to be set so that they allow (or disallow) only certain types of traffic. Only traffic that meets certain rules based on its origination, destination, and its dimensions will be allowed.

Sounds great, right? The problem is that firewalls have become quite complicated and now sometimes aren’t even advertised as firewalls, but instead are billed as a network-device-that-does-anything-you-want-that-can-also-be-a-firewall-too. This is due in part to the hardware getting faster, IT budgets shrinking and scope creep. The “firewall” now handles VPN, traffic acceleration, IDS duties, deep packet inspection and making sure your employees aren’t watching YouTube videos when they should be working. All of those are great, but it causes firewalls to be expensive, fragile and difficult to configure.  And to have the firewall watching all ingress and egress points across your network, you usually have to buy several devices to scatter throughout your topology.

Additionally, another recurring problem is that most firewall analysis and implementation is done with an intent to secure the perimeter. Which make sense, but it often stops there and doesn’t protect the interior parts of your network. Most IT security firms that do consulting and penetration tests don’t generally come through the “front door” by that I mean, they don’t generally try to get in through the front-facing web servers but instead they will go through other channels such as dial-in, wireless, partner, third-party services, social engineering, FTP servers or that demo system that was setup back 5 years ago that no hasn’t been taken down–you know the one I am talking about. Once inside, if there are no well-defined DMZs, then it is pretty much game-over because at that point there are no additional security controls. A DMZ will not fix all your problems, but it will provide an extra layer of protection that could protect you from malicious activity. And like I mentioned earlier it can also help prevent configuration errors crossing from dev to test to prod.

In short, a DMZ is a really good idea and should be implemented for every system that you have. The most optimal DMZ would be a firewall in front of each service that applies rules to determine what traffic is allowed in and out. This, however, is expensive to set up and very rarely gets implemented. That was the old days, this is now and good news, the cloud has an answer.

I am most familiar with Amazon Web Services so we will include an example of how do this with security groups from an AWS perspective. The following code creates a web server group and a database server group and allows the web server to talk to the database on port 3306 only.

ec2-add-group web-server-sec-group -d "this is the group for web servers" #This creates the web server group with a description
ec2-add-group db-server-sec-group -d "this is the group for db server" #This creates the db server group with a description
ec2-authorize web-server-sec-group -P tcp -p 80 -s 0.0.0.0/0 #This allows external internet users to talk to the web servers on port 80 only
ec2-authorize db-server-sec-group --source-group web-server-sec-group --source-group-user AWS_ACCOUNT_NUMBER -P tcp -p 3306 #This allows only traffic from the web server group on port 3306 (mysql) to ingress

Under the above example the database server is in a DMZ and only traffic from the web servers are allowed to ingress into it. Additionally, the web server is in a DMZ in that it is protected from the internet on all other ports except for port 80. If you were to implement this for every role in your system, you would be in effect implementing a DMZ between each layer and would provide excellent security protection.

The cloud seems to get a bad rap in terms of security. But I would counter that in some ways the cloud is more secure since it lets you actually implement a DMZ for each service. Sure, they won’t do deep packet analysis or replace an intrusion detection system, but they will allow you to specifically define what ingresses and egresses are allowed on each instance.  We may not ever get a chicken in every pot, but with the cloud you can now put a DMZ on every service.

I thought I would take this opportunity to invite the agile admin readers to LASCON.   LASCON (Lonestar Application Security Conference) is happening in Austin, TX on October 29th, 2010. The conference is sponsored by OWASP (the Open Web App Security Project) and is an entire day of quality content on web app security.  We’ll be there!

The speaker list is still in the works, but so far we have two presentations from this years BlackHat conference, several published authors, and the Director for Software Assurance in the National Cyber Security Division of the Department of Homeland Security just to name a few, and that’s only the preliminary round of acceptances.

Do you remember a few years ago when there was a worm going around MySpace that infected user profile pages at the rate of over one million in 20 hours?  Yeah, the author of that worm is speaking at the conference.  How can you beat that?

I have been planning this conference for a few months and am pretty excited about it.  If you are can make it to Austin on October 29th, we would love to meet you at LASCON.

Talk by James Ryan.

An Advanced Persistent Threat is basically a massively coordinated long term hack attack, often accomplished by nation states or other “very large” organizations, like a business looking for intellectual property and information.  They try to avoid getting caught because they have invested capital in the break in and want to avoid the re-break in.  APTs are often categorized by slow access to data.  They avoid doing things rapidly to avoid detection.

Targets.  There is a question about targets and who is being targeted.  Anything that is crucial infrastructure is targeted.  James Ryan says that we are losing the battle.  We are now fighting (as a nation) nation states with an organized crime type of feel.  We haven’t really found religion to make security happen.  We still treat security as a way to stop rogue 17 year old hackers.

The most prevalent ways to engage in APT is through spear phishing with malware.  The attacker at this point is looking for credentials (key loggers, fake website, …).  Then damage by doing data exfiltration, data tampering, shutdown capabilities.  One other way to avoid getting caught is have the APT get hired in the company.

APT uses zero-day threats and sits on them.  They them it to stay on the network.

We should think that the APT is always going to be on our network and they are going to get there regularly.  We can avoid risk to APT by doing the following.

• Implement PKI on smartcards, enterprise wide (PKI is mathematically proven to be secure for the next 20 years)
• Hardware based PKI, not software
• Implement network authentication and enterprise single sign on eSSO with PKI
• Remote access tied to PKI keycard/smartcard
• Implement Security Event Information Management and correlate accounts and run triggers on multiple simultaneous session trigger.  Also tie this with physical access control.
• Implement PKI with privileged users as well (admins, power users)
• Decrease access per person and evaluate and change
• Create email tagging from external (avoid spear phishing)
• Training and testing using spear phishing in the organization
• Implement USB control to stop external USB
• Background checks and procedures

James Ryan spent time talking about PKI and the necessity of using it.  I agree that we need to have better user management and if you operate on the assumption that Advanced Persistent Threat operators try to go undetected for a long amount of time and also try to get valid user credentials then it is even more so.  The thing that we need to do is control users and access.  This is our biggest vector.

Takeaways:

• APT is real and dangerous
• Assume network is owned already
• Communicate in terms of business continuity
• PKI should be part of the plan
• Use proven methods for executing your strategy

Talk given at TRISC 2010 by Kai Axford from Accretive

Kai has delivered over 300 presentations and he is a manager at an IT solutions company.  Background at Microsoft in security.

Kai starts with talking about noteworthy espionage events:

• Anna Chapman.  The Russian spy that recently got arrested.  Pulls up her facebook and linkedin page.  Later in the talk he goes into the adhoc wireless network she setup to transfer files to other intelligence agents.

• Gary Min (aka Yonggang Min) is a researcher at DuPont.  He accessed over 22,000 abstracts and 16,706 documents from the library at DuPont.  He downloaded 15x more documents than anyone else.  Gary was printing the documents instead of transferring on a USB.  Risk to DuPont was $400,000,000.  He got a $30,000 fine.

• Jerome Kerviel was a trader that worked in compliance before he started abusing the company. Stock trading and was using insider knowledge to abuse trading.

Cyberespionage is a priority in China’s five-year plan.  Acquire Intellectual Property and technology for China.  R&D is at risk from tons of international exposure.  Washington Post released a map of all top secret government agencies and private in the US.  http://projects.washingtonpost.com/top-secret-america/map/

Microsoft and 0-day from SCADA is another example.  SCADA is a very dangerous area for us.  2600 did a recent piece about SCADA.

Lets step back and take a look at the threat.  Insiders.  They are the ones that will take our data and information.  The insider is a greater risk.  We are all worried about the 17-year old kid in Finland, but it is really insiders.

There is a question of, if you gave your employees access to something, are they ‘breaking in’ when they access a bunch of data and take it home with them?

Types of users:

• Elevated users who have been with the company for a long time
• Janitors and cleaning crew
• Insider affiliate (spouse, girlfriend)
• Outside affiliate

Why do people do this?

• Out of work intelligence operators world wide
• Risk and reward is very much out of skew.  Penalties are light.
• Motivators: MICE (Money, Ideology, Coercion, Ego)

In everyone that does espionage, there is a trigger.  There is something that makes it happen.  Carnegie-Mellon did some research stating that everyone who was stealing data had someone else that knew it was happening.

Tools he mentioned and I don’t know where else to mention them:

• Maltego
• USB U3 tool.  Switchblade downloads docs upon plugin.  Hacksaw sets up smtp and stunnel to send out all the docs outbound of the computer.
• Steganography tools >  S-Tools.  This is what Anna was doing by putting info in images.
• Cell phone bridge between laptop and network.
• Tor

Mitigate using:

• defense in depth
• Background checks, Credit Check,
• gates, guards, guns,
• shredding and burning of docs
• clean desk policy
• locks, cameras
• network device blocking
• encryption devices
• Application Security.
• Enterprise Rights Management.
• Data classification.

This talk was presented by Michael Harvey from Avansic.

This has little to do with my day job, but I am big fan of Mac and have really enjoyed using it for the last several years both personally and professionally.  Security tools are also really great for the Apple platform, which I use using Mac Ports: nmap, wireshark, fping, metasploit… Enough about me, on to the talk.

There is a lot of objection about doing forensics on Macs and it is really needed, but in reality it is about 10% of the compute base and a lot of higher level officers in a company are using Macs because they can do whatever they want and aren’t subject to IT restrictions.

Collection of data is the most important.  In a Mac, just pulling the hard drive can be difficult.  Might be useful to pre-download the PDFs on how to do this.  You want to use a firewire write-blocker to copy the drive.  Live CDs (Helix and Raptor LiveCD) lets you copy the data and write block.  Michael really likes Raptor because of its support for legacy Macs and Intel based Macs.  In a follow-up conversation with him he emphasized how Raptor is great for people that don’t do forensics all the time.

Forensics cares about Modified, Accessed, Created time stamps.  Macs add on a time stamp called “Birth Time.”  This is the real created date.  Look at the file properties.  You can use SleuthKit (Open Source forensics tool) to assemble a timeline with M-A-C and Birth Time.

Macs use .plist files in lieu of the windows registry that most people are familiar with. “Property List” files.  These can be ASCII, XML and Binary.  ASCII is pretty rare these days for plist files.  Macs more often dont use the standard epoch unix time and instead uses Jan 1, 2001.  Michael is releasing information on plist format.  Right now there is not a lot of documentation on it.  Plist is more or less equivalent to the windows registry.

Two ways to analyze plist: plutil.pl and Plist Edit Pro.

Dmg files.  Disk images, similar to iso or zip files.  Pretty much a dmg file is crucial for using a Mac.  We can keep an eye out for past-used dmg files to know what has been installed or created…

SQLite Databases.  Lightweight SQL database.  This is heavily used by Firefox, iPhone, or Mac apps.  This is real common on Macs.

Email.  Email forensics will usually come in three flavors: MS Entourage (Outlook), Mail.app, and Mozilla Thunderbird.  A good tool for this is Emailchemy and is forensically sound.  It takes in all the formats.

Useful plist File Examples to look at for more info

• Installed Applications: ~/Library/Preferences/com.apple.finder.plist
• CD/DVD Burning: ~/Library/Preferences/com.apple.DiskUtility.plist
• Recent Accessed Docuents, Servers, and Aplications: ~/Library/Preferences/com.recentitems.plist
• Safari History: ~/Library/Preferences/com.apple.Safari.plist
• Safari Cache: ~/Library/Preferences/com.apple.Safari/cache.db
• Firefox: didn’t get this one

Forensic Software that you can use

• AccessData FTK3
• Mac Forensics Lab
• Sleuth Kit (great timeline)
• Others exist

In conclusion, Mac OSX Investigations are not that scary.  Be prepared with hard drive removal guides and how to extract data off of them.  The best forensic imaging tool out there should be chosen by hardware speed (and firewire), write-blocking capabilities, ability to use dual-core.  You need to know your tools handle HFS+, Birthed times, plist files, dmg files, SQLite Databases.

Audience member asked about harddrive copying tool.  Michael recommends Tableau (sp?).

Here are some resources:

• Apple Examiner – appleexaminer.com
• Mac Forensics Lab Tips – macforensicslab.com
• Access Data – accessdata.com
• Emailchemy – weirdkid.com/products/emailchemy

File Juicer.  Extracts info from databases used by browsers for cache.  Favicons are a good browser history tool…  You can point File Juicer at a SQLite Database or .dmg files.

Also, talking with Michael afterward ended with two book recommendations: SysInternals for Mac OSX and Mac OSX Forensics (unsure of title but it includes a DVD).
All in all, a really interesting talk and I look forward to seeing what else Mike produces in this arena.