Why ha.ckers.org Doesn’t Get Hacked
The first LASCON session I went to was Why ha.ckers.org Doesn’t Get Hacked by James Flom (who with rsnake is ha.ckers.org). By its nature, it gets like 500-1000 hack attempts a week, but they’ve kept it secure for six years now.
From the network perspective, they use dual firewalls running the openBSD open source pf, which does Cisco-style traffic inspection. Systems inside have no egress, and they have the user traffic and admin traffic segmented to different firewalls sets and switches.
On the systems, they use chroot jails mounted read only. Old school! Jails are virtualization on the cheap, and if combined with a read only filesystem, give you a single out of band point of update, and you can do upgrades with minimal downtime. They monitor them from the parent host.
Rsnake has done a whole separate presentation on how he’s secured his browser – the biggest attack vector is often “compromise the browser of an admin” and not direct attack on the asset.
They went to WordPress for their software – how to secure that? Obviously code security’s a nightmare there. So they set up a defense in depth scheme where they check the source ip, cert, and user/pass auth at the firewall, then to admin proxy check source IP, path, htaccess user/pass, and finally do the app auth.
Other stuff they do:
- Secure logging to OSSEC – pflogd, waf logs, os logs, apache logs, parent logs, it goes off host so it’s reasonably tamper-proof
- On-host WAF – custom, more of a “Web IDS” really, which feeds back “naughty people” to the firewall for blocking
- For Apache – have your content owned by a different user, in their case there’s not even a user in the jail that can write to the files.
- Use file ACLs, too.
Use case – they found an Apache flaw, reported it, and as is too often the case, Apache didn’t care. So they modded their pf to detect the premise of the attack and block it (not just the specific attack). (Heard of slowloris?)
Their ISP has been an issue – as they’ve moved their ISPs have shut them down out of cluelessness sometimes (Time Warner Business Class FTL).
They are moving to relayd for load balancing and SSL. The PCI rule about “stay encrypted all the way to the box” is dumb, because it would prevent them from doing useful security inspection at that layer.
A good talk, though sadly a lot of the direct takeaways would mean “go to FreeBSD,” which I would rather not do. But a lot of the concepts can port to other OSes and pure virtualization/cloud scenarios. And note how joining network security, OS security, and appsec gets you way more leverage than having “separate layers” where each layer only worries about itself.n
And may I just say that I love how Apache can be run “read only” – sadly, most software, even other open source software like Tomcat, can’t be. It all wants to write down into its config and its running directories itself, and it’s a horrible design practice and security risk. If you’re writing software, remember that if it’s compromised and it can write to its own exes/config/etc. you’re owned. Make your software run on a read only FS (with read/write in /tmp for stuff acceptable). It’s the right thing to do.
the agile admin 