We’re researching all kinds of tools as we set up our new cloud environment, I figure I may as well share for the benefit of the public…
Most recently, we’re looking at log management. That is, a tool to aggregate and analyze your log files from across your infrastructure. We love Splunk and it’s been our tool of choice in the past, but it has two major drawbacks. One, it’s quite expensive. In our new environment where we’re using a lot of open source and other new-format vendors, Splunk is a comparatively big line item for a comparatively small part of an overall systems management portfolio.
Two, which is somewhat related, it’s licensed by amount of logs it processes per day. Which is a problem because when something goes wrong in our systems, it tends to cause logging levels to spike up. In our old environment, we keep having to play this game where an app will get rolled to production with debug on (accidentally or deliberately) or just be logging too much or be having a problem causing it to log too much, and then we have to blacklist it in Splunk so it doesn’t run us over our license and cause the whole damn installation to shut off. It took an annoying amount of micromanagement for this reason.
Other than that, Splunk is the gold standard; it pulls anything in, graphs it, has Google-like search, dashboards, reports, alerts, and even crazier capabilities.
Now on the “low end” there are really simple log watchers like swatch or logwatch. But we’d really like something that will aggregate ALL our logs (not just syslog stuff using syslog-ng – app server logs, application logs, etc.), ideally from both UNIX and Windows systems, and make them usefully searchable. Trying to make everything and everyone log using syslog is an ever receding goal. It’s a fool’s paradise.
There’s the big appliance vendors on the “high end” like LogLogic and LogRhythm, but we looked at them when we looked at Splunk and they are not only expensive but also seem to be “write only solutions” – they aggregate your logs to meet compliance requirements, and do some limited pattern matching, but they don’t put your logs to work to help you in your actual work of application administration the dozen ways Splunk does. At best they are “SIEM”s – security information and event managers – that alert on naughty intruders. But with Splunk I can do everything from generate a report of 404s to send to our designers to fix their bad links/missing images to graph site traffic to make dashboards for specific applications for their developers to review. Plus, as we’re doing this in the cloud, appliances need not apply. (Ooo, that’s a catchy phrase, I’ll have to use that for a separate post!)
I came across three other tools that seem promising:
- Logscape from Liquidlabs – does graphing and dashboards like Splunk does. And “live tail” – Splunk mysteriously took this out when they revved from version 3 to 4! Internet rumor is that it’s a lot cheaper. Seems like a smaller, less expensive Splunk, which is a nice thing to be, all considered.
- Octopussy – open source and Perl based (might work on Windows but I wouldn’t put money on it). Does alerting and reporting. Much more basic, but you can’t beat the price. Don’t think it’ll meet our needs though.
- Xpolog – seems nice and kinda like Splunk. Most of the info I can find on it, though, are “What about xpolog, is good!” comments appended to every forum thread/blog post about Splunk I can find, which is usually a warning sign – that kind of guerrilla marketing gets old quick IMO. One article mentions looking into it and finding it more expensive but with some nice features like autodiscovery, but not as open as Splunk.
Anyone have anything to add? Used any of these? We’ve gotten kind of addicted to having our logs be immediately accessible, converted into metrics, etc. I probably wouldn’t even begrudge Splunk the money if it weren’t for all the micromanagement you have to put into running it. It’s like telling the fire department “you’re licensed for a maximum of three fires at a time” – it verges on irresponsible.
the agile admin 