centralized logging with logstash

Last week I migrated some services off servers in a rack that was being decommissioned.  There was a distinct lack of system documentation, either up to date or otherwise, so I thought configuring nagios, monit, and munin would be a good start to checking the performance and reliability of the service on the new instance.

It had been suggested that the development team have a log monitoring system based around the log4j library, however it turns out that this system is mostly reactive, and there were a few undocumented configurations that didn't make the migration, and caused some problems. So after some hasty "diff -r" and rsync everything seemed to be well.

But I decided that this weekend I am putting together a central logging management system using logstash and chef, that I can deploy zero config style to hosts via a cookbook recipe, to catch 404 and 503 errors and other alerts coming from remote systems in some timely manner.