log4j Log Indexing using Solr

We are finding it very hard to monitor the logs spread over a cluster of four managed servers. So, I am trying to build a simple log4j appender whi开发者_运维技巧ch uses solrj api to store the logs in the solr server. The idea is to use leverage REST of solr to build a better GUI which could help us

1. search the logs and the display the previous and the next 50 lines or so and
2. tail the logs

Being awful on front ends, I am trying to cookup something with GWT (a prototype version). I am planning to host the project on googlecode under ASL.

Greatly appreciate if you could throw some insights on

1. Whether it makes sense to create a project like this ?
2. Is using Solr for this an overkill?
3. Any suggestions on web framework/tool which will help me build a tab-based front end for tailing.

You can use a combination of logstash (for shipping and filtering logs) + elasticsearch (for indexing and storage) + kibana (for a pretty GUI).

The loggly folks have also built logstash, which can be backed by quite a few things, including lucene via elastic search. It can forward to graylog also.

Totally doable thing. Many folks have done the roll your own. A couple of useful links.. there is an online service, www.loggly.com that does this. They are actually based on Solr as the core storage engine! Obviously they have built a proprietary interface.

Another option is http://www.graylog2.org/. It is opensource. Not backed by Solr, but still very cool!