Flume(1.4)+Solr(4.3) Log analysis


Flume: event collector, typical usage is log collection. Solr:  search engine based on Lucene
Function: watch  /var/log/a1.new.log file. If new lines append to this file, it will send the event( new lines) to flume source and make index of the event , then send to solr engine.  You can quickly search the new event by solr.
Download: flume 1.4: http://archive.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-bin.tar.gz
solr 4.3:  http://archive.apache.org/dist/lucene/solr/4.3.0/


a1.new.log's format like following: # cat  /var/log/a1.new.log
2014-05-29 10:37:56,777 INFO org.apache.hadoop.http.HttpServer: HttpServer.start() threw a non Bind IOException
2014-05-15 19:06:52,373 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1


Configure solr  (server-1941/192.168.100.110)
  • configure solr cloud
  • start solr:  java -DzkRun  -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=myconf -jar start.jar
  • brownse :http://192.168.100.110:8983/solr/#/ 

    • Configure flume (server-1941/192.168.100.110)
    1. extract apache-flume-1.4.0-bin.tar.gz  2.  edit flume-env.sh
  • How to test
  •  curl -g http://192.168.100.110:8983/solr/collection1/select?q=msg:*hadoop*&wt=xml&indent=true 
    •  <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"><int name="status">0</int><int name="QTime">5</int><lst name="params"><str name="q">msg:*hadoop*</str></lst></lst><result name="response" numFound="1" start="0"><doc><str name="id">63566aed-7438-4c8e-8b02-7f6fa0be85b3</str><str name="timestamp">2014-06-03T17:16:52.373Z</str><str name="msg">hadoop will shutdown</str><long name="_version_">1469853891281551360</long></doc></result> </response>




    相关内容