ELKstack日志分析平台

ELKstack日志分析平台

ELKstack是Elasticsearch、Logstash、Kibana三个开源软件的组合。目前都在Elastic.co公司名下。
ELK是一套常用的开源日志监控和分析系统，包括一个分布式索引与搜索服务Elasticsearch，一个管理日志和事件的工具logstash，和一个数据可视化服务Kibana
logstash_1.5.3                负责日志的收集，处理和储存
elasticsearch-1.7.2            负责日志检索和分析
kibana-4.1.2-linux-x64.tar.gz  负责日志的可视化
jdk-1.7.0_03                  java环境
redis-2.4.14                  DB
 
一、基础环境
1、角色、ip、版本、内核
serverA 10.1.10.185 3.2.0-4-amd64 7.8 java elasticsearch redis kibana logstash(agent indexer)
clientB 10.1.10.117 3.2.0-4-amd64 7.8 java logstash(agent)
 
2、安装基础包
apt-get -y install curl wget lrzsz axel
 
二、安装redis server
1、安装包
apt-get -y install redis-server
 
2、创建redis存储目录
mkdir /opt/redis -p
 
3、权限
chown redis /opt/redis/ -R
 
4、配置
1)备份配置
cp /etc/redis/redis.conf /etc/redis/redis.conf.bak
2)修改配置
sed -i 's!^bind.*!bind 10.1.10.185!g' /etc/redis/redis.conf
sed -i 's!^dir.*!dir /opt/redis!g' /etc/redis/redis.conf
 
5、重启服务
/etc/init.d/redis-server restart
 
6、查看进程和端口
1)查看进程
ps -ef |grep redis
redis    23193      1  0 16:41 ?        00:00:00 /usr/bin/redis-server /etc/redis/redis.conf
2)查看端口
netstat -tupnl |grep redis
tcp        0      0 10.1.10.185:6379        0.0.0.0:*              LISTEN      25188/redis-server
 
7、检查开机启动(默认设置开机启动了)
ll /etc/rc2.d/ |grep redis
lrwxrwxrwx 1 root root  22 Sep 20 16:41 S02redis-server -> ../init.d/redis-server
 
三、安装java环境
1、安装包
apt-get -y install openjdk-7-jdk
 
2、查看版本
java -version
java version "1.7.0_03"
OpenJDK Runtime Environment (IcedTea7 2.1.7) (7u3-2.1.7-1)
OpenJDK 64-Bit Server VM (build 22.0-b10, mixed mode)
 
四、安装elasticsearch
1、下载elasticsearch
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb
 
2、安装elasticsearch
dpkg -i elasticsearch-1.7.2.deb
Selecting previously unselected package elasticsearch.
(Reading database ... 30240 files and directories currently installed.)
Unpacking elasticsearch (from elasticsearch-1.7.2.deb) ...
Creating elasticsearch group... OK
Creating elasticsearch user... OK
Setting up elasticsearch (1.7.2) ...
 
3、配置
1)备份配置
cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
2)修改配置
echo "network.bind_host: 10.1.10.185" >> /etc/elasticsearch/elasticsearch.yml
 
4、启动elasticsearch服务
/etc/init.d/elasticsearch start
 
5、查看进程和端口
1)查看进程
ps -ef |grep java
106      22835      1 63 15:14 ?        00:00:03 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.7.2.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
2)查看端口
netstat -tupnl |grep java
tcp6      0      0 10.1.10.185:9200        :::*                    LISTEN      22835/java     
tcp6      0      0 10.1.10.185:9300        :::*                    LISTEN      22835/java     
udp6      0      0 :::54328                :::*                                22835/java   
 
6、测试
curl -X GET http://10.1.10.185:9200
{
  "status" : 200,
  "name" : "Ned Leeds",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.2",
    "build_hash" : "e43676b1385b8125d647f593f7202acbd816e8ec",
    "build_timestamp" : "2015-09-14T09:49:53Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}
 
7、添加到开机启动
update-rc.d elasticsearch defaults
update-rc.d: using dependency based boot sequencing
 
五、安装logstash
1、下载logstash
wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_1.5.3-1_all.deb
 
2、安装logstash
dpkg -i logstash_1.5.3-1_all.deb
(Reading database ... 30338 files and directories currently installed.)
Unpacking logstash (from logstash_1.5.3-1_all.deb) ...
Setting up logstash (1:1.5.3-1) ...
 
3、配置(默认没有这个配置文件)
1)配置logstash_agent
cat /etc/logstash/conf.d/logstash_agent.conf
input {
        file {
                type => "messages"
                path => ["/var/log/messages"]
        }
        file {
                type => "elasticsearch"
                path => ['/var/log/elasticsearch/elasticsearch.log*']
        }
}
output {
        redis {
                host => "10.1.10.185"
                data_type => "list"
                key => "logstash:redis"
        }
}
2)配置logstash_indexer
cat /etc/logstash/conf.d/logstash_indexer.conf
input {
        redis {
                host => "10.1.10.185"
                data_type => "list"
                key => "logstash:redis"
                type => "redis-input"
        port => "6379"
        }
}
output {
        elasticsearch {
                host => "10.1.10.185"
        }
}
 
4、启动服务
/etc/init.d/logstash start
logstash started.
 
5、使用jps -mlv或ps -ef来查看下进程
ps -ef|grep logst
logstash  22932      1 16 15:19 pts/0    00:00:01 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log
 
6、设置开机启动
update-rc.d logstash defaults
update-rc.d: using dependency based boot sequencing
 
六、安装kibana(前端web)
1、下载
axel -n 10 https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
 
2、解压到指定目录
tar zxvf kibana-4.1.2-linux-x64.tar.gz -C /opt
 
3、创建日志目录
mkdir -p /opt/kibanalog
 
4、配置
1)备份配置
cp /opt/kibana-4.1.2-linux-x64/config/kibana.yml /opt/kibana-4.1.2-linux-x64/config/kibana.yml.bak
2)修改配置
sed -i 's!^elasticsearch_url: .*!elasticsearch_url: "http://10.1.10.185:9200"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
sed -i 's!^host: .*!host: "10.1.10.185"!g' /opt/kibana-4.1.2-linux-x64/config/kibana.yml
 
5、启动服务
cd /opt/kibanalog && nohup /opt/kibana-4.1.2-linux-x64/bin/kibana &
 
6、查看进程和端口
1)查看进程
ps aux |grep kibana
root      22982  5.4 20.1 612576 47716 pts/0    Sl  15:22  0:01 /opt/kibana-4.1.2-linux-x64/bin/../node/bin/node /opt/kibana-4.1.2-linux-x64/bin/../src/bin/kibana.js
2)查看端口
netstat -tupnl|grep 5601
tcp        0      0 10.1.10.185:5601        0.0.0.0:*              LISTEN      22982/node 
 
7、在windows上访问http://10.1.10.185:5601

更多详情见请继续阅读下一页的精彩内容：