JStrom,jstormgithub
JStrom,jstormgithub
1. JStrom说明
JStorm 是一个类似Hadoop MapReduce的系统, 用户按照指定的接口实现一个任务,然后将这个任务递交给JStorm系统,Jstorm将这个任务跑起来,并且按7 * 24小时运行起来,一旦中间一个worker 发生意外故障, 调度器立即分配一个新的worker替换这个失效的worker。
本文只简单说明如何安装JStorm,参考文档:点击打开链接2. 安装环境准备
- Python 2.6 以上
- JDK 1.5 以上
3. 安装ZooKeeper
3.1 下载ZooKeeper
从Apache网站上下载ZooKeeper,目前的最新版本为:3.4.6,下载地址:http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz# mkdir /work/jstorm # cd /work/jstorm # wget http://mirrors.cnnic.cn/apache/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz # sudo tar -zxvf zookeeper-3.4.6.tar.gz # chown -R root:root zookeeper-3.4.6第一行是创建一个jstorm根目录,最后一行改变zookeeper-3.4.6文件夹的权限,使之可写。
3.2 设置环境变量
用vi打开~/.bash_profile,将下面代码加入到文件最后,保存退出。#java export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_05.jdk/Contents/Home" #zookeeper ZOOKEEPER_HOME=/work/jstorm/zookeeper-3.4.6 export PATH="$PATH:$ZOOKEEPER_HOME" export ZOOKEEPER_HOME如果JAVA_HOME已经设置,则不用再设置,注意JAVA_HOME的路径。
敲入下面代码,让更改立即生效。
# source ~/.bash_profile
3.3 修改配置
# cd $ZOOKEEPER_HOME/conf/ # cp zoo_sample.cfg zoo.cfg # vi zoo.cfg修改zoo.cfg配置:
# The number of milliseconds of each tick tickTime=2000 # The number of ticks that the initial # synchronization phase can take initLimit=10 # The number of ticks that can pass between # sending a request and getting an acknowledgement syncLimit=5 # the directory where the snapshot is stored. # do not use /tmp for storage, /tmp here is just # example sakes. dataDir=/work/jstorm/zookeeper-3.4.6/data # the port at which the clients will connect clientPort=2181 # the maximum number of client connections. # increase this if you need to handle more clients #maxClientCnxns=60 # # Be sure to read the maintenance section of the # administrator guide before turning on autopurge. # # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance # # The number of snapshots to retain in dataDir #autopurge.snapRetainCount=3 # Purge task interval in hours # Set to "0" to disable auto purge feature #autopurge.purgeInterval=1注意dataDir的改变,可以改成其他目录。
3.4 启动和停止
# $ZOOKEEPER_HOME/bin/zkServer.sh start # $ZOOKEEPER_HOME/bin/zkServer.sh stop查看是否启动:
netstat -nat|grep 2181
4. 安装JStorm
4.1 下载
下载地址:https://github.com/alibaba/jstorm/wiki/Downloads# cd /work/jstorm # wget http://42.121.19.155/jstorm/jstorm-0.9.6.2.zip # unzip jstorm-0.9.6.2.zip # vi ~/.bash_profile加入如下代码:
#jstorm export JSTORM_HOME=/work/jstorm/jstorm-0.9.6.2 export PATH=$PATH:$JSTORM_HOME/bin
4.2 配置
在此处如果用默认配置,会出现JVM错误,修改$JSTORM_HOME/conf/storm.yaml,在文件开始,加入如下代码:nimbus.childopts: "-Xms2g -Xmx2g -Xmn768m -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=15 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:CMSMaxAbortablePrecleanTime=5000"
4.3 运行
# nohup jstorm nimbus & # nohup jstorm supervisor &相应的日志在$JSTORM_HOME/logs/nimbus.log和$JSTORM_HOME/logs/supervisor.log中,可查看运行是否正常。
5. 安装JStorm WebUI
5.1 下载Tomcat
下载地址:http://mirrors.cnnic.cn/apache/tomcat/tomcat-8/v8.0.15/bin/apache-tomcat-8.0.15.tar.gz5.2 安装Tomcat
# tar -xzf apache-tomcat-8.0.15.tar.gz # cd apache-tomcat-8.0.15 # cd webapps # cp $JSTORM_HOME/jstorm-ui-0.9.6.2.war ./ # mv ROOT ROOT.old # ln -s jstorm-ui-0.9.6.2 ROOT # cd ../bin # ./startup.sh在浏览器内输入http://localhost:8080,即可查看JStorm的管理界面.
评论暂时关闭