【甘道夫】Hadoop2.4.1尝鲜部署+完整版配置文件，

文章由LinuxBoy分享于2019-03-27 04:03:46热评（54）

【甘道夫】Hadoop2.4.1尝鲜部署+完整版配置文件，

引言转眼间，Hadoop的stable版本已经升级到2.4.1了，社区的力量真是强大！3.0啥时候release呢？今天做了个调研，尝鲜了一下2.4.1版本的分布式部署，包括NN HA（目前已经部署好了2.2.0的NN HA，ZK和ZKFC用现成的），顺便也结合官方文档 http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/ClusterSetup.html 梳理、补全了关键的配置文件属性，将同类属性归类，方便以后阅读修改，及作为模板使用。下面记录参照官方文档及过去经验部署2.4.1的过程。欢迎转载，请注明来源：http://blog.csdn.net/u010967382/article/details/37653177

注意 1.本文只记录配置文件，不记录其余部署过程，其余过程和2.2.0相同，参见 http://blog.csdn.net/u010967382/article/details/20380387 http://blog.csdn.net/u010967382/article/details/30976935 2.配置中所有的路径、IP、hostname均需根据实际情况修改。

1.实验环境： 4节点集群，ZK节点3个，hosts文件和各节点角色分配如下： hosts： 192.168.66.91 master 192.168.66.92 slave1 192.168.66.93 slave2 192.168.66.94 slave3
角色分配：

	Active NN	Standby NN	DN	JournalNode	Zookeeper	FailoverController
master	V			V	V	V
slave1		V	V	V	V	V
slave2			V	V	V
slave3			V

2.hadoop-env.sh 修改以下三处即可 # The java implementation to use. export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_07
# The directory where pid files are stored. /tmp by default. # NOTE: this should be set to a directory that can only be written to by the user that will run the hadoop daemons. Otherwise there is the potential for a symlink attack. export HADOOP_PID_DIR=/home/yarn/Hadoop/hadoop-2.4.1/hadoop_pid_dir export HADOOP_SECURE_DN_PID_DIR=/home/yarn/Hadoop/hadoop-2.4.1/hadoop_pid_dir

3.core-site.xml 完整文件
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>   <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://myhadoop</value> <description>NameNode UR，格式是hdfs://host:port/，如果开启了NN HA特性，则配置集群的逻辑名，具体参见我的博客http://blog.csdn.net/u010967382/article/details/30976935 </description> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/tmp</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> <description>Size of read/write buffer used in SequenceFiles. </description> </property> <property> <name>ha.zookeeper.quorum</name> <value>master:2181,slave1:2181,slave2:2181</value> <description>注意，配置了ZK以后，在格式化、启动NameNode之前必须先启动ZK，否则会报连接错误 </description> </property> </configuration>

4.hdfs-site.xml 完整文件
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>   <configuration>  <property> <name>dfs.nameservices</name> <value>myhadoop</value> <description> Comma-separated list of nameservices. as same as fs.defaultFS in core-site.xml. </description> </property> <property> <name>dfs.ha.namenodes.myhadoop</name> <value>nn1,nn2</value> <description> The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). </description> </property> <property> <name>dfs.namenode.rpc-address.myhadoop.nn1</name> <value>master:8020</value> <description> RPC address for nomenode1 of hadoop-test </description> </property> <property> <name>dfs.namenode.rpc-address.myhadoop.nn2</name> <value>slave1:8020</value> <description> RPC address for nomenode2 of hadoop-test </description> </property> <property> <name>dfs.namenode.http-address.myhadoop.nn1</name> <value>master:50070</value> <description> The address and the base port where the dfs namenode1 web ui will listen on. </description> </property> <property> <name>dfs.namenode.http-address.myhadoop.nn2</name> <value>slave1:50070</value> <description> The address and the base port where the dfs namenode2 web ui will listen on. </description> </property> <property> <name>dfs.namenode.servicerpc-address.myhadoop.n1</name> <value>master:53310</value> </property> <property> <name>dfs.namenode.servicerpc-address.myhadoop.n2</name> <value>slave1:53310</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> <description> Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. </description> </property> <property> <name>dfs.client.failover.proxy.provider.myhadoop</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider </value> <description>Configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the current Active, and therefore which NameNode is currently serving client requests. 这个类是Client的访问代理，是HA特性对于Client透明的关键！ </description> </property> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> <description>how to communicate in the switch process</description> </property> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/yarn/.ssh/id_rsa</value> <description>the location stored ssh key</description> </property> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>1000</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/journal/</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://master:8485;slave1:8485;slave2:8485/hadoop-journal </value> <description>A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. </description> </property>   <property> <name>dfs.namenode.name.dir</name> <value>file:///home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/name</value> <description>Path on the local filesystem where the NameNode stores the namespace and transactions logs persistently.If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy.</description> </property> <property> <name>dfs.blocksize</name> <value>1048576</value> <description> HDFS blocksize of 128MB for large file-systems. Minimum block size is 1048576. </description> </property> <property> <name>dfs.namenode.handler.count</name> <value>10</value> <description>More NameNode server threads to handle RPCs from large number of DataNodes.</description> </property>    <property> <name>dfs.datanode.data.dir</name> <value>file:///home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/data</value> <description>Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks.If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices.</description> </property>  </configuration>

5.yarn-site.xml
<?xml version="1.0"?>  <configuration>  <property> <name>yarn.acl.enable</name> <value>false</value> <description>Enable ACLs? Defaults to false.</description> </property> <property> <name>yarn.admin.acl</name> <value>*</value> <description> ACL to set admins on the cluster. ACLs are of for comma-separated-usersspace comma-separated-groups. Defaults to special value of * which means anyone. Special value of just space means no one has access. </description> </property> <property> <name>yarn.log-aggregation-enable</name> <value>false</value> <description>Configuration to enable or disable log aggregation</description> </property>   <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> <description>The hostname of the RM.</description> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname}:8090</value> <description>The https adddress of the RM web application.</description> </property> <property> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> <description>ResourceManager host:port for clients to submit jobs.</description> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> <description>ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources.</description> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> <description>ResourceManager host:port for NodeManagers.</description> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> <description>ResourceManager host:port for administrative commands.</description> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> <description>ResourceManager web-ui host:port.</description> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> <description> ResourceManager Scheduler class. CapacityScheduler (recommended), FairScheduler (also recommended), or FifoScheduler </description> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>512</value> <description> Minimum limit of memory to allocate to each container request at the Resource Manager. In MBs </description> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>2048</value> <description> Maximum limit of memory to allocate to each container request at the Resource Manager. In MBs. According to my configuration,yarn.scheduler.maximum-allocation-mb > yarn.nodemanager.resource.memory-mb </description> </property>    <property> <name>yarn.nodemanager.resource.memory-mb</name> <value>1024</value> <description> Resource i.e. available physical memory, in MB, for given NodeManager. Defines total available resources on the NodeManager to be made available to running containers. </description> </property> <property> <name>yarn.nodemanager.vmem-pmem-ratio</name> <value>2.1</value> <description> Ratio between virtual memory to physical memory when setting memory limits for containers. Container allocations are expressed in terms of physical memory, and virtual memory usage is allowed to exceed this allocation by this ratio. </description> </property> <property> <name>yarn.nodemanager.local-dirs</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/yarn_dir/local</value> <description> Comma-separated list of paths on the local filesystem where intermediate data is written. Multiple paths help spread disk i/o. </description> </property> <property> <name>yarn.nodemanager.log-dirs</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/yarn_dir/log</value> <description> Comma-separated list of paths on the local filesystem where logs are written. Multiple paths help spread disk i/o. </description> </property> <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>10800</value> <description> Default time (in seconds) to retain log files on the NodeManager. ***Only applicable if log-aggregation is disabled. </description> </property> <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/yarn/log-aggregation</value> <description> HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. ***Only applicable if log-aggregation is enabled. </description> </property> <property> <name>yarn.nodemanager.remote-app-log-dir-suffix</name> <value>logs</value> <description> Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam}. ***Only applicable if log-aggregation is enabled. </description> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>Shuffle service that needs to be set for Map Reduce applications.</description> </property> <property> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>1</value> <description>Number of CPU cores that can be allocated for containers.</description> </property>   <property> <name>yarn.log-aggregation.retain-seconds</name> <value>-1</value> <description> How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node. </description> </property> <property> <name>yarn.log-aggregation.retain-check-interval-seconds</name> <value>-1</value> <description> Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node. </description> </property>  <property> <name>yarn.scheduler.fair.allocation.file</name> <value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value> <description>fairscheduler config file path</description>  </property> </configuration>

6.创建fairscheduler.xml
<?xml version="1.0"?> <allocations>  <user name="yarn"> <maxRunningApps>30</maxRunningApps> </user> </allocations>

7.mapred-site.xml
<?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>   <configuration>  <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>Execution framework set to Hadoop YARN.</description> </property> <property> <name>mapreduce.map.memory.mb</name> <value>1024</value> <description>Larger resource limit for maps.</description> </property> <property> <name>mapreduce.map.java.opts</name> <value>-Xmx1024M</value> <description>Larger heap-size for child jvms of maps.</description> </property> <property> <name>mapreduce.reduce.memory.mb</name> <value>1024</value> <description>Larger resource limit for reduces.</description> </property> <property> <name>mapreduce.reduce.java.opts</name> <value>-Xmx1024M</value> <description>Larger heap-size for child jvms of reduces.</description> </property> <property> <name>mapreduce.task.io.sort.mb</name> <value>1024</value> <description>Higher memory-limit while sorting data for efficiency.</description> </property> <property> <name>mapreduce.task.io.sort.factor</name> <value>10</value> <description>More streams merged at once while sorting files.</description> </property> <property> <name>mapreduce.reduce.shuffle.parallelcopies</name> <value>20</value> <description>Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.</description> </property> 
 <property> <name>mapreduce.jobhistory.address</name> <value>slave1:10020</value> <description>MapReduce JobHistory Server host:port. Default port is 10020.</description> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>slave1:19888</value> <description>MapReduce JobHistory Server Web UI host:port. Default port is 19888.</description> </property> <property> <name>mapreduce.jobhistory.intermediate-done-dir</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/mr_history/tmp</value> <description>Directory where history files are written by MapReduce jobs.</description> </property> <property> <name>mapreduce.jobhistory.done-dir</name> <value>/home/yarn/Hadoop/hadoop-2.4.1/mr_history/done</value> <description>Directory where history files are managed by the MR JobHistory Server.</description> </property>  </configuration>

8.slaves
slave1 slave2 slave3

配置文件就涉及以上这些，在一个节点修改好，然后：

9.启动集群
（1）启动ZK 在所有的ZK节点执行命令： zkServer.sh start
查看各个ZK的从属关系： yarn@master:~$ zkServer.sh status JMX enabled by default Using config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower
yarn@slave1:~$ zkServer.sh status JMX enabled by default Using config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: follower
yarn@slave2:~$ zkServer.sh status JMX enabled by default Using config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: leader
注意： 哪个ZK节点会成为leader是随机的，第一次实验时slave2成为了leader，第二次实验时slave1成为了leader！
此时，在各个节点都可以查看到ZK进程： yarn@master:~$ jps 3084 QuorumPeerMain 3212 Jps
（2）格式化ZK（仅第一次需要做） 任意ZK节点上执行： hdfs zkfc -formatZK
（3）启动ZKFC ZookeeperFailoverController是用来监控NN状态，协助实现主备NN切换的，所以仅仅在主备NN节点上启动就行： hadoop-daemon.sh start zkfc
启动后我们可以看到ZKFC进程： yarn@master:~$ jps 3084 QuorumPeerMain 3292 Jps 3247 DFSZKFailoverController
（4）启动用于主备NN之间同步元数据信息的共享存储系统JournalNode 参见角色分配表，在各个JN节点上启动： hadoop-daemon.sh start journalnode
启动后在各个JN节点都可以看到JournalNode进程： yarn@master:~$ jps 3084 QuorumPeerMain 3358 Jps 3325 JournalNode 3247 DFSZKFailoverController
（5）格式化并启动主NN 格式化： hdfs namenode -format
注意：只有第一次启动系统时需格式化，请勿重复格式化！
在主NN节点执行命令启动NN: hadoop-daemon.sh start namenode
启动后可以看到NN进程： yarn@master:~$ jps 3084 QuorumPeerMain 3480 Jps 3325 JournalNode 3411 NameNode 3247 DFSZKFailoverController

（6）在备NN上同步主NN的元数据信息 hdfs namenode -bootstrapStandby
以下是正常执行时的最后部分日志： Re-format filesystem in Storage Directory /home/yarn/Hadoop/hdfs2.0/name ? (Y or N) Y 14/06/15 10:09:08 INFO common.Storage: Storage directory /home/yarn/Hadoop/hdfs2.0/name has been successfully formatted. 14/06/15 10:09:09 INFO namenode.TransferFsImage: Opening connection to http://master:50070/getimage?getimage=1&txid=935&storageInfo=-47:564636372:0:CID-d899b10e-10c9-4851-b60d-3e158e322a62 14/06/15 10:09:09 INFO namenode.TransferFsImage: Transfer took 0.11s at 63.64 KB/s 14/06/15 10:09:09 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000935 size 7545 bytes. 14/06/15 10:09:09 INFO util.ExitUtil: Exiting with status 0 14/06/15 10:09:09 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at slave1/192.168.66.92 ************************************************************/
（7）启动备NN 在备NN上执行命令： hadoop-daemon.sh start namenode
（8）设置主NN（这一步可以省略，这是在设置手动切换NN时的步骤，ZK已经自动选择一个节点作为主NN了） 到目前为止，其实HDFS还不知道谁是主NN，可以通过监控页面查看，两个节点的NN都是Standby状态。下面我们需要在主NN节点上执行命令激活主NN： hdfs haadmin -transitionToActive nn1
（9）在主NN上启动Datanode 在[nn1]上，启动所有datanode hadoop-daemons.sh start datanode
（10）启动yarn
在ResourceManager所在节点执行（yarn启动真是方便！！！）： start-yarn.sh
（11）在运行MRJS的slave1上执行以下命令启动MR JobHistory Server： mr-jobhistory-daemon.sh start historyserver
至此，启动完毕，可以看到2.4.1的界面了：

10.停止集群 在RM和NN所在节点master执行： 停止yarn： stop-yarn.sh
停止hdfs： stop-dfs.sh
停止zookeeper： zkServer.sh stop

在运行JobHistoryServer的slave1上执行： 停止JobHistoryServer： mr-jobhistory-daemon.sh stop historyserver

推荐文章：

【甘道夫】Hadoop2.4.1尝鲜部署+完整版配置文件，