Apache Hadoop 2.2.0 HDFS HA + YARN多机部署

文章由LinuxBoy分享于2019-03-27 03:03:37热评（426）

Apache Hadoop 2.2.0 HDFS HA + YARN多机部署

部署逻辑架构：

HDFS HA部署物理架构

注意： JournalNode使用资源很少，即使在实际的生产环境中，也是把JournalNode和DataNode部署在同一台机器上；生产环境中，建议主备NameNode各单独一台机器。

YARN部署架构：

个人实验环境部署图：

ubuntu12 32bit apache hadoop 2.2.0 jdk1.7

准备工作： 1.在4台机器都配置hosts； 2.配置NameNode节点可以免密码登录到其余所有节点，只需要单向免密登录即可，无需双向；免密码登录仅仅在启动、停止集群时使用。 3.安装jdk 4.创建专门的账号，不要用root账号部署、管理hadoop

部署hadoop：第一步：把hadoop安装包解压到每一个节点（可以解压到一个节点，然后完成后续第2步的配置后，再scp拷贝到其余节点）的固定目录下（各节点目录统一），比如/home/yarn/Hadoop/hadoop-2.2.0 第二步：修改配置文件（只需在一个节点上配置，配置好后再用scp分发到其余节点）配置文件路径：etc/hadoop/ hadoop-env.sh 修改JDK路径，在文件中搜索以下行，将JAVA_HOME设置为JDK安装路径即可： # The java implementation to use. export JAVA_HOME=/usr/lib/jvm/java-6-sun

core-site.xml 指定Active NameNode的host名/ip和端口号，端口号可以根据自己的需要修改： <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://SY-0217:8020</value> </property> </configuration> 注意：以上配置的SY-0217是固定host，只适用于手动切换主备NameNode的场景，如果需要通过ZooKeeper来自动切换，则需要配置逻辑名称，后面会详述。

mapred-site.xml
<configuration>  <property> <name>mapreduce.framework.name</name> <value>yarn</value> <description>The runtime framework for executing MapReduce jobs. Can be one of local, classic or yarn. </description> </property>
 <property> <name>mapreduce.jobhistory.address</name> <value>SY-0355:10020</value> <description>MapReduce JobHistory Server IPC host:port</description> </property>
<property> <name>mapreduce.jobhistory.webapp.address</name> <value>SY-0355:19888</value> <description>MapReduce JobHistory Server Web UI host:port</description> </property> </configuration>

hdfs-site.xml 非常关键的配置文件！ <configuration>
<property> <name>dfs.nameservices</name> <value>hadoop-test</value> <description> 指定命名空间名称，可随意起名 Comma-separated list of nameservices. </description> </property>
<property> <name>dfs.ha.namenodes.hadoop-test</name> <value>nn1,nn2</value> <description> 在命名空间下指定NameNode逻辑名 The prefix for a given nameservice, contains a comma-separated list of namenodes for a given nameservice (eg EXAMPLENAMESERVICE). </description> </property>
<property> <name>dfs.namenode.rpc-address.hadoop-test.nn1</name> <value>SY-0217:8020</value> <description> 为“命名空间名.NameNode逻辑名”配置rpc地址 RPC address for nomenode1 of hadoop-test </description> </property>
<property> <name>dfs.namenode.rpc-address.hadoop-test.nn2</name> <value>SY-0355:8020</value> <description>   为“命名空间名.NameNode逻辑名”配置rpc地址 RPC address for nomenode2 of hadoop-test </description> </property>
<property> <name>dfs.namenode.http-address.hadoop-test.nn1</name> <value>SY-0217:50070</value> <description>   为“命名空间名.NameNode逻辑名”配置http地址 The address and the base port where the dfs namenode1 web ui will listen on. </description> </property>
<property> <name>dfs.namenode.http-address.hadoop-test.nn2</name> <value>SY-0355:50070</value> <description>   为“命名空间名.NameNode逻辑名”配置http地址 The address and the base port where the dfs namenode2 web ui will listen on. </description> </property>
<property> <name>dfs.namenode.name.dir</name> <value>file:///home/dongxicheng/hadoop/hdfs/name</value> <description> 配置NameNode元数据存放的路径；如果机器上有多块硬盘的话，推荐配置多个路径，用逗号分隔。 Determines where on the local filesystem the DFS name node should store the name table(fsimage). If this is a comma-delimited list of directories then the name table is replicated in all of the directories, for redundancy. </description> </property>
<property> <name>dfs.datanode.data.dir</name> <value>file:///home/dongxicheng/hadoop/hdfs/data</value> <description>     配置DataNode数据存放的路径；如果机器上有多块硬盘的话，推荐配置多个路径，用逗号分隔。 Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored. </description> </property>
<property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://SY-0355:8485;SY-0225:8485;SY-0226:8485/hadoop-journal</value> <description> 配置JournalNode，包含三部分：（1）qjournal是协议，无需修改；（2）然后就是三台部署JournalNode的主机host/ip：端口，三台机器之间用分号分隔；（3）最后的hadoop-journal是journalnode的命名空间，可以随意取名。 A directory on shared storage between the multiple namenodes in an HA cluster. This directory will be written by the active and read by the standby in order to keep the namespaces synchronized. This directory does not need to be listed in dfs.namenode.edits.dir above. It should be left empty in a non-HA cluster. </description> </property>
<property> <name>dfs.journalnode.edits.dir</name> <value>/home/dongxicheng/hadoop/hdfs/journal/</value> <description>   journalnode的本地数据存放目录，指定一个路径就够。 </description> </property>
<property> <name>dfs.ha.automatic-failover.enabled</name> <value>false</value> <description>   是否自动切换。由于没有配置ZooKeeper，所以不能实现自动切换，所以这里配置的是false。 Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HA configuration. </description> </property>
</configuration>

yarn-site.xml <configuration>  <property> <description> 指定ResourceManager The hostname of the RM.</description> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <description>The address of the applications manager interface in the RM.</description> <name>yarn.resourcemanager.address</name> <value>${yarn.resourcemanager.hostname}:8032</value> </property>
<property> <description>The address of the scheduler interface.</description> <name>yarn.resourcemanager.scheduler.address</name> <value>${yarn.resourcemanager.hostname}:8030</value> </property>
<property> <description>The http address of the RM web application.</description> <name>yarn.resourcemanager.webapp.address</name> <value>${yarn.resourcemanager.hostname}:8088</value> </property>
<property> <description>The https adddress of the RM web application.</description> <name>yarn.resourcemanager.webapp.https.address</name> <value>${yarn.resourcemanager.hostname}:8090</value> </property>
<property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>${yarn.resourcemanager.hostname}:8031</value> </property>
<property> <description>The address of the RM admin interface.</description> <name>yarn.resourcemanager.admin.address</name> <value>${yarn.resourcemanager.hostname}:8033</value> </property>
<property> <description> 指定fairscheduler调度器 The class to use as the resource scheduler. </description> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property>
<property> <description> 指定fairscheduler调度器配置文件路径 fair-scheduler conf location </description> <name>yarn.scheduler.fair.allocation.file</name> <value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value> </property>
<property> <description> 指定nodemanager的本地工作目录，推荐配置多个路径，用逗号分隔 List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. </description> <name>yarn.nodemanager.local-dirs</name> <value>/home/yarn/Hadoop/yarn/local</value> </property>
<property> <description>Whether to enable log aggregation</description> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
<property> <description>Where to aggregate logs to.</description> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/home/yarn/Hadoop/yarn/tmp/logs</value> </property>
<property> <description> 每个nodemanager上可以用的内存大小 Amount of physical memory, in MB, that can be allocated for containers. 注意：我的NM虚拟机是1G内存，1核CPU，当该值配置小于1024时，NM是无法启动的！会报错： NodeManager from slavenode2 doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager. </description> <name>yarn.nodemanager.resource.memory-mb</name> <value>1024</value> </property>
<property> <description> 每个nodemanager上可用的CPU核数 Number of CPU cores that can be allocated for containers.</description> <name>yarn.nodemanager.resource.cpu-vcores</name> <value>1</value> </property>
<property> <description>the valid service name should only contain a-zA-Z0-9_ and can not start with numbers</description> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>

slaves 指定slave机器的host名即可

fairscheduler.xml 下面的例子把把集群资源配置成3个队列，为每个队列配置内存、CPU核、运行程序上限个数、权重等信息。 <?xml version="1.0"?> <allocations>
<queue name="infrastructure"> <minResources>5 mb, 1 vcores </minResources> <maxResources>60 mb, 1 vcores </maxResources> <maxRunningApps>10</maxRunningApps> <minSharePreemptionTimeout>300</minSharePreemptionTimeout> <weight>1.0</weight> <aclSubmitApps>root,yarn</aclSubmitApps> </queue>
<queue name="tool"> <minResources>5 mb, 1 vcores</minResources> <maxResources>10 mb, 1 vcores</maxResources> </queue>
<queue name="sentiment"> <minResources>5 mb, 1 vcores</minResources> <maxResources>15 mb, 1 vcores</maxResources> </queue>
</allocations>
第三步：将在一台机器上配好的所有配置文件scp到其它所有节点机器
第四步：启动HDFS HA + YARN集群注意：所有操作均在Hadoop部署目录下进行。
启动Hadoop集群： Step1 : 在各个JournalNode节点上，输入以下命令启动journalnode服务： sbin/hadoop-daemon.sh start journalnode
Step2: 在[nn1]上，对其进行格式化，并启动： bin/hdfs namenode -format sbin/hadoop-daemon.sh start namenode
Step3: 在[nn2]上，同步nn1的元数据信息： bin/hdfs namenode -bootstrapStandby
Step4: 启动[nn2]： sbin/hadoop-daemon.sh start namenode
经过以上四步操作，nn1和nn2均处理standby状态 Step5: 将[nn1]切换为Active bin/hdfs haadmin -transitionToActive nn1
Step6: 在[nn1]上，启动所有datanode sbin/hadoop-daemons.sh start datanode

下面在RM所在master节点启动YARN: sbin/start-yarn.sh
在运行MRJS的slave1上执行以下命令启动MR JobHistory Server： sbin/mr-jobhistory-daemon.sh start historyserver

至此，HDFS HA + YARN都成功启动完毕，在各个节点输入jps查看进程：

也可以用web查看： HDFS HA界面： master:50070/dfshealth.jsp slave1:50070/dfshealth.jsp
YARN界面： master:8088

第五步：停止集群在RM和NN所在节点master执行： 先停止yarn： sbin/stop-yarn.sh 再停止hdfs： sbin/stop-dfs.sh
在运行JobHistoryServer的slave1上执行： 停止JobHistoryServer： sbin/mr-jobhistory-daemon.sh stop historyserver

第六步：再次启动注意，再次启动时，所有的格式化命令都不用执行了！！！

推荐文章：

YARN应用实例

Apache Hadoop 2.2.0 HDFS HA + YARN多机部署