_00014 hadoop-2.2.0 伪分布式环境搭建

_00014 hadoop-2.2.0 伪分布式环境搭建

博文作者：妳那伊抹微笑
个性签名：世界上最遥远的距离不是天涯，也不是海角，而是我站在妳的面前，妳却感觉不到我的存在
技术方向：hadoop，数据分析与挖掘
转载声明：可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明，谢谢合作！
qq交流群：214293307  （期待与你一起学习，共同进步）

# Hado-2-2.0伪分布式安装

# 从Apache的官网下载Hadoop-2.2.0的安装包

http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.2.0/

# 前言

一些基本需要安装的东西这里就不再赘述的，如果不懂的话看前面hadoop-1.x的安装就ok了，比如 ssh免密钥登录，时间同步等！

# 解压

[root@master hadoop]# pwd

/usr/local/yting/hadoop

[root@masterhadoop]# ll

total 106672

-rwxr--r--. 1root root 109229073 May 11 14:46 hadoop-2.2.0.tar.gz

[root@masterhadoop]# tar -zxvf hadoop-2.2.0.tar.gz

[root@masterhadoop]# ll

total 106676

drwxr-xr-x. 967974 users      4096 Oct  7  2013hadoop-2.2.0

-rwxr--r--. 1root  root  109229073 May 11 14:46 hadoop-2.2.0.tar.gz

# 修改7个配置文件

~/hadoop-2.2.0/etc/hadoop/hadoop-env.sh

~/hadoop-2.2.0/etc/hadoop/core-site.xml

~/hadoop-2.2.0/etc/hadoop/hdfs-site.xml

~/hadoop-2.2.0/etc/hadoop/mapred-site.xml

~/hadoop-2.2.0/etc/hadoop/yarn-env.sh

~/hadoop-2.2.0/etc/hadoop/yarn-site.xml

~/hadoop-2.2.0/etc/hadoop/slaves

# 1修改hadoop-env.sh配置文件（jdk 路径）

# 将#export JAVA_HOME=${JAVA_HOME}

修改成exportJAVA_HOME=/usr/local/java/jdk1.6.0_45（具体JDK目录以你自己的为准）

[root@master.ssh]# cd /usr/local/yting/hadoop/hadoop-2.2.0/etc/hadoop/

[root@masterhadoop]# ls

capacity-scheduler.xml      httpfs-site.xml

configuration.xsl           log4j.properties

container-executor.cfg      mapred-env.cmd

core-site.xml               mapred-env.sh

hadoop-env.cmd              mapred-queues.xml.template

hadoop-env.sh               mapred-site.xml.template

hadoop-metrics2.properties  slaves

hadoop-metrics.properties   ssl-client.xml.example

hadoop-policy.xml           ssl-server.xml.example

hdfs-site.xml               yarn-env.cmd

httpfs-env.sh               yarn-env.sh

httpfs-log4j.properties     yarn-site.xml

httpfs-signature.secret

[root@master hadoop]# vi hadoop-env.sh

 

# The javaimplementation to use.

#exportJAVA_HOME=${JAVA_HOME}

export JAVA_HOME=/usr/local/java/jdk1.6.0_45

 

# 2修改core-site.xml文件修改

[root@masterhadoop]# vi core-site.xml

 

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version2.0 (the "License");

  you may not use this file except incompliance with the License.

  You may obtain a copy of the License at

 

    http://www.apache.org/licenses/LICENSE-2.0

 

  Unless required by applicable law or agreedto in writing, software

  distributed under the License is distributedon an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.

  See the License for the specific languagegoverning permissions and

  limitations under the License. Seeaccompanying LICENSE file.

-->

 

<!-- Putsite-specific property overrides in this file. -->

 

<configuration>

       <property>

              <name>fs.defaultFS</name>

              <value>hdfs://master:9000</value>

       </property>

    <property>

              <name>hadoop.tmp.dir</name>

              <value>/usr/local/yting/hadoop/hadoop-2.2.0/tmp</value>

              <description>A base for other temporarydirectories.</description>

       </property>

</configuration>

# 3修改hdfs-site.xml配置文件

[root@master hadoop]# vi hdfs-site.xml

 

<?xmlversion="1.0" encoding="UTF-8"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version2.0 (the "License");

  you may not use this file except incompliance with the License.

  You may obtain a copy of the License at

 

    http://www.apache.org/licenses/LICENSE-2.0

 

  Unless required by applicable law or agreedto in writing, software

  distributed under the License is distributedon an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.

  See the License for the specific languagegoverning permissions and

  limitations under the License. Seeaccompanying LICENSE file.

-->

 

<!-- Putsite-specific property overrides in this file. -->

 

<configuration>

       <property>

              <name>dfs.namenode.secondary.http-address</name>

              <value>master:9001</value>

       </property>

       <property>

              <name>dfs.replication</name>

              <value>1</value>

       </property>

       <property>

              <name>dfs.webhdfs.enabled</name>

       <value>true</value>

       </property>

</configuration>

# 4修改 mapred­-site.xml配置文件

[root@masterhadoop]# cp mapred-site.xml.template mapred-site.xml

[root@masterhadoop]# vi mapred-site.xml

 

 

<?xmlversion="1.0"?>

<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>

<!--

  Licensed under the Apache License, Version2.0 (the "License");

  you may not use this file except incompliance with the License.

  You may obtain a copy of the License at

 

    http://www.apache.org/licenses/LICENSE-2.0

 

  Unless required by applicable law or agreedto in writing, software

  distributed under the License is distributedon an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.

  See the License for the specific languagegoverning permissions and

  limitations under the License. See accompanyingLICENSE file.

-->

 

<!-- Putsite-specific property overrides in this file. -->

 

<configuration>

       <property>

              <name>mapreduce.framework.name</name>

              <value>yarn</value>

       </property>

       <property>

              <name>mapreduce.jobhistory.address</name>

              <value>master:10020</value>

       </property>

       <property>

              <name>mapreduce.jobhistory.webapp.address</name>

              <value>master:19888</value>

       </property>

</configuration>

# 5修改yarn-env.sh配置文件

[root@masterhadoop]# vi yarn-env.sh

 

# Licensed tothe Apache Software Foundation (ASF) under one or more

# contributorlicense agreements.  See the NOTICE filedistributed with

# this work foradditional information regarding copyright ownership.

# The ASFlicenses this file to You under the Apache License, Version 2.0

# (the"License"); you may not use this file except in compliance with

# theLicense.  You may obtain a copy of theLicense at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unlessrequired by applicable law or agreed to in writing, software

# distributedunder the License is distributed on an "AS IS" BASIS,

# WITHOUTWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See theLicense for the specific language governing permissions and

# limitationsunder the License.

 

# User for YARNdaemons

exportHADOOP_YARN_USER=${HADOOP_YARN_USER:-yarn}

 

# resolve links- $0 may be a softlink

exportYARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"

 

# some Javaparameters

export JAVA_HOME=/usr/local/java/jdk1.6.0_45

if ["$JAVA_HOME" != "" ]; then

  #echo "run java in $JAVA_HOME"

  JAVA_HOME=$JAVA_HOME

fi

 

if ["$JAVA_HOME" = "" ]; then

  echo "Error: JAVA_HOME is not set."

  exit 1

fi

 

JAVA=$JAVA_HOME/bin/java

JAVA_HEAP_MAX=-Xmx1000m

 

# For settingYARN specific HEAP sizes please use this

# Parameter andset appropriately

#YARN_HEAPSIZE=1000

 

# check envvarswhich might override default args

if ["$YARN_HEAPSIZE" != "" ]; then

  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"

fi

 

# ResourceManager specific parameters

 

# Specify themax Heapsize for the ResourceManager using a numerical value

# in the scaleof MB. For example, to specify an jvm option of -Xmx1000m, set

# the value to1000.

# This valuewill be overridden by an Xmx setting specified in either YARN_OPTS

# and/orYARN_RESOURCEMANAGER_OPTS.

# If notspecified, the default value will be picked from either YARN_HEAPMAX

# orJAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_RESOURCEMANAGER_HEAPSIZE=1000

 

# Specify theJVM options to be used when starting the ResourceManager.

# These optionswill be appended to the options specified as YARN_OPTS

# and thereforemay override any similar flags set in YARN_OPTS

#exportYARN_RESOURCEMANAGER_OPTS=

 

# Node Managerspecific parameters

 

# Specify themax Heapsize for the NodeManager using a numerical value

# in the scaleof MB. For example, to specify an jvm option of -Xmx1000m, set

# the value to1000.

# This valuewill be overridden by an Xmx setting specified in either YARN_OPTS

# and/orYARN_NODEMANAGER_OPTS.

# If notspecified, the default value will be picked from either YARN_HEAPMAX

# orJAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#exportYARN_NODEMANAGER_HEAPSIZE=1000

 

# Specify theJVM options to be used when starting the NodeManager.

# These optionswill be appended to the options specified as YARN_OPTS

# and thereforemay override any similar flags set in YARN_OPTS

#exportYARN_NODEMANAGER_OPTS=

 

# so thatfilenames w/ spaces are handled correctly in loops below

IFS=

 

 

# default logdirectory & file

if ["$YARN_LOG_DIR" = "" ]; then

 YARN_LOG_DIR="$HADOOP_YARN_HOME/logs"

fi

if ["$YARN_LOGFILE" = "" ]; then

  YARN_LOGFILE='yarn.log'

fi

 

# default policyfile for service-level authorization

if ["$YARN_POLICYFILE" = "" ]; then

  YARN_POLICYFILE="hadoop-policy.xml"

fi

 

# restoreordinary behaviour

unset IFS

 

 

YARN_OPTS="$YARN_OPTS-Dhadoop.log.dir=$YARN_LOG_DIR"

YARN_OPTS="$YARN_OPTS-Dyarn.log.dir=$YARN_LOG_DIR"

YARN_OPTS="$YARN_OPTS-Dhadoop.log.file=$YARN_LOGFILE"

YARN_OPTS="$YARN_OPTS-Dyarn.log.file=$YARN_LOGFILE"

YARN_OPTS="$YARN_OPTS-Dyarn.home.dir=$YARN_COMMON_HOME"

YARN_OPTS="$YARN_OPTS-Dyarn.id.str=$YARN_IDENT_STRING"

YARN_OPTS="$YARN_OPTS-Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"

YARN_OPTS="$YARN_OPTS-Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"

if ["x$JAVA_LIBRARY_PATH" != "x" ]; then

  YARN_OPTS="$YARN_OPTS-Djava.library.path=$JAVA_LIBRARY_PATH"

fi 

YARN_OPTS="$YARN_OPTS-Dyarn.policy.file=$YARN_POLICYFILE"

 

# 6修改yarn-site.xml配置文件

[root@masterhadoop]# vi yarn-site.xml

 

<?xmlversion="1.0"?>

<!--

  Licensed under the Apache License, Version2.0 (the "License");

  you may not use this file except incompliance with the License.

  You may obtain a copy of the License at

 

    http://www.apache.org/licenses/LICENSE-2.0

 

  Unless required by applicable law or agreedto in writing, software

  distributed under the License is distributedon an "AS IS" BASIS,

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,either express or implied.

  See the License for the specific languagegoverning permissions and

  limitations under the License. Seeaccompanying LICENSE file.

-->

<configuration>

 

<!-- Site specific YARN configuration properties -->

       <property>

              <name>yarn.nodemanager.aux-services</name>

              <value>mapreduce_shuffle</value>

       </property>

       <property>

              <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

              <value>org.apache.hadoop.mapred.ShuffleHandler</value>

       </property>

       <property>

              <name>yarn.resourcemanager.address</name>

              <value>master:8032</value>

       </property>

       <property>

              <name>yarn.resourcemanager.scheduler.address</name>

              <value>master:8030</value>

       </property>

       <property>

              <name>yarn.resourcemanager.resource-tracker.address</name>

              <value>master:8031</value>

       </property>

       <property>

              <name>yarn.resourcemanager.admin.address</name>

              <value>master:8033</value>

       </property>

       <property>

              <name>yarn.resourcemanager.webapp.address</name>

              <value>master:8088</value>

       </property>

</configuration>

# 7修改slaves配置文件

[root@masterhadoop]# vi slaves

 

master

 

# 伪分布式 启动Hadoop

# 格式化namenode

[root@master ~]# hdfs namenode -format

# 启动start-dfs.sh

[root@master bin]# start-dfs.sh

[root@master bin]# jps

6381 Jps

6034 NameNode

6120 DataNode

6253 SecondaryNameNode

# 启动start-yarn.sh

[root@masterlogs]# start-yarn.sh

# 注意：启动每个命令的时候，注意去看看对应的日志，有错误发生的话就可以解决了，因为上面以前用的是Hadoop1.X的版本，现在用的是Hadoop2.X的版本，有许多地方不一样了，有空再弄一下这方面的资料。

# 运行Hadoop2自带的例子，看看环境是否搭建成功了

# 如果能出现下面的结果，那么证明你运行成功了

# 注意：这里的话运行hadoop jar hadoop-mapreduce-examples-2.2.0.jar pi 10 100这个命令灰常消耗内存，小心点用了，一开始用的自己的电脑，使用虚拟机，虚拟机直接卡死了，饿还以为是饿装错了，然后把map跟reduce的数量减少hadoop jar hadoop-mapreduce-examples-2.2.0.jarpi 2 4 能出结果，大概是3.5，这样的话消耗的内存很小，不会出现死机的现象，然后饿就使用公司的服务器，65G内存，运行该命令，能出现下面这样的结果，但是消耗了大概1.3G左右的内存，坑爹啊！自己的笔记本不卡死才怪、、、

[root@mastermapreduce]# pwd

/usr/local/yting/hadoop/hadoop-2.2.0/share/hadoop/mapreduce

[root@mastermapreduce]# hadoop jar hadoop-mapreduce-examples-2.2.0.jar pi 10 100

Number ofMaps  = 10

Samples per Map= 100

14/05/1120:51:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library foryour platform... using builtin-java classes where applicable

Wrote input forMap #0

Wrote input forMap #1

Wrote input forMap #2

Wrote input forMap #3

Wrote input forMap #4

Wrote input forMap #5

Wrote input forMap #6

Wrote input forMap #7

Wrote input forMap #8

Wrote input forMap #9

Starting Job

14/05/1120:51:25 INFO client.RMProxy: Connecting to ResourceManager atmaster/116.255.224.229:8032

14/05/1120:51:26 INFO input.FileInputFormat: Total input paths to process : 10

14/05/1120:51:26 INFO mapreduce.JobSubmitter: number of splits:10

14/05/1120:51:26 INFO Configuration.deprecation: user.name is deprecated. Instead, usemapreduce.job.user.name

14/05/1120:51:26 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, usemapreduce.job.jar

14/05/1120:51:26 INFO Configuration.deprecation: mapred.map.tasks.speculative.executionis deprecated. Instead, use mapreduce.map.speculative

14/05/1120:51:26 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated.Instead, use mapreduce.job.reduces

14/05/1120:51:26 INFO Configuration.deprecation: mapred.output.value.class isdeprecated. Instead, use mapreduce.job.output.value.class

14/05/1120:51:26 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.executionis deprecated. Instead, use mapreduce.reduce.speculative

14/05/1120:51:26 INFO Configuration.deprecation: mapreduce.map.class is deprecated.Instead, use mapreduce.job.map.class

14/05/1120:51:26 INFO Configuration.deprecation: mapred.job.name is deprecated.Instead, use mapreduce.job.name

14/05/1120:51:26 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated.Instead, use mapreduce.job.reduce.class

14/05/1120:51:26 INFO Configuration.deprecation: mapreduce.inputformat.class isdeprecated. Instead, use mapreduce.job.inputformat.class

14/05/1120:51:26 INFO Configuration.deprecation: mapred.input.dir is deprecated.Instead, use mapreduce.input.fileinputformat.inputdir

14/05/1120:51:26 INFO Configuration.deprecation: mapred.output.dir is deprecated.Instead, use mapreduce.output.fileoutputformat.outputdir

14/05/1120:51:26 INFO Configuration.deprecation: mapreduce.outputformat.class isdeprecated. Instead, use mapreduce.job.outputformat.class

14/05/1120:51:26 INFO Configuration.deprecation: mapred.map.tasks is deprecated.Instead, use mapreduce.job.maps

14/05/1120:51:26 INFO Configuration.deprecation: mapred.output.key.class is deprecated.Instead, use mapreduce.job.output.key.class

14/05/1120:51:26 INFO Configuration.deprecation: mapred.working.dir is deprecated.Instead, use mapreduce.job.working.dir

14/05/1120:51:26 INFO mapreduce.JobSubmitter: Submitting tokens for job:job_1399812409150_0002

14/05/1120:51:26 INFO impl.YarnClientImpl: Submitted application application_1399812409150_0002to ResourceManager at master/116.255.224.229:8032

14/05/1120:51:26 INFO mapreduce.Job: The url to track the job:http://master:8088/proxy/application_1399812409150_0002/

14/05/1120:51:26 INFO mapreduce.Job: Running job: job_1399812409150_0002

14/05/1120:51:31 INFO mapreduce.Job: Job job_1399812409150_0002 running in uber mode :false

14/05/1120:51:31 INFO mapreduce.Job:  map 0%reduce 0%

14/05/1120:51:37 INFO mapreduce.Job:  map 40%reduce 0%

14/05/1120:51:38 INFO mapreduce.Job:  map 60%reduce 0%

14/05/1120:51:42 INFO mapreduce.Job:  map 100%reduce 0%

14/05/1120:51:43 INFO mapreduce.Job:  map 100%reduce 100%

14/05/1120:51:43 INFO mapreduce.Job: Job job_1399812409150_0002 completed successfully

14/05/1120:51:43 INFO mapreduce.Job: Counters: 43

       File System Counters

              FILE: Number of bytes read=226

              FILE: Number of byteswritten=874440

              FILE: Number of read operations=0

              FILE: Number of large readoperations=0

              FILE: Number of write operations=0

              HDFS: Number of bytes read=2590

              HDFS: Number of bytes written=215

              HDFS: Number of read operations=43

              HDFS: Number of large readoperations=0

              HDFS: Number of write operations=3

       Job Counters

              Launched map tasks=10

              Launched reduce tasks=1

              Data-local map tasks=10

              Totaltime spent by all maps in occupied slots (ms)=31111

              Total time spent by all reduces inoccupied slots (ms)=3075

       Map-Reduce Framework

              Map input records=10

              Map output records=20

              Map output bytes=180

              Map output materialized bytes=280

              Input split bytes=1410

              Combine input records=0

              Combine output records=0

              Reduce input groups=2

              Reduce shuffle bytes=280

              Reduce input records=20

              Reduce output records=0

              Spilled Records=40

              Shuffled Maps =10

              Failed Shuffles=0

              Merged Map outputs=10

              GC time elapsed (ms)=227

              CPU time spent (ms)=8910

              Physical memory (bytes)snapshot=2826170368

              Virtual memory (bytes)snapshot=9664532480

              Total committed heap usage(bytes)=2210988032

       Shuffle Errors

              BAD_ID=0

              CONNECTION=0

              IO_ERROR=0

              WRONG_LENGTH=0

              WRONG_MAP=0

              WRONG_REDUCE=0

       File Input Format Counters

              Bytes Read=1180

       File Output Format Counters

              Bytes Written=97

Job Finished in17.716 seconds

Estimated valueof Pi is 3.14800000000000000000

 

# hadoop.apache.org官网的一些命令Hadoop Startup

Tostart a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Formata new distributed filesystem:

$$HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

Startthe HDFS with the following command, run on the designated NameNode:

$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstart namenode

Runa script to start DataNodes on all slaves:

$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstart datanode

Startthe YARN with the following command, run on the designated ResourceManager:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR startresourcemanager

Runa script to start NodeManagers on all slaves:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR startnodemanager

Starta standalone WebAppProxy server. If multiple servers are used with loadbalancing it should be run on each of them:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config$HADOOP_CONF_DIR

Startthe MapReduce JobHistory Server with the following command, run on thedesignated server:

$$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config$HADOOP_CONF_DIR

HadoopShutdown

Stop the NameNode withthe following command, run on the designated NameNode:

$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstop namenode

Run a script to stopDataNodes on all slaves:

$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstop datanode

Stop the ResourceManagerwith the following command, run on the designated ResourceManager:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stopresourcemanager

Run a script to stopNodeManagers on all slaves:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stopnodemanager

Stop the WebAppProxyserver. If multiple servers are used with load balancing it should be run oneach of them:

$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config$HADOOP_CONF_DIR

Stop the MapReduceJobHistory Server with the following command, run on the designated server:

$$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config$HADOOP_CONF_DIR

Operating the Hadoop Cluster

Once all the necessaryconfiguration is complete, distribute the files to the HADOOP_CONF_DIR directoryon all the machines.

This section alsodescribes the various Unix users who should be starting the various componentsand uses the same Unix accounts and groups used previously:

Hadoop Startup

Tostart a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Formata new distributed filesystem as hdfs:

[hdfs]$$HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

Startthe HDFS with the following command, run on the designated NameNode as hdfs:

[hdfs]$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstart namenode

Runa script to start DataNodes on all slaves as root with a special environmentvariable HADOOP_SECURE_DN_USER set to hdfs:

[root]$HADOOP_SECURE_DN_USER=hdfs $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config$HADOOP_CONF_DIR --script hdfs start datanode

Startthe YARN with the following command, run on the designated ResourceManager as yarn:

[yarn]$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager

Runa script to start NodeManagers on all slaves as yarn:

[yarn]$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR startnodemanager

Starta standalone WebAppProxy server. Run on the WebAppProxy server as yarn. If multiple servers are used with load balancing it should be runon each of them:

[yarn]$$HADOOP_YARN_HOME/bin/yarn start proxyserver --config $HADOOP_CONF_DIR

Startthe MapReduce JobHistory Server with the following command, run on thedesignated server as mapred:

[mapred]$$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config$HADOOP_CONF_DIR

HadoopShutdown

Stop the NameNode with the followingcommand, run on the designated NameNode as hdfs:

[hdfs]$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstop namenode

Run a script to stop DataNodes on allslaves as root:

[root]$$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfsstop datanode

Stop the ResourceManager with the followingcommand, run on the designated ResourceManager as yarn:

[yarn]$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stopresourcemanager

Run a script to stop NodeManagers on allslaves as yarn:

[yarn]$$HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stopnodemanager

Stop the WebAppProxy server. Run on theWebAppProxy server as yarn. If multiple servers are used withload balancing it should be run on each of them:

[yarn]$ $HADOOP_YARN_HOME/bin/yarn stopproxyserver --config $HADOOP_CONF_DIR

Stop the MapReduce JobHistory Server withthe following command, run on the designated server as mapred:

[mapred]$$HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config$HADOOP_CONF_DIR

Web Interfaces

Once the Hadoop cluster is up and runningcheck the web-ui of the components as described below:

Daemon	Web Interface	Notes
NameNode	http://nn_host:port/	Default HTTP port is 50070.
ResourceManager	http://rm_host:port/	Default HTTP port is 8088.
MapReduce JobHistory Server	http://jhs_host:port/	Default HTTP port is 19888.