Hadoop学习笔记(一)——编译安装和配置
Hadoop学习笔记(一)——编译安装和配置
最近工作调动,打算补一下大数据处理的知识,可能会陆续涉及hadoop、mongodb、ddbs等。首先Apache提供二进制的Hadoop版本是32位的,在启动时总是有警告,所以想自己编译一遍,部分内容来源于网络。
0、环境说明:
操作系统环境:
ububtu 14.04 LTS 64位操作系统
JDK环境:
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
1、JDK安装
2、安装编译依赖文件
sudo apt-get install g++ autoconf automake libtool make cmake zlib1g-dev pkg-config libssl-dev maven
之后还要安装 protobuf和findbugs,这两个只要编译安装就行了很简单,可以自行百度,
3、下载并编译Hadoop源码
hadoop 2.2.0源码下载地址 http://www.apache.org/dyn/closer.cgi/hadoop/common/
目前的2.2.0 的Source Code 压缩包解压出来的code有个bug 需要patch后才能编译。否则编译hadoop-auth 会提示错误。
需要修改下面的pom文件。该文件在hadoop源码包下寻找:
hadoop-common-project/hadoop-auth/pom.xml
打开上面的的pom文件,在54行加入如下的依赖:
<dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty-util</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.mortbay.jetty</groupId> <artifactId>jetty</artifactId> <scope>test</scope> </dependency>
之后在源码目录执行如下命令即可,编译过程十分漫长,因为需要下载好多maven的依赖文件,需要耐心等待
mvn package -Pdist,native -DskipTests -Dtar
编译完之后将生成的tar$HADOOP_HOME/etc/hadoop包解压到一个目录安装即可,我放到了/usr/local/hadoop目录下
并在配置hadoop的环境变量 同样是执行vim /etc/profile 在最下面添加
export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
4、配置hadoop为伪分布式模式
配置文件在$HADOOP_HOME/etc/hadoop下
首先是core-site.xml添加如下配置
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop/tmp</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> <final>true</final> </property> </configuration>yarn-site.xml添加如下配置
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>shuffle service that needsto be set for Map Reduce to run </description> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>localhost</value> <description>hostanem of RM</description> </property> </configuration>hdfx-site.xml添加入下配置,并建立配置中的文件夹
<configuration> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/dfs/name</value> <final>true</final> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/dfs/data</value> <final>true</final> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> </configuration>mapred-site.xml添加如下配置
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.system.dir</name> <value>file:/usr/local/hadoop/mapred/system</value> <final>true</final> </property> <property> <name>mapred.local.dir</name> <value>file:/usr/local/hadoop/mapred/local</value> <final>true</final> </property> </configuration>
之后可以在/usr/local/hadoop/sbin中执行 start-dfs.sh 如果出现类似下面结果则代表主从文件系统启动成功
Starting namenodes on [localhost] localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-yang-namenode-ubuntu.out localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-yang-datanode-ubuntu.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-yang-secondarynamenode-ubuntu.out接着运行start-yarn.sh 如果没有错误将出现类似下面的结果
starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-yang-resourcemanager-ubuntu.out localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-yang-nodemanager-ubuntu.out至此hadoop安装配置阶段就完成了。如果有机会将会接着写hadoop mapreduce初级编程,高级编程,hive等内容。
评论暂时关闭