hadoop2.x编译（三），hadoop2.x编译

文章由LinuxBoy分享于2019-03-27 10:03:30热评（22）

hadoop2.x编译（三），hadoop2.x编译

4.3.编译

4.3.1.检查

检查jdk版本：

[root@hadoop ~]# java -version
java version "1.8.0_51"
Java(TM) SE Runtime Environment (build 1.8.0_51-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)
确保jdk版本大于等于1.7

4.3.1 进入hadoop-2.7.1-src

4.3.2 切换到root用户编译（以下指令需手动输入，不可采用拷贝、粘贴方式）

mvn clean package -DskipTests -Pdist,native –Dtar

上面这行命令大意是先清理target目录,再打包,同时跳过测试,应用dist,native这2种profile,且最终目标是生成tar包

编译成功则看到如下结果

-------------------------------------------------------------------------------------------------------------------------------

main:

[exec] $ tar cf hadoop-2.7.1.tar hadoop-2.7.1
[exec] $ gzip -f hadoop-2.7.1.tar
[exec]
[exec] Hadoop dist tar available at: /usr/local/hadoop-2.7.1-src/hadoop-dist/target/hadoop-2.7.1.tar.gz
[exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /usr/local/hadoop-2.7.1-src/hadoop-dist/target/hadoop-dist-2.7.1-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 4.452 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 3.543 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 8.639 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.445 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 2.324 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 7.837 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 7.171 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 7.337 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 4.318 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [02:56 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 19.335 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 16.521 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.083 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [05:42 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 42.298 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 21.173 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 7.716 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.204 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.101 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [02:27 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [01:01 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.282 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 32.689 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 27.916 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 8.713 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 14.560 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 34.421 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 13.602 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 10.493 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 5.543 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.114 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.971 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 2.391 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.115 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 7.855 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 7.444 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.322 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 45.166 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 34.255 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 13.295 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 16.393 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 11.293 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 10.609 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 2.290 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 8.980 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 5.338 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 6.591 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 25.947 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 4.821 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 8.802 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 5.837 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 4.083 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 2.938 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 3.847 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 12.748 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 9.641 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 11.696 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 8.377 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 18.626 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.469 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 12.486 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 12.119 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.120 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:18 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:54 min
[INFO] Finished at: 2015-08-31T00:01:54-07:00
[INFO] Final Memory: 163M/494M
[INFO] ------------------------------------------------------------------------

[root@hadoop hadoop-2.7.1-src]#

4.3.3.编译结果

编译后的打包结果保存在路径 /usr/local/hadoop-2.7.1-src/hadoop-dist/target

[root@hadoop target]# ll
total 571120
drwxr-xr-x. 2 root root 4096 Aug 31 00:00 antrun
-rw-r--r--. 1 root root 1867 Aug 31 00:00 dist-layout-stitching.sh
-rw-r--r--. 1 root root 640 Aug 31 00:00 dist-tar-stitching.sh
drwxr-xr-x. 9 root root 4096 Aug 31 00:00 hadoop-2.7.1
-rw-r--r--. 1 root root 194638968 Aug 31 00:01 hadoop-2.7.1.tar.gz
-rw-r--r--. 1 root root 2824 Aug 31 00:00 hadoop-dist-2.7.1.jar
-rw-r--r--. 1 root root 390148326 Aug 31 00:01 hadoop-dist-2.7.1-javadoc.jar
drwxr-xr-x. 2 root root 4096 Aug 31 00:01 javadoc-bundle-options
drwxr-xr-x. 2 root root 4096 Aug 31 00:00 maven-archiver
drwxr-xr-x. 2 root root 4096 Aug 31 00:00 test-dir
[root@hadoop target]# pwd
/usr/local/hadoop-2.7.1-src/hadoop-dist/target

5.编译问题：

5.1在编译到Apache Hadoop HDFS遇阻，错误提示没有能记录下来，最后重启虚拟机，重新编译，就过去了。

5.2在编译到Apache Hadoop Amazon Web Services support的时候提示缺少几个jar包，检查了以下没有修改maven的setting，重新添加如下内容上去

<mirror>
<id>nexus</id>
<mirrorOf>*</mirrorOf>
<url>http://maven.oschina.net/content/groups/public/</url>
</mirror>

然后就一次性编译通过了。

6.其他

有其他问题欢迎hadoop交流群：158532099

推荐文章：

hadoop2.x编译（三），hadoop2.x编译