【甘道夫】Java Hello World on Spark,hellospark
【甘道夫】Java Hello World on Spark,hellospark
引言 通过Java编写Spark应用程序的HelloWorld,虽然有点寒碜,没用Scala简洁明了,但还是得尝试和记录下。环境 Windows7 Eclipse+Maven Jdk1.7 Ubuntu 14.04
步骤一:在eclipse中创建maven工程,过程很简单,不详述。 pom文件为: <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <groupId>edu.berkeley</groupId> <artifactId>SparkProj</artifactId> <modelVersion>4.0.0</modelVersion> <name>Spark Project</name> <packaging>jar</packaging> <version>1.0</version> <dependencies> <dependency> <!-- Spark dependency --> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.3.0</version> </dependency> </dependencies> </project>
步骤二:编写核心逻辑代码 功能很简单,统计集群中Spark根目录下README.md中有多少行包含a,多少行包含b。 说实话该功能用Scala编写十分简单,Java实在是恶心。
package edu.berkeley.SparkProj;
/* SimpleApp.java */ import org.apache.spark.api.java.*; import org.apache.spark.SparkConf; import org.apache.spark.api.java.function.Function;
public class SimpleApp { public static void main(String[] args) { String logFile = "file:///home/fulong/Spark/spark-1.3.0-bin-hadoop2.4/README.md"; // Should be some file on your system SparkConf conf = new SparkConf().setAppName("Simple Application"); JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<String> logData = sc.textFile(logFile).cache();
long numAs = logData.filter(new Function<String, Boolean>() { public Boolean call(String s) { return s.contains("a"); } }).count();
long numBs = logData.filter(new Function<String, Boolean>() { public Boolean call(String s) { return s.contains("b"); } }).count();
System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs); } }
步骤三:Windows下CMD中进入到工程根目录下打包 D:\WorkSpace2015\SparkProj>mvn package 生成jar包: D:\WorkSpace2014\SparkProj\target\SparkProj-1.0.jar
步骤四:将该包通过WinSCP工具拷贝到集群某节点的目录下 /home/fulong/Workspace/Spark/SparkProj-1.0.jar
最后一步:通过spark-submit提交程序到Spark集群 fulong@FBI016:~/Spark/spark-1.3.0-bin-hadoop2.4$ ./bin/spark-submit --class edu.berkeley.SparkProj.SimpleApp --master yarn-client /home/fulong/Workspace/Spark/SparkProj-1.0.jar
运行结果,包含a的有60行,包含b的有29行:
评论暂时关闭