手机网站建设步骤,建设网站一定需要云服务器么,物联网平台层的主要功能,扬州市住房和城乡建设局网站1、环境准备 java -version scala -version mvn -version spark -version 2、创建spark项目
创建spark项目#xff0c;有两种方式#xff1b;一种是本地搭建hadoop和spark环境#xff0c;另一种是下载maven依赖#xff1b;最后在idea中进行配置#xff0c;下面分别记录两…1、环境准备 java -version scala -version mvn -version spark -version 2、创建spark项目
创建spark项目有两种方式一种是本地搭建hadoop和spark环境另一种是下载maven依赖最后在idea中进行配置下面分别记录两种方法
2.1 本地搭建hadoop和spark环境
参考window搭建spark IDEA开发环境
2.2 下载maven依赖
参考 Windows平台搭建Spark开发环境Intellij idea 2020.1社区版Maven 3.6.3Scala 2.11.8
参考 Intellij IDEA编写Spark应用程序超详细步骤IDEAMavenScala
2.2.1 maven项目pom配置 propertiesproject.build.sourceEncodingUTF-8/project.build.sourceEncodingspark.version2.4.0/spark.versionscala.version2.11/scala.versionscope.flagprovide/scope.flag
/propertiesdependencies!--spark 依赖--dependencygroupIdorg.apache.spark/groupIdartifactIdspark-core_${scala.version}/artifactIdversion${spark.version}/version/dependencydependencygroupIdorg.apache.spark/groupIdartifactIdspark-streaming_${scala.version}/artifactIdversion${spark.version}/version/dependencydependencygroupIdorg.apache.spark/groupIdartifactIdspark-sql_${scala.version}/artifactIdversion${spark.version}/version/dependencydependencygroupIdorg.apache.spark/groupIdartifactIdspark-hive_${scala.version}/artifactIdversion${spark.version}/version/dependencydependencygroupIdorg.apache.spark/groupIdartifactIdspark-mllib_${scala.version}/artifactIdversion${spark.version}/version/dependency!--maven自带依赖--dependencygroupIdjunit/groupIdartifactIdjunit/artifactIdversion3.8.1/versionscopetest/scope/dependency
/dependencies2.2.2 maven中settings文件配置
?xml version1.0 encodingUTF-8?
settings xmlnshttp://maven.apache.org/SETTINGS/1.0.0xmlns:xsihttp://www.w3.org/2001/XMLSchema-instancexsi:schemaLocationhttp://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd!--设置本地maven仓库--
localRepositoryD:\development\LocalMaven/localRepository!--设置镜像--
mirrorsmirroridnexus-aliyun/idmirrorOfcentral/mirrorOfnameNexus aliyun/nameurlhttp://maven.aliyun.com/nexus/content/groups/public/url/mirror
/mirrors/settings2.3 Project Settings 和 Project Structure配置 2.4 创建spark maven项目
2.4.1 Archetype选择quickstart选择JDK 2.4.2 modules新建scala Sources文件 2.4.3 libraries新增sacla sdk可以创建scala项目 3. spark程序
word count 和spark show函数
import org.apache.spark.sql.SparkSessionobject HelloWord {def main(args: Array[String]): Unit {val spark SparkSession.builder.master(local).appName(Spark CSV Reader).getOrCreateval sc spark.sparkContext// 输入文件val input D:\\Project\\RecommendSystem\\src\\main\\scala\\weekwlkl// 计算频次val count sc.textFile(input).flatMap(x x.split( )).map(x (x, 1)).reduceByKey((x, y) x y);// 打印结果count.foreach(x println(x._1 : x._2));import spark.implicits._Seq(1, 2).toDF().show()// 结束sc.stop()}
}4. 总结
创建spark项目并且本地调试通过有很多注意点包括idea的配置再次记录一下以便后面学习
tips
maven helper用来查看是否存在jar包冲突 \weekwlkl)
4. 总结
创建spark项目并且本地调试通过有很多注意点包括idea的配置再次记录一下以便后面学习
tips
maven helper用来查看是否存在jar包冲突
