Intellij IDEA连接Spark集群
1. 首先安装Scala插件,File->Settings->Plugins,搜索出Scla插件,点击Install安装;
2. File->New Project->maven,新建一个Maven项目,填写GroupId和ArtifactId;
3. 编辑pom.xml文件,添加项目所需要的依赖:
<properties>
<scala.version>2.10.5</scala.version>
<Hadoop.version>2.6.5</hadoop.version>
</properties>
<repositories>
<repository>
<id>scala-tools.org</id>
<name>Scala-Tools Maven2 Repository</name>
<url>http://scala-tools.org/repo-releases</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.10</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
4. File->Project Structure->Libraries,选择和Spark运行环境一致的Scala版本:
5. File->Project Structure->Modules,在src/main/下面增加一个scala文件夹,并且设置成source文件夹;
完整文档可以到Linux公社资源站下载:
------------------------------------------分割线------------------------------------------
免费下载地址在 http://linux.linuxidc.com/
用户名与密码都是www.linuxidc.com
具体下载目录在 /2017年资料/5月/18日/Intellij IDEA连接Spark集群/
下载方法见 http://www.linuxidc.com/Linux/2013-07/87684.htm
------------------------------------------分割线------------------------------------------
更多Spark相关教程见以下内容:
CentOS 7.0下安装并配置Spark http://www.linuxidc.com/Linux/2015-08/122284.htm
Spark1.0.0部署指南 http://www.linuxidc.com/Linux/2014-07/104304.htm
Spark2.0安装配置文档 http://www.linuxidc.com/Linux/2016-09/135352.htm
Spark 1.5、Hadoop 2.7 集群环境搭建 http://www.linuxidc.com/Linux/2016-09/135067.htm
Spark官方文档 - 中文翻译 http://www.linuxidc.com/Linux/2016-04/130621.htm
CentOS 6.2(64位)下安装Spark0.8.0详细记录 http://www.linuxidc.com/Linux/2014-06/102583.htm
Spark2.0.2 Hadoop2.6.4全分布式配置详解 http://www.linuxidc.com/Linux/2016-11/137367.htm
Ubuntu 14.04 LTS 安装 Spark 1.6.0 (伪分布式) http://www.linuxidc.com/Linux/2016-03/129068.htm
Spark 的详细介绍:请点这里
Spark 的下载地址:请点这里
本文永久更新链接地址:http://www.linuxidc.com/Linux/2017-05/143957.htm