spark1.3.1+hive1.2.1+MySQL 集群安装

Spark1.3.1+hive1.2.1+mysql集群安装

本文基于Spark1.3.1，介绍一下Spark基于hadoop-2.6.0的安装配置和简单使用。

我是在一台机器上完成了Spark的部署，其实也是集群，只不过Master和Slave都在一台机器上。如果是多台机器的集群部署，步骤完全一样，只不过多些Slave而已,修改slave 文件。

一、配置免密码ssh登陆

因为Master和Slave处于一台机器，因此配置本机到本机的免密码ssh登陆，如有其他Slave，都需要配置Master到Slave的无密码ssh登陆。

cd ~/

ssh-keygen （一路回车）

cd .ssh/

cat id_rsa.pub >>authorized_keys

chmod 600 authorized_keys

二、环境需求

1.下载并安装jdk-7u71-linux-x64.tar.gz

安装命令：tar –zxvfjdk-7u71-linux-x64.tar.gz

配置环境变量: vi /etc/profile

export JAVA_HOME=/usr/local/jdk1.7.0_71

export PATH=.:$JAVA_HOME/bin:$PATH

然后 source /etc/profile 生效

如图所示

2.下载并安装scala-2.9.

3.tgz,建议换成2.10以上的版本

安装命令:tar –zxvfscala-2.9.3.tgz

配置环境变量： vi /etc/profile

export SCALA_HOME=/opt/modules/scala-2.9.3

export PATH=$PATH:$SCALA_HOME/bin

然后 source /etc/profile 生效

如图所示：

3.下载编译好的Spark安装包，这里选择的是spark-1.3.1-bin-hadoop2.6.tgz，因为我

的hadoop安装的是hadoop-2.6的版本。便于集成

解压缩：tar –zxvfspark-1.3.1-bin-hadoop2.6.tgz

解压后根目录为：/opt/modules/spark-1.3.1-bin-hadoop2.6

配置环境变量：vi ~/.bash_profile

export SPARK_HOME=/opt/modules/spark-1.3.1-bin-hadoop2.6 export PATH=.:$SPARK_HOME/bin:$PATH

然后 source /etc/profile 生效

三、Spark配置

1. 配置Spark环境变量

Cd $SPARK_HOME/conf

cp spark-env.sh.template spark-env.sh

vi spark-env.sh 添加以下内容：

export JAVA_HOME=/usr/local/jdk1.7.0_71

export HADOOP_HOME=/opt/ modules/hadoop-2.6.0

export HADOOP_CONF_DIR=/etc/hadoop/conf

export SCALA_HOME=/opt/modules/scala-2.9.3

export SPARK_HOME/opt/modules/spark-1.3.1-bin-hadoop2.6 export SPARK_MASTER_IP=172.18.29.174

export SPARK_MASTER_PORT=7077

export SPARK_PID_DIR=/var/run #pid

##根据机器情况配置#

#每个Worker使用的CPU核数

export SPARK_WORKER_CORES=2

#每个Slave中启动几个Worker实例

export SPARK_WORKER_INSTANCES=1

#每个Worker使用多大的内存

export SPARK_WORKER_MEMORY=1G

# 每个Executor使用使用的核数

export SPARK_EXECUTOR_CORES=1

export SPARK_EXECUTOR_MEMORY=1G //每个Executor使用的内存如图所示：

2. 配置Slave

cpslaves.templateslaves

vi slaves 添加以下内容：

localhost

4.如果是集群安装

配置好的profile文件复制到集群机器里面去

把配置好的 spark-1.3.1-bin-hadoop2.6 文件复制到集群机器里去

命令：scpspark-1.3.1-bin-hadoop2.6@DataNode1:/opt/softwore/

四、启动Spark Master

cd $SPARK_HOME/sbin/

./start-all.sh

启动日志位于$SPARK_HOME/logs/目录下，启动成功后在浏览器输入http://172.22.39.198:8080，即可看到Spark的WebUI界面，如图：

Mysql安装

参考百度上MySQL 安装的

到mysql官网下载mysql编译好的二进制安装包，在下载页面Select Platform:选项选择linux-generic，然后把页面拉到底部，64位系统下载Linux - Generic (glibc 2.5) (x86, 64-bit)，32位系统下载Linux - Generic (glibc 2.5) (x86, 32-bit)

?解压32位安装包:

进入安装包所在目录，执行命令：

tar mysql-5.6.17-linux-glibc2.5-i686.tar.gz

?复制解压后的mysql目录到系统的本地软件目录:

执行命令：cp mysql-5.6.17-linux-glibc2.5-i686 /usr/local/mysql -r 注意：目录结尾不要加/

?添加系统mysql组和mysql用户：

执行命令：groupaddmysql和useradd -r -gmysqlmysql

?安装数据库：

进入安装mysql软件目录：执行命令 cd /usr/local/mysql

修改当前目录拥有者为mysql用户：执行命令chown -R mysql:mysql ./

安装数据库：执行命令./scripts/mysql_install_db --user=mysql

修改当前目录拥有者为root用户：执行命令chown -R root:root ./

修改当前data目录拥有者为mysql用户：执行命令chown -R mysql:mysql data 到此数据库安装完毕

?启动mysql服务和添加开机启动mysql服务：

添加开机启动：执行命令cp support-files/mysql.server /etc/init.d/mysql，把启动脚本放到开机初始化目录

启动mysql服务：执行命令service mysql start

执行命令：ps -ef|grepmysql 看到mysql服务说明启动成功，如图

?修改mysql的root用户密码，root初始密码为空的：

执行命令：./bin/mysqladmin -u root password '密码'

?把mysql客户端放到默认路径：

ln -s /usr/local/mysql/bin/mysql /usr/local/bin/mysql

注意：建议使用软链过去，不要直接包文件复制，便于系统安装多个版本的mysql

设置mysql启动mysql服务sudo service mysql start

设置为开机自启动sudochkconfigmysql on

设置root用户登录密码sudo /usr/bin/mysqladmin -u root password 'wu123'

登录mysql 以root用户身份登录mysql -uroot -pwu123

创建hive用户,数据库等insert into https://www.360docs.net/doc/269722385.html,er(Host,User,Password)

values("localhost","hive",password("hive"));

create database hive;

grant all on hive.* to hive@'%' identified by 'hive';

grant all on hive.* to hive@'localhost' identified by 'hive';

grant all on hive.* to hive@'hadoop196' identified by 'hive';

flush privileges;

查询下 select Host,User,Password from https://www.360docs.net/doc/269722385.html,er;

退出mysql exit

验证hive用户mysql -uhive -phiveshow databases;看到如下反馈信息，则说明创建成功mysql> show databases;+--------------------+|

Database |+--------------------+| information_schema ||

hive || test |+--------------------+3 rows in set (0.00 sec)

退出mysql>exit

Hive安装

这里以hive-1.2.1安装为例，hive-1.1.1的版本bug太多，不建议安装使用

一、下载hive

下载地址：https://www.360docs.net/doc/269722385.html,/apache/hive/选择下图红色方框的版本。

点击进去，选择下图中方框内的下载。下面的是源码，对源码感兴趣的也可以下载下来研究或者自己编译。，这里直接下载编译好的hive。

2.解压缩

1.把文件上传到 /opt/software 目录下，解压缩

tar–zxvfapache-hive-1.2.1-bin.tar.gz

2. 重命名

mvapache-hive-1.2.1-binhive-1.2.1

3. 在环境变量加入hive

vi /etc/profile

#hive

export HIVE_HOME=/opt/software/hive-1.2.1

export PATH=$PATH:$HIVE_HOME/bin

如图：

4.进入/opt/software/hive-1.2.1/conf 目录下

5.修改配置文件

1).cp hive-default.xml.template hive-site.xml

修改hive-site.xml文件。这里用的mysql5.45作为数据库的。

vi hive-site.xml

hive.metastore.local

false

javax.jdo.option.ConnectionURL

jdbc:mysql://172.18.29.174:3306/hive?createDatabaseIfNotExist=true

javax.jdo.option.ConnectionDriverName

com.mysql.jdbc.Driver

javax.jdo.option.ConnectionUserName

hive

javax.jdo.option.ConnectionPassword

hive

hive.metastore.uris

thrift://172.18.29.174:9083

2)修改hive-env.sh.template

Mv hive-env.sh.template hive-env.sh

在这这里加入hive ,hadoop，jdk 的路径

export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64

export HIVE_HOME=/opt/software/hive-1.2.1

export HADOOP_HOME=/opt/software/hadoop-2.6.0

如图所示，然后保存。

3）修改日志配置文件hive-log4j.properties.template

Cphive-log4j.properties.templatehive-log4j.properties

然后 vi hive-log4j.properties设置日志的级别和日志所在的路径，也可以不修改。

6 .先启动hadoop,mysql数据库，然后启动hive

进入hive的安装目录下启动hive的前台进程

./bin/hive–servicemetastore

在另一窗口启动:./bin/hive

出现如下信息，这说明安装成功

6.执行hive操作show databases; 显示所有的数据库。

在spark中运行spark-sql

1．在spark 的安装目录中confm目录下添加hive-site.xml的配置文件，并保存。

添加的内容如下

hive.metastore.uris

thrift://impala197:9083

Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore.

如图所示：

2.启动spark-sql

先启动hadoop ,mysql ,hive

启动HIVE的后台服务：$nohup hive --service metastore> metastore.log 2>&1 &

通过jobs 可以看到

3.启动spark-sql

./spark-sql --master spark://impala197:7077 --executor-memory 512M

4. usenwqc;

Spark 安装出现的问题

一、如果页面无法显示，看spark-env.sh配置是否正确，如果没有配置

export SPARK_MASTER_IP这一项的会无法访问。有两种配置

配置一：export SPARK_MASTER_IP=impala197 这样几点之前通过主机名进行访问

配置二：export SPARK_MASTER_IP=172.18.29.174 这样几点之前通过ip进行访问二、端口被占用，例如8080的端口被占用了。查看端口命令：netstat –apn，发现端口占

用了，修改端口

安装hive过程中所遇到的问题

1.解决${system:java.io.tmpdir}/${system:https://www.360docs.net/doc/269722385.html,}引起的错误：

Logging initialized using configuration

injar:file:/opt/sofeware/hive-1.2.1/lib/hive-common-1.2.1.jar!/hive-log4j.properties Exception in thread "main"https://www.360docs.net/doc/269722385.html,ng.RuntimeException:

https://www.360docs.net/doc/269722385.html,ng.IllegalArgumentException:https://www.360docs.net/doc/269722385.html,.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:https://www.360docs.net/doc/269722385.html,%7D

atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:519)

atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)

atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)

atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:4 3)

https://www.360docs.net/doc/269722385.html,ng.reflect.Method.invoke(Method.java:606)

atorg.apache.hadoop.util.RunJar.main(RunJar.java:160)

Caused by: https://www.360docs.net/doc/269722385.html,ng.IllegalArgumentException:https://www.360docs.net/doc/269722385.html,.URISyntaxException: Relative path in absolute URI:${system:java.io.tmpdir%7D/$%7Bsystem:https://www.360docs.net/doc/269722385.html,%7D

atorg.apache.hadoop.fs.Path.initialize(Path.java:148)

atorg.apache.hadoop.fs.Path.(Path.java:126)

atorg.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:560) atorg.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:505)

... 7 more

Caused by: https://www.360docs.net/doc/269722385.html,.URISyntaxException: Relative path inabsolute URI:

${system:java.io.tmpdir%7D/$%7Bsystem:https://www.360docs.net/doc/269722385.html,%7D

https://www.360docs.net/doc/269722385.html,.URI.checkPath(URI.java:1804)

https://www.360docs.net/doc/269722385.html,.URI.(URI.java:752)

atorg.apache.hadoop.fs.Path.initialize(Path.java:145)

... 10 more

解决办法如下：

1.查看hive-site.xml配置，会看到配置值含有"system:java.io.tmpdir"的配置项，修改属性：

system:java.io.tmpdir

/opt/sofeware/hive-1.2.1/iotmp

system:https://www.360docs.net/doc/269722385.html,

hive

2.权限问题

ogging initialized using configuration in

jar:file:/opt/software/hive-1.2.1/lib/hive-common-0.14.0.jar!/hive-log4j.properties Exception in thread "main" https://www.360docs.net/doc/269722385.html,ng.RuntimeException: https://www.360docs.net/doc/269722385.html,ng.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--x--x at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)

atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)

atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) https://www.360docs.net/doc/269722385.html,ng.reflect.Method.invoke(Method.java:597)

atorg.apache.hadoop.util.RunJar.main(RunJar.java:160)

Caused by: https://www.360docs.net/doc/269722385.html,ng.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--x--x

at

org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:5

29)

at

org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430) ... 7 more

解决方法：

[root@impala197 ~]# hadoop fs -chmod -R 777 /tmp/hive

[root@impala197 ~]# hadoop fs -ls /tmp

Found 1 items

drwxrwxrwx - root supergroup 0 2015-02-04 08:48 /tmp/hive

3.hadoop 没有启动问题

Logging initialized using configuration in

jar:file:/usr/local/hive/lib/hive-common-0.14.0.jar!/hive-log4j.properties

Exception in thread "main" https://www.360docs.net/doc/269722385.html,ng.RuntimeException: https://www.360docs.net/doc/269722385.html,.ConnectException: Call to hadoop/impala197:9200 failed on connection exception: https://www.360docs.net/doc/269722385.html,.ConnectException: Connection refused

at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)

atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)

atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) https://www.360docs.net/doc/269722385.html,ng.reflect.Method.invoke(Method.java:597)

atorg.apache.hadoop.util.RunJar.main(RunJar.java:160)

Caused by: https://www.360docs.net/doc/269722385.html,.ConnectException: Call to hadoop/192.168.52.139:9000 failed on connection exception: https://www.360docs.net/doc/269722385.html,.ConnectException: Connection refused

解决方法:

此问题是因为hadoop没有启动，启动hadoop后问题解决

4.启动报错[ERROR] Terminal initialization failed; falling back to unsupported

[ERROR] Terminal initialization failed; falling back to unsupported

https://www.360docs.net/doc/269722385.html,ng.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected atjline.TerminalFactory.create(TerminalFactory.java:101)

atjline.TerminalFactory.get(TerminalFactory.java:158)

atjline.console.ConsoleReader.(ConsoleReader.java:229)

atjline.console.ConsoleReader.(ConsoleReader.java:221)

atjline.console.ConsoleReader.(ConsoleReader.java:209)

at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721)

atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) https://www.360docs.net/doc/269722385.html,ng.reflect.Method.invoke(Method.java:606)

atorg.apache.hadoop.util.RunJar.run(RunJar.java:221)

atorg.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread "main" https://www.360docs.net/doc/269722385.html,ng.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected

atjline.console.ConsoleReader.(ConsoleReader.java:230)

atjline.console.ConsoleReader.(ConsoleReader.java:221)

atjline.console.ConsoleReader.(ConsoleReader.java:209)

at org.apache.hadoop.hive.cli.CliDriver.setupConsoleReader(CliDriver.java:787)

at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:721)

atorg.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)

atorg.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) https://www.360docs.net/doc/269722385.html,ng.reflect.Method.invoke(Method.java:606)

atorg.apache.hadoop.util.RunJar.run(RunJar.java:221)

atorg.apache.hadoop.util.RunJar.main(RunJar.java:136)

1.解决方法

把hadoop目录下的share/hadoop/lib目录下的jline-0.94.jar换成jline-2.12.jar即可2.据说删除jline-0.94.jar 也可以，没试过。