hadoop-HA-部署

hadoop hdfs HA yarn HA 部署

1 java的环境变量

#set java environment

export JAVA_HOME=/opt/zhang_hadoop/jdk1.7.0_67

export JRE_HOME=$JAVA_HOME/jre

export PATH=$JAVA_HOME/bin:$PATH

export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib

#set zookeeper environment

export ZOOKEEPER_HOME=/opt/zhang_hadoop/zookeeper

export PATH=$ZOOKEEPER_HOME/bin:$PATH

#set hadoop environment

export HADOOP_HOME=/opt/zhang_hadoop/hadoop-2.3.0-cdh5.0.0

export HADOOP_DEV_HOME=${HADOOP_HOME}

export HADOOP_MAPRED_HOME=${HADOOP_DEV_HOME}

export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}

export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}

export YARN_HOME=${HADOOP_DEV_HOME}

export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export HDFS_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export YARN_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

export HADOOP_LOG_DIR=${HADOOP_DEV_HOME}/logs

export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

2、免密配置

1 、机器上创建 .ssh 文件夹 : mkdir ~/.ssh 然后执行 chmod 700 ~/.ssh （若文件夹以存在则不需要创建）

2 、然后执行 chmod 600 ~/.ssh/authorized_keys

3 、执行命令 ssh-keygen -t rsa 之后一路回车，查看刚生成的无密码钥对： cd .ssh 后执行 ll

4 、把 id_rsa.pub 追加到授权的 key 里面去。执行命令 cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

5 、修改权限：执行 chmod 600 ~/.ssh/authorized_keys

登录的机子可有私钥，被登录的机子要有登录机子的公钥。这个公钥/私钥对一般在私钥宿主机产生。

6 、确保 cat /etc/ssh/sshd_config 中存在如下内容

RSAAuthentication yes

PubkeyAuthentication yes

AuthorizedKeysFile .ssh/authorized_keys

如需修改 vi /etc/ssh/sshd_config，则在修改后执行重启 SSH 服务命令使其生效 :service sshd restart

3、关闭防火墙

重启后生效

开启： chkconfig iptables on

关闭： chkconfig iptables off 或者 /sbin/chkconfig --level 2345 iptables off 2) 即时生效，重启后失效service 方式

开启： service iptables start

关闭： service iptables stopiptables方式查看防火墙状态：

/etc/init.d/iptables status暂时关闭防火墙：

/etc/init.d/iptables stop重启iptables:

/etc/init.d/iptables restart

4、配置zookeeper

cp zoo_sample.cfg zoo.cfg

Vi conf/zoo.cfg

# The number of milliseconds of each tick

tickTime=2000

# The number of ticks that the initial

# synchronization phase can take

initLimit=10

# The number of ticks that can pass between

# sending a request and getting an acknowledgement

syncLimit=5

# the directory where the snapshot is stored.

# do not use /tmp for storage, /tmp here is just

# example sakes.

dataDir=/home/hadoop/repo1/zookeeper

# the port at which the clients will connect

clientPort=2181

#

# Be sure to read the maintenance section of the

# administrator guide before turning on autopurge.

#

# https://www.360docs.net/doc/cf8449706.html,/doc/current/zookeeperAdmin.html#sc_maintenance

#

# The number of snapshots to retain in dataDir

#autopurge.snapRetainCount=3

# Purge task interval in hours

# Set to "0" to disable auto purge feature

#autopurge.purgeInterval=1

server.1=hadoop001:2888:3888

server.2=hadoop002:2888:3888

server.3=hadoop003:2888:3888

编辑hadoop001 hadoop002 hadoop003的

vi /opt/zhang_hadoop/zookeeper/data/myid

按照下面的表哥填写myid

Hadoop001写1

Hadoop002写2

Hadoop003写3

5.配置HDFS High Availability Using the Quorum Journal Manager

1、hdfs-site.xml

#https://www.360docs.net/doc/cf8449706.html,services

#Note: If you are also using HDFS Federation, this configuration setting should also# include the list of other nameservices,HA or otherwise, as a comma-separated list.

https://www.360docs.net/doc/cf8449706.html,services

zhangcluster

https://www.360docs.net/doc/cf8449706.html,nodes.zhangcluster

nn1,nn2

#https://www.360docs.net/doc/cf8449706.html,node.rpc-address.[nameservice ID].[name node ID]

https://www.360docs.net/doc/cf8449706.html,node.rpc-address.zhangcluster.nn1

hadoop001:9000

https://www.360docs.net/doc/cf8449706.html,node.rpc-address.zhangcluster.nn2

hadoop002:9000

#https://www.360docs.net/doc/cf8449706.html,node.http-address.[nameservice ID].[name node ID]

https://www.360docs.net/doc/cf8449706.html,node.http-address.zhangcluster.nn1

hadoop001:50070

https://www.360docs.net/doc/cf8449706.html,node.http-address.zhangcluster.nn2

hadoop002:50070

#https://www.360docs.net/doc/cf8449706.html,node.shared.edits.dir - the URI which identifies the group of JNs where the NameNodes will write /read edits

#The URI should be of the form: "qjournal://host1:port1;host2:port2;host3:port3/journalId". The Journal I D is a unique identifier for this nameservice

#For example, if the JournalNodes for this cluster were running on the machines "hadoop001", "hadoop002", and "hadoop003" and the nameservice ID were "zhangcluster", you would use the following as the value for this setting

https://www.360docs.net/doc/cf8449706.html,node.shared.edits.dir

qjournal://hadoop001:8485;hadoop002:8485;hadoop003:8485/zhangcluster

#dfs.client.failover.proxy.provider.[nameservice ID]

#Configure the name of the Java class which will be used by the DFS Client to determine which NameNode is the current Active, and therefore which NameNode is currently serving client requests. The only implemen tation which currently ships with Hadoop is the ConfiguredFailoverProxyProvider, so use this unless you a re using a custom one.

dfs.client.failover.proxy.provider.zhangcluster

https://www.360docs.net/doc/cf8449706.html,node.ha.ConfiguredFailoverProxyProvider

dfs.ha.fencing.methods

sshfence

dfs.ha.fencing.ssh.private-key-files

/home/hadoop/.ssh/id_rsa

dfs.ha.automatic-failover.enabled

true

dfs.replication

1

https://www.360docs.net/doc/cf8449706.html,.dir

file:///opt/zhang_hadoop/data/name

dfs.datanode.data.dir

f ile:///op t/zhang_hadoop/data/data

#dfs.journalnode.edits.dir - the path where the JournalNode daemon will store its local state

dfs.journalnode.edits.dir

/opt/zhang_hadoop/data/journal

dfs.permissions

false

dfs.permissions.enabled

false

2、core-site.xml

hadoop.tmp.dir

/opt/zhang_hadoop/data/tmp

A base for other temporary directories.

#fs.defaultFS - the default path prefix used by the Hadoop FS client when none is given

fs.defaultFS

hdfs://zhangcluster

ha.zookeeper.quorum

hadoop001:2181,hadoop002:2181,hadoop003:2181

6、配置 ResourceManager High Availability

补充: Hadoop ResourceManager的实现不像namenode只能有两个做HA，ResourceManager的HA可以多台。

(1)yarn-site.xml

Classpath for typical applications.

yarn.application.classpath

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,

$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,

$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,

$YARN_HOME/*,$YARN_HOME/lib/*

yarn.resourcemanager.connect.retry-interval.ms

2000

yarn.resourcemanager.ha.enabled

true

yarn.resourcemanager.ha.automatic-failover.enabled

true

yarn.resourcemanager.ha.automatic-failover.embedded

true

yarn.resourcemanager.cluster-id

zhangcluster

yarn.resourcemanager.ha.rm-ids

rm1,rm2

yarn.resourcemanager.ha.id

rm1

yarn.resourcemanager.scheduler.class

org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler

yarn.resourcemanager.recovery.enabled

true

yarn.resourcemanager.store.class

org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore

yarn.resourcemanager.zk.state-store.address

localhost:2181

yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms

5000

yarn.resourcemanager.work-preserving-recovery.enabled

true

yarn.resourcemanager.address.rm1

hadoop001:23140

yarn.resourcemanager.scheduler.address.rm1

hadoop001:23130

yarn.resourcemanager.webapp.https.address.rm1

hadoop001:23189

yarn.resourcemanager.webapp.address.rm1

hadoop001:23188

yarn.resourcemanager.resource-tracker.address.rm1

hadoop001:23125

yarn.resourcemanager.admin.address.rm1

hadoop001:23141

Cloudera Administration | 235

High Availability

yarn.resourcemanager.address.rm2

hadoop002:23140

yarn.resourcemanager.scheduler.address.rm2

hadoop002:23130

yarn.resourcemanager.webapp.https.address.rm2

hadoop002:23189

yarn.resourcemanager.webapp.address.rm2

hadoop002:23188

yarn.resourcemanager.resource-tracker.address.rm2 hadoop002:23125

yarn.resourcemanager.admin.address.rm2

hadoop002:23141

Address where the localizer IPC is. yarn.nodemanager.localizer.address

0.0.0.0:23344

NM Webapp address.

yarn.nodemanager.webapp.address

0.0.0.0:23999

yarn.nodemanager.aux-services

mapreduce_shuffle

yarn.nodemanager.local-dirs

${hadoop.tmp.dir}/yarn/local

yarn.nodemanager.log-dirs

${hadoop.tmp.dir}/yarn/log

mapreduce.shuffle.port

23080

yarn.resourcemanager.work-preserving-recovery.enabled true

yarn.resourcemanager.zk-address

hadoop001:2181,hadoop002:2181,hadoop003:2181

（2）mapred-site.xml

https://www.360docs.net/doc/cf8449706.html,

yarn

yarn.app.mapreduce.am.staging-dir

${hadoop.tmp.dir}/user

mapreduce.jobhistory.address

0.0.0.0:10020

mapreduce.jobhistory.webapp.address

0.0.0.0:19888

7 、配置datanode

修改 $HADOP_HOME/etc/hadoop/slaves 即可

[root@hadoop001 hadoop]# vi slaves

hadoop001

hadoop002

hadoop003

8、配置 hadoop hdfs yarn 的java的环境变量

修改 $HADOP_HOME/etc/hadoop/ 下 hadoop-env.sh mapred-env.sh yarn-env.sh 的java的环境变量 export JAVA_HOME=/opt/zhang_hadoop/jdk1.7.0_67

9、启动集群

所有的节点执行

zkServer.sh start 启动zookeeper hadoop-daemon.sh start journalnode

在某个namenode执行

hdfs zkfc -formatZK

1.Start the primary (formatted) NameNode: 在第一个namonode执行

$ hdfs namenode -format (如果两个集群都未格式化先选一个namenode格式化)

$ hadoop-daemon.sh start namenode

2. Start the standby NameNode: 在另一个namonode执行

$ hdfs namenode -bootstrapStandby

3、hadoop-daemon.sh start datanode （所有的datanode）

4、hadoop-daemon.sh start zkfc (两个namenode)

5、yarn-daemon.sh start resourcemanager (两个namenode)

6、yarn-daemon.sh start nodemanager（所有的datanode）

7、mr-jobhistory-daemon.sh start historyserver

（配置的节点 mapred-site.xml 写的那个节点的host在哪个节点执行，不同的节点可以配置成自己的host 启动多个historyserver

）

mr-jobhistory-daemon.sh stop historyserver

2、rm application http://hadoop001:23188

http://hadoop002:23188

3、mr jobhistory http://hadoop001:19888

11、出现的错误及解决方案

error1 第二次格式化后启动datanode时候报如下的错误

解决方法 rm -rf /opt/zhang_hadoop/data/data/* 重新格式化 namenode

error2

hahoop fs -ls / 命令不能执行

启动 jobhistoryserver 时报 unkonwnhostname zhangcluster

发现 hdfs-site.xml 的下面的配置写错了

dfs.client.failover.proxy.provider.mycluster

https://www.360docs.net/doc/cf8449706.html,node.ha.ConfiguredFailoverProxyProvider

改为

dfs.client.failover.proxy.provider.zhangcluster

https://www.360docs.net/doc/cf8449706.html,node.ha.ConfiguredFailoverProxyProvider

error3

格式化namenode时未启动 journalnode 报连接不上一直重连

hadoop001:8485 hadoop002:8485 hadoop003:8485 的错误

先启动 journalnode 在格式化 namenode （hdfs namenode -format）然后启动 namenode 然后再另一个namonode执行 hdfs namenode -bootstrapStandby

error4 yarn resourcemanager HA

hadoop002 启动 resourcemanager 时报错

yarn.resourcemanager.ha.id

rm1

把 hadoop002 相应的配置改成 rm2

yarn.resourcemanager.ha.id

rm2

然后分别启动两个 rm yarn-daemon.sh start resourcemanager

（rm 的HA可以配多个 nn 的HA是两个）