hadoop2.4.1集群配置

3.core-site.xml 完整文件<?xml version="1.0"encoding="UTF-8"?><?xml-stylesheet type="text/xsl"href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); youmay not use this file except in compliance with the License. You may obtaina copy of the License at /licenses/LICENSE-2 .0 Unlessrequired by applicable law or agreed to in writing, software dist ributedunder the License is distributed on an "AS IS" BASIS, WITHOUT WAR RANTIESOR CONDITIONS OF ANY KIND, either express or implied. See the Lic ense forthe specific language governing permissions and limitations under the License.See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><property><name>fs.defaultFS</name><value>hdfs://myhadoop</value><description>NameNode UR，格式是hdfs://host:port/，如果开启了NN HA特性，则配置集群的逻辑名，具体参见我的博客/u010967382/article/details/30976935</description></property><property><name>hadoop.tmp.dir</name><value>/home/yarn/Hadoop/hadoop-2.4.1/tmp</value> </property><property><name>io.file.buffer.size</name><value>131072</value><description>Size of read/write buffer used in SequenceFiles.</description></property><property><name>ha.zookeeper.quorum</name><value>master:2181,slave1:2181,slave2:2181</value><description>注意，配置了ZK以后，在格式化、启动NameNode之前必须先启动ZK，否则会报连接错误</description></property></configuration>4.hdfs-site.xml 完整文件<?xml version="1.0"encoding="UTF-8"?><?xml-stylesheet type="text/xsl"href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); youmay not use this file except in compliance with the License. You may obtaina copy of the License at /licenses/LICENSE-2 .0 Unlessrequired by applicable law or agreed to in writing, software dist ributedunder the License is distributed on an "AS IS" BASIS, WITHOUT WAR RANTIESOR CONDITIONS OF ANY KIND, either express or implied. See the Lic ense forthe specific language governing permissions and limitations under the License.See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><!-- NN HA related configuration **BEGIN** --><property><name>services</name><value>myhadoop</value><description>Comma-separated list of nameservices.as same as fs.defaultFS in core-site.xml.</description></property><property><name>nodes.myhadoop</name><value>nn1,nn2</value><description>The prefix for a given nameservice, contains a comma-sepa ratedlist of namenodes for a given nameservice (eg EXAMPLENAME SERVICE).</description></property><property><name>node.rpc-address.myhadoop.nn1</name><value>master:8020</value><description>RPC address for nomenode1 of hadoop-test</description></property><property><name>node.rpc-address.myhadoop.nn2</name><value>slave1:8020</value><description>RPC address for nomenode2 of hadoop-test</description></property><property><name>node.http-address.myhadoop.nn1</name><value>master:50070</value><description>The address and the base port where the dfs namenode1 web ui will listenon.</description></property><property><name>node.http-address.myhadoop.nn2</name><value>slave1:50070</value><description>The address and the base port where the dfs namenode2 web ui will listenon.</description></property><property><name>node.servicerpc-address.myhadoop.n1</name><value>master:53310</value></property><property><name>node.servicerpc-address.myhadoop.n2</name><value>slave1:53310</value></property><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value><description>Whether automatic failover is enabled. See the HDFS High Availability documentation for details on automatic HAconfiguration.</description></property><property><name>dfs.client.failover.proxy.provider.myhadoop</name><value>node.ha.ConfiguredFa iloverProxyProvider</value><description>Configure the name of the Java class which will be usedby the DFS Client to determine which NameNode is the curr ent Active,and therefore which NameNode is currently serving client requests.这个类是Client的访问代理，是HA特性对于Client透明的关键！</description></property><property><name>dfs.ha.fencing.methods</name><value>sshfence</value><description>how to communicate in the switch process</descri ption></property><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/home/yarn/.ssh/id_rsa</value><description>the location stored ssh key</description> </property><property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>1000</value></property><property><name>dfs.journalnode.edits.dir</name><value>/home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/journal/</valu e></property><property><name>node.shared.edits.dir</name><value>qjournal://master:8485;slave1:8485;slave2:8485/hadoop-journal</value><description>A directory on shared storage between the multip lenamenodesin an HA cluster. This directory will be written by the a ctive and readby the standby in order to keep the namespaces synchroniz ed. Thisdirectorydoes not need to be listed in node.edits.dir abov e. It should beleft empty in a non-HA cluster.</description></property><!-- NN HA related configuration **END** --><!-- NameNode related configuration **BEGIN** --><property><name>.dir</name><value>file:///home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/name</v alue><description>Path on the local filesystem where the NameNode storesthe namespace and transactions logs persistently.If this is acomma-delimited list of directories then the name table i s replicatedin all of the directories, for redundancy.</description> </property><property><name>dfs.blocksize</name><value>1048576</value><description>HDFS blocksize of 128MB for large file-systems.Minimum block size is 1048576.</description></property><property><name>node.handler.count</name><value>10</value><description>More NameNode server threads to handle RPCs from largenumber of DataNodes.</description></property><!-- <property> <name>node.hosts</name> <value>master</va lue> <description>Ifnecessary, use this to control the list of allowable datanode s.</description></property> <property> <name>node.hosts.exclude</name > <value>slave1,slave2,slave3</value><description>If necessary, use this to control the list of ex clude datanodes.</description></property> --><!-- NameNode related configuration **END** --><!-- DataNode related configuration **BEGIN** --><property><name>dfs.datanode.data.dir</name><value>file:///home/yarn/Hadoop/hadoop-2.4.1/hdfs_dir/data</v alue><description>Comma separated list of paths on the local files ystem ofa DataNode where it should store its blocks.If this is a comma-delimited list of directories, then data will be st ored in allnamed directories, typically on different devices.</descr iption></property><!-- DataNode related configuration **END** --></configuration>5.yarn-site.xml<?xml version="1.0"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); youmay not use this file except in compliance with the License. You may obtaina copy of the License at /licenses/LICENSE-2 .0 Unlessrequired by applicable law or agreed to in writing, software dist ributedunder the License is distributed on an "AS IS" BASIS, WITHOUT WAR RANTIESOR CONDITIONS OF ANY KIND, either express or implied. See the Lic ense forthe specific language governing permissions and limitations under the License.See accompanying LICENSE file. --><configuration><!-- ResourceManager and NodeManager related configuration ***BEG IN*** --><property><name>yarn.acl.enable</name><value>false</value><description>Enable ACLs? Defaults to false.</description> </property><property><name>yarn.admin.acl</name><value>*</value><description>ACL to set admins on the cluster. ACLs are of for comma-separ ated-usersspace comma-separated-groups.Defaults to special value of * which means anyone. Special va lue of just space means no one has access.</description></property><property><name>yarn.log-aggregation-enable</name><value>false</value><description>Configuration to enable or disable log aggregati on</description></property><!-- ResourceManager and NodeManager related configuration ***END *** --><!-- ResourceManager related configuration ***BEGIN*** --><property><name>yarn.resourcemanager.hostname</name><value>master</value><description>The hostname of the RM.</description> </property><property><name>yarn.resourcemanager.webapp.https.address</name><value>${yarn.resourcemanager.hostname}:8090</value><description>The https adddress of the RM web application.</d escription></property><property><name>yarn.resourcemanager.address</name><value>${yarn.resourcemanager.hostname}:8032</value><description>ResourceManager host:port for clients to submit jobs.</description></property><property><name>yarn.resourcemanager.scheduler.address</name><value>${yarn.resourcemanager.hostname}:8030</value><description>ResourceManager host:port for ApplicationMasters to talk to Scheduler to obtain resources.</description> </property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value><description>ResourceManager host:port for NodeManagers.</des cription></property><property><name>yarn.resourcemanager.admin.address</name><value>${yarn.resourcemanager.hostname}:8033</value><description>ResourceManager host:port for administrative com mands.</description></property><property><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value><description>ResourceManager web-ui host:port.</description> </property><property><name>yarn.resourcemanager.scheduler.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.schedule r.fair.FairScheduler</value><description>ResourceManager Scheduler class.CapacityScheduler (recommended), FairScheduler (also recommen ded), or FifoScheduler</description></property><property><name>yarn.scheduler.minimum-allocation-mb</name><value>512</value><description>Minimum limit of memory to allocate to each container request at the Resource Manager.In MBs</description></property><property><name>yarn.scheduler.maximum-allocation-mb</name><value>2048</value><description>Maximum limit of memory to allocate to each container request at the Resource Manager.In MBs.According to myconfiguration,yarn.scheduler.maximum-allocation-mb > yarn.nodemanager .resource.memory-mb</description></property><!--<property><name>yarn.resourcemanager.nodes.include-path</name><value></value><description>List of permitted NodeManagers.If necessary, use this to control the list of allowable NodeM anagers.</description></property><property><name>yarn.resourcemanager.nodes.exclude-path</name><value></value><description>List of exclude NodeManagers.If necessary, use this to control the list of exclude NodeMan agers.</description></property>--><!-- ResourceManager related configuration ***END*** --><!-- NodeManager related configuration ***BEGIN*** --><property><name>yarn.nodemanager.resource.memory-mb</name><value>1024</value><description>Resource i.e. available physical memory, in MB, for given Nod eManager.Defines total available resources on the NodeManager to be ma de available to running containers.</description></property><property><name>yarn.nodemanager.vmem-pmem-ratio</name><value>2.1</value><description>Ratio between virtual memory to physical memory when setting memory limits for containers.Container allocations are expressed in terms of physical memo ry,and virtual memory usage is allowed to exceed this allocation by this ratio.</description></property><property><name>yarn.nodemanager.local-dirs</name><value>/home/yarn/Hadoop/hadoop-2.4.1/yarn_dir/local</value><description>Comma-separated list of paths on the local filesystem where i ntermediate data is written.Multiple paths help spread disk i/o.</description></property><property><name>yarn.nodemanager.log-dirs</name><value>/home/yarn/Hadoop/hadoop-2.4.1/yarn_dir/log</value><description>Comma-separated list of paths on the local filesystem where l ogs are written.Multiple paths help spread disk i/o.</description></property><property><name>yarn.nodemanager.log.retain-seconds</name><value>10800</value><description>Default time (in seconds) to retain log files on the NodeMana ger.***Only applicable if log-aggregation is disabled.</description></property><property><name>yarn.nodemanager.remote-app-log-dir</name><value>/yarn/log-aggregation</value><description>HDFS directory where the application logs are moved on applic ation completion.Need to set appropriate permissions.***Only applicable if log-aggregation is enabled.</description></property><property><name>yarn.nodemanager.remote-app-log-dir-suffix</name><value>logs</value><description>Suffix appended to the remote log dir.Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam}.***Only applicable if log-aggregation is enabled.</description></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value><description>Shuffle service that needs to be set for Map Red uce applications.</description></property><property><name>yarn.nodemanager.resource.cpu-vcores</name><value>1</value><description>Number of CPU cores that can be allocated for co ntainers.</description></property><!-- NodeManager related configuration ***END*** --><!-- History Server related configuration ***BEGIN*** --><property><name>yarn.log-aggregation.retain-seconds</name><value>-1</value><description>How long to keep aggregation logs before deleting them.-1 disables.Be careful, set this too small and you will spam the name nod e.</description></property><property><name>yarn.log-aggregation.retain-check-interval-seconds</nam e><value>-1</value><description>Time between checks for aggregated log retention.If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time.Be careful, set this too small and you will spam the name nod e.</description></property><!-- History Server related configuration ***END*** --><property><name>yarn.scheduler.fair.allocation.file</name><value>${yarn.home.dir}/etc/hadoop/fairscheduler.xml</value><description>fairscheduler config file path</description><!-- 官网文档居然找不到该属性！但该属性还是work的！ --> </property></configuration>6.创建fairscheduler.xml<?xml version="1.0"?><allocations><!--<queue name="hadooptest"><minResources>1024 mb, 1 vcores</minResources><maxResources>2048 mb, 2 vcores</maxResources><maxRunningApps>10</maxRunningApps><weight>2.0</weight><schedulingMode>fair</schedulingMode><aclAdministerApps> hadooptest</aclAdministerApps><aclSubmitApps> hadooptest</aclSubmitApps></queue><queue name="hadoopdev"><minResources>1024 mb, 2 vcores</minResources><maxResources>2048 mb, 4 vcores</maxResources><maxRunningApps>20</maxRunningApps><weight>2.0</weight><schedulingMode>fair</schedulingMode><aclAdministerApps> hadoopdev</aclAdministerApps><aclSubmitApps> hadoopdev</aclSubmitApps></queue>--><user name="yarn"><maxRunningApps>30</maxRunningApps></user></allocations>7.mapred-site.xml<?xml version="1.0"?><?xml-stylesheet type="text/xsl"href="configuration.xsl"?><!-- Licensed under the Apache License, Version 2.0 (the "License"); youmay not use this file except in compliance with the License. You may obtaina copy of the License at /licenses/LICENSE-2 .0 Unlessrequired by applicable law or agreed to in writing, software dist ributedunder the License is distributed on an "AS IS" BASIS, WITHOUT WAR RANTIESOR CONDITIONS OF ANY KIND, either express or implied. See the Lic ense forthe specific language governing permissions and limitations under the License.See accompanying LICENSE file. --><!-- Put site-specific property overrides in this file. --><configuration><!-- MapReduce Applications related configuration ***BEGIN*** --><name></name><value>yarn</value><description>Execution framework set to Hadoop YARN.</descrip tion></property><property><name>mapreduce.map.memory.mb</name><value>1024</value><description>Larger resource limit for maps.</description> </property><property><name>mapreduce.map.java.opts</name><value>-Xmx1024M</value><description>Larger heap-size for child jvms of maps.</descri ption></property><property><name>mapreduce.reduce.memory.mb</name><value>1024</value><description>Larger resource limit for reduces.</description> </property><property><name>mapreduce.reduce.java.opts</name><value>-Xmx1024M</value><description>Larger heap-size for child jvms of reduces.</des cription></property><property><name>mapreduce.task.io.sort.mb</name><value>1024</value><description>Higher memory-limit while sorting data for effic iency.</description></property><property><name>mapreduce.task.io.sort.factor</name><value>10</value><description>More streams merged at once while sorting files. </description></property><name>mapreduce.reduce.shuffle.parallelcopies</name><value>20</value><description>Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.</description> </property><!-- MapReduce Applications related configuration ***END*** --><!-- MapReduce JobHistory Server related configuration ***BEGIN** * --><property><name>mapreduce.jobhistory.address</name><value>slave1:10020</value><description>MapReduce JobHistory Server host:port. Defaul t port is 10020.</description></property><property><name>mapreduce.jobhistory.webapp.address</name><value>slave1:19888</value><description>MapReduce JobHistory Server Web UI host:port. De fault port is 19888.</description></property><property><name>mapreduce.jobhistory.intermediate-done-dir</name><value>/home/yarn/Hadoop/hadoop-2.4.1/mr_history/tmp</value><description>Directory where history files are written by Map Reduce jobs.</description></property><property><name>mapreduce.jobhistory.done-dir</name><value>/home/yarn/Hadoop/hadoop-2.4.1/mr_history/done</value><description>Directory where history files are managed by the MR JobHistory Server.</description></property><!-- MapReduce JobHistory Server related configuration ***END*** --></configuration>8.slaves1.scp相关目录到各台机器2.修改各台机器环境变量，添加新的HADOOP_HOME，#掉老的HADOOP_HOME9.启动集群（1）启动ZK在所有的ZK节点执行命令：zkServer.sh start查看各个ZK的从属关系：yarn@master:~$ zkServer.sh statusJMX enabled by defaultUsing config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: followeryarn@slave1:~$ zkServer.sh statusJMX enabled by defaultUsing config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfgMode: followeryarn@slave2:~$ zkServer.sh statusJMX enabled by defaultUsing config: /home/yarn/Zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg Mode: leader注意：哪个ZK节点会成为leader是随机的，第一次实验时slave2成为了leader，第二次实验时slave1成为了leader！此时，在各个节点都可以查看到ZK进程：yarn@master:~$ jps3084 QuorumPeerMain3212 Jps（2）格式化ZK（仅第一次需要做）任意ZK节点上执行：hdfs zkfc -formatZK（3）启动ZKFCZookeeperFailoverController是用来监控NN状态，协助实现主备NN切换的，所以仅仅在主备NN节点上启动就行：hadoop-daemon.sh start zkfc启动后我们可以看到ZKFC进程：yarn@master:~$ jps3084 QuorumPeerMain3292 Jps3247 DFSZKFailoverController（4）启动用于主备NN之间同步元数据信息的共享存储系统JournalNode参见角色分配表，在各个JN节点上启动：hadoop-daemon.sh start journalnode启动后在各个JN节点都可以看到JournalNode进程：yarn@master:~$ jps3084 QuorumPeerMain3358 Jps3325 JournalNode3247 DFSZKFailoverController（5）格式化并启动主NN格式化：hdfs namenode -format注意：只有第一次启动系统时需格式化，请勿重复格式化！在主NN节点执行命令启动NN:hadoop-daemon.sh start namenode启动后可以看到NN进程：yarn@master:~$ jps3084 QuorumPeerMain3480 Jps3325 JournalNode3411 NameNode3247 DFSZKFailoverController（6）在备NN上同步主NN的元数据信息hdfs namenode -bootstrapStandby以下是正常执行时的最后部分日志：Re-format filesystem in Storage Directory /home/yarn/Hadoop/hdfs2.0/name ? (Y or N) Y14/06/15 10:09:08 INFO common.Storage: Storage directory/home/yarn/Hadoop/hdfs2.0/name has been successfully formatted.14/06/15 10:09:09 INFO namenode.TransferFsImage: Opening connection to http://master:50070/getimage?getimage=1&txid=935&storageInfo=-47:5646363 72:0:CID-d899b10e-10c9-4851-b60d-3e158e322a6214/06/15 10:09:09 INFO namenode.TransferFsImage: Transfer took 0.11s at 63.64 KB/s14/06/15 10:09:09 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000935 size 7545 bytes.14/06/15 10:09:09 INFO util.ExitUtil: Exiting with status 014/06/15 10:09:09 INFO Node: SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NameNode at slave1/192.168.66.92************************************************************/（7）启动备NN在备NN上执行命令：hadoop-daemon.sh start namenode（8）设置主NN（这一步可以省略，这是在设置手动切换NN时的步骤，ZK已经自动选择一个节点作为主NN了）到目前为止，其实HDFS还不知道谁是主NN，可以通过监控页面查看，两个节点的NN 都是Standby状态。