GPFS文件系统安装配置和测试

GPFS文件系统安装配置和测试目录GPFS文件系统安装配置和测试 (1)一、目的 (2)二、测试架构和环境 (2)三、系统安装配置 (3)1.安装前的准备： (3)2.系统和相关支持包的安装 (3)3..配置SSH 无密码登录 (3)1)编辑hosts和配置密钥 (3)2)关闭防火墙及selinux安全服务 (4)3)测试ssh无密码登录配置 (4)4.安装gpfs3.4安装包 (4)5.配置GPFS (5)1)配置集群 (5)2)配置共享盘 (6)3)编辑共享盘配置文件 (6)4)使用mmcrnsd创建NSD (7)5)授权节点license (7)6)启动gpfs 文件系统 (7)7)创建GPFS 文件系统 (8)四、优化GPFS 集群配置 (8)1.查看当前集群配置 (8)2.修改集群配置参数 (8)五、变更节点配置 (9)1.增加节点 (9)2.增加NSD (9)3.增加硬盘及NSD控制服务器 (9)4.增加硬盘到文件系统中 (9)六、测试步骤和结果 (10)测试环境： (10)具体测试过程： (10)从小文件读写测试结果总结： (11)使用大文件单个10G，进行测试： (11)大文件读写测试总结: (11)七、总结 (11)一、目的测试和验证gpfs并行文件系统功能和性能，为将来部署准备二、测试架构和环境架构图三、系统安装配置1.安装前的准备：在IBM 3512 划分3个lun,空间容量分别为：380G,88 G,30 G，映射到两台服务器确保服务器和存储连接正常确保网络连接正常2.系统和相关支持包的安装安装redhat enterprise server Linux 6.3 或6.5安装其它系统需要支持包：yum install libstdc++ compat-libstdc++-296 compat-libstdc++-33 libXp imake gcc-c++ kernel kernel-headers kernel-devel kernel-smp kernel-smp-devel xorg-x11-xauth3..配置SSH 无密码登录1)编辑hosts和配置密钥编辑hostS文件如下所示10.8.8.133 gpfs110.8.8.134 gpfs210.8.8.136 gpfs710.8.8.137 gpfs6登录到各个系统gpfs2,gpfs6,gpfs7上分别执行以下命令#ssh-keygen -t dsa在gpfs2上执行以下命令#cat ~/.ssh/id_dsa.pub > ~/.ssh/authorized_keys#ssh gpfs6 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys#ssh gpfs7 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys#scp ~/.ssh/authorized_keys gpfs6:~/.ssh/authorized_keys#scp ~/.ssh/authorized_keys gpfs7:~/.ssh/authorized_keys完成以后测试gpfs2,gpfs6,gpfs7连通性ssh gpfsx date注意因为版本不一致gpfs2(6.5)版本需要执行restorecon –R –v /root/.ssh2)关闭防火墙及selinux安全服务service iptables stopvi /etc/sysconfig/selinux# This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of these two values:# targeted - Targeted processes are protected,# mls - Multi Level Security protection.SELINUXTYPE=targeted3)测试ssh无密码登录配置登录gpfs1执行命令ssh gpfs6 datessh gpfs7 datessh gpfs3 date4.安装gpfs3.4安装包执行./gpfs_install-3.4.0-0_x86_64 –text-only进行安装Cd /usr/lpp/mmfsRpm –ivh *.rpmtar -zxcvf解压gpfs-3.4.0-6.x86_64.tar.gz生成升级包gpfs.basegpfs.msg.en_USgpfs.docsgpfs.gp安装所有的rpm软件包，并升级补丁Rpm –U *.rom检查GPFS软件包安装情况rpm -qa|grep gpfs增加.bash_profile PATH$PATH=$PATH:$HOME/bin:/usr/lpp/mmfs/bin编译软件cd /usr/lpp/mmfs/srcmake Autoconfigmake Worldmake InstallImages5.配置GPFS1)配置集群在gpfs7上创建集群建立文件/tmp/gpfs.nodes内容为：gpfs7:quorum-managergpfs6:quorum-manager执行命令：/usr/lpp/mmfs/bin/mmcrcluster -n /tmp/gpfs.nodes -p gpfs7 -s gpfs6 -r /usr/bin/ssh -R /usr/bin/scp显示集群信息mmlsclusterGPFS cluster information========================GPFS cluster name: dlokcluster.gpfs7GPFS cluster id: 722837121801769718GPFS UID domain: dlokcluster.gpfs7Remote shell command: /usr/bin/sshRemote file copy command: /usr/bin/scpGPFS cluster configuration servers:-----------------------------------Primary server: gpfs7Secondary server: gpfs6Node Daemon node name IP address Admin node name Designation-----------------------------------------------------------------------------------------------1 gpfs7 10.8.8.136 gpfs7 quorum-manager2 gpfs6 10.8.8.137 gpfs6 quorum-manager2)配置共享盘首先检查硬盘连接正常root@gpfs7 gpfs]# fdisk -lDisk /dev/sda: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0006222cDevice Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 LinuxPartition 1 does not end on cylinder boundary./dev/sda2 64 17770 142224384 8e Linux LVM Disk /dev/sdb: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0003851bDevice Boot Start End Blocks Id System Disk /dev/sdc: 146.2 GB, 146163105792 bytes255 heads, 63 sectors/track, 17769 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk identifier: 0x0004d5b33)编辑共享盘配置文件[root@gpfs7 gpfs]# vi disk_file/dev/sdh:gpfs7:gpfs6:dataAndMetadata::g1/dev/sdi:gpfs7:gpfs6:dataAndMetadata::g2/dev/sdq:gpfs7:gpfs6:dataAndMetadata::g34)使用mmcrnsd创建NSDmmcrnsd -F /tmp/gpfs_disk -v yes显示NSDMmlsnsd –m[root@gpfs6 home]# mmlsnsd -mDisk name NSD volume ID Device Node name Remarks---------------------------------------------------------------------------------------g1 0A0808885379C4F2 /dev/sdc gpfs6 server nodeg1 0A0808885379C4F2 /dev/sdh gpfs7 server nodeg2 0A0808885379C4F3 /dev/sdd gpfs6 server nodeg2 0A0808885379C4F3 /dev/sdi gpfs7 server nodeg3 0A0808885379C4F4 /dev/sdl gpfs6 server nodeg3 0A0808885379C4F4 /dev/sdq gpfs7 server node5)授权节点licensemmchlicense server --accept –N hpc1,hpc2,hpc36)启动gpfs 文件系统Mmstartup –a查看文件系统状态Mmgeststate –a[root@gpfs6 home]# mmgetstate -aNode number Node name GPFS state------------------------------------------1 gpfs7 active2 gpfs6 active3 gpfs2 active7)创建GPFS 文件系统# mmcrfs gpfsdir1 -F /home/GPFS/nsd -T /gpfsdir1 -A yes -K always -B 2m -E no -m 2 -M 2 -n 32 -Q no -r 2 -R 2 -S yes -v no注：➢其mount点为/oradata, blocksize为2m;注意此参数已经创建，不可修改，blocksize一般要参照存储端LUN设置，以获得最优性能；➢-m 2 -M 2表示写两份metadata数据，-r 2 -R 2表示写两份data数据；➢我们已经将数据盘的failgroup分别设为2、3，GPFS会自动将数据均衡写入不同failgroup中；四、优化GPFS 集群配置1.查看当前集群配置# mmlsconfig[root@gpfs6 home]# mmlsconfigConfiguration data for cluster dlokcluster.gpfs7:-------------------------------------------------myNodeConfigNumber 2clusterName dlokcluster.gpfs7clusterId 722837121801769718autoload nominReleaseLevel 3.4.0.7dmapiFileHandleSize 32pagepool 2048MadminMode centralFile systems in cluster dlokcluster.gpfs7:------------------------------------------/dev/g22.修改集群配置参数Mmchconfig pagepool=2048m五、变更节点配置1.增加节点Mmaddnode -N gpfs12.增加NSD首先编辑磁盘配置文件Vi /root/gpfs/disk_file2/dev/sdb:gpfs1::dataAndMetadata::nsd43.m增加硬盘及NSD控制服务器mmcrnsd -F /root/gpfs/disk_file2 -v no4.增加硬盘到文件系统中mmadddisk g2 -F /root/gpfs/disk_file2显示文件系统mmlsfs g2flag value description------------------- ------------------------ ------------------------------------f 32768 Minimum fragment size in bytes-i 512 Inode size in bytes-I 32768 Indirect block size in bytes-m 1 Default number of metadata replicas-M 2 Maximum number of metadata replicas-r 1 Default number of data replicas-R 2 Maximum number of data replicas -j cluster Block allocation type-D nfs4 File locking semantics in effect-k all ACL semantics in effect-n 32 Estimated number of nodes that will mount file system-B 1048576 Block size-Q none Quotas enforcednone Default quotas enabled --filesetdf No Fileset df enabled?-V 12.10 (3.4.0.7) File system version--create-time Wed May 21 15:21:30 2014 File system creation time-u Yes Support for large LUNs?-z No Is DMAPI enabled?-L 4194304 Logfile size-E Yes Exact mtime mount option-S No Suppress atime mount option-K whenpossible Strict replica allocation option--fastea Yes Fast external attributes enabled?--inode-limit 520192 Maximum number of inodes-P system Disk storage pools in file system -d g1;g2;g3;nsd4 Disks in file system-A yes Automatic mount option-o none Additional mount options-T /gpfsdir1 Default mount point--mount-priority 0 Mount priority六、测试步骤和结果测试环境：Gpfs3 ibm 3850 虚拟机，linux 6.5,本地SAS盘15k转，千兆络Gpfs6 Dell 2950 linux 6.3, 本地SAS盘15k转, 4G HBA 光纤卡Gpfs6 Dell r420 linux 6.3, 本地SAS盘15k转，4G HBA 光纤卡具体测试过程：使用相同资料（普通目录文件4G）：1.从gpfs2 本地盘拷贝资料到gpfs 盘，65M/s2从gpfs7 本地盘拷贝资料到gpfs 盘，161M/s3从gpfs6本地盘拷贝资料到gpfs 盘，142M/s4.从gpfs6 gpfs盘拷贝资料到gpfs 盘不同目录122M/s5. 从gpfs6，gpfs7,拷贝同时拷贝本地盘资料到GPFS 盘，gpfs672M/s,pgfs7 98M/s6登录gpfs7,拷贝本地SAS盘资料到本地SAS盘, 60M/s从小文件读写测试结果总结：本次测试环境GPFS客户端从本地到gpfs盘拷贝资料，受限千兆网络影响只能达到65M。