ocfs2 oracle集群文件系统常见问题解决方案

Node 0：centos238

Ip：192.168.192.238

Node 1：xen241

Ip：192.168.192.241

Node 2：xen242

Ip：192.168.192.242

Problem 1、

Discription：

[root@xen241 init.d]# service o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on boot. The current values will be shown in brackets ('[]'). Hitting

without typing an answer will keep that current value. Ctrl-C will abort.

Load O2CB driver on boot (y/n) [n]: n

Cluster stack backing O2CB [o2cb]: o2cb

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2

Specify heartbeat dead threshold (>=7) [301]: 301

Specify network idle timeout in ms (>=5000) [30000]: 30000

Specify network keepalive delay in ms (>=1000) [2000]: 2000

Specify network reconnect delay in ms (>=2000) [2000]: 2000

Writing O2CB configuration: OK

Stopping O2CB cluster ocfs2: OK

Unloading module "ocfs2": OK

Unmounting ocfs2_dlmfs filesystem: OK

Unloading module "ocfs2_dlmfs": OK

Unmounting configfs filesystem: OK

Unloading module "configfs": OK

[root@xen241 init.d]# mount.ocfs2 /dev/sdc1 /mnt/ocfs2/

mount.ocfs2: Unable to access cluster service while trying initialize cluster

这是由于刚刚配置了o2cb，需要重启一下ocfs2：

[root@xen241 init.d]# service o2cb stop

ls: /config: No such file or directory

[root@xen241 init.d]# service o2cb start

[root@xen241 init.d]# service o2cb load

Loading filesystem "configfs": OK

Mounting configfs filesystem at /sys/kernel/config: OK

Loading filesystem "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

[root@xen241 init.d]# service o2cb online ocfs2

Starting O2CB cluster ocfs2: OK

Problem 2、

Discription：所有节点都成功挂载文件系统，挂载次序为node2→node1→node0。当在node0上更改文件后，在node1上马上能够看得到更新的文件内容，但是在node2上，看不到更新的内容。

Problem 3、

Discription：接problem2，当将node2上磁盘解挂（即umount）时，node0自动重启了！！等node0重启后，成功启动o2cb，成功挂载了文件系统。

Problem 4、

Discription：有三个节点，前连个节点成功挂载文件系统，当在第三个节点上挂在时出现如下问题：

o2net: connected to node xen238 (num 0) at 192.168.192.238:7777

o2net: connected to node xen241 (num 1) at 192.168.192.241:7777

ocfs2_dlm: Nodes in domain ("0C5DABF82BC34FAE9290742F68B411C9"): 0 1 2

(mount.ocfs2,18846,0):ocfs2_find_slot:243 ERROR: no free slots available!

(mount.ocfs2,18846,0):ocfs2_mount_volume:1461 ERROR: status = -22

ocfs2: Unmounting device (8,17) on (node 2)

o2net: no longer connected to node xen238 (num 0) at 192.168.192.238:7777

o2net: no longer connected to node xen241 (num 1) at 192.168.192.241:7777

分析：

Slots 在此是节点的意思？

节点个数不足？

节点0 1 2 已经在域中？

解决方案：

这是由于节点个数不够，参看#man mkfs.ocfs2得知。

使用tunefs.ocfs2增加节点个数：

#tunefs.ocfs2 –N 4 /dev/sda1

注：这里的4是由于原来设定的三个节点不够，所以增加1，设定为4。

Problem 5、

Discription：当集群有多个节点时，能够成功挂载第一个节点的文件系统，但是在挂载第二个节点时出现错误，提示如下：

(4801,2):o2net_connect_expired:1583ERROR: no connection established with node 0 after 30.0 seconds, giving up and returning errors.

(5348,2):dlm_request_join:1019 ERROR: status = -107

(5348,2):dlm_try_to_join_domain:1194 ERROR: status = -107

(5348,2):dlm_join_domain:1467 ERROR: status = -107

(5348,2):dlm_register_domain:1702 ERROR: status = -107

(5348,2):ocfs2_dlm_init:2547 ERROR: status = -107

(5348,2):ocfs2_mount_volume:1253 ERROR: status = -107

ocfs2: Unmounting device (8,129) on (node 2)

分析：这里是由于防火墙的原因，关闭防火墙就能成功挂载了

解决方案：

1、关闭防火墙

马上生效：

#service iptables stop 停用防火墙

#service iptables start 启用防火墙

2、为ocfs2打开需要的端口（推荐）

在/etc/sysconfig/iptables里加入如下两行：（注意添加的位置）

-A INPUT -p tcp -m state --state NEW -m tcp --dport 7777 -j ACCEPT

-A OUTPUT -p tcp -m state --state NEW -m tcp --dport 7777 -j ACCEPT

Problem 6、

Discription：昨天晚上三个节点正常挂载上后，未做任何改变，上午也没出错，下午14：40来了发现节点0上满屏被刷成如下的错误，关机也不行，且提示不能卸载ocfs2文件系统。最后一直按着电源，强行关机！

end_request:I/O error, dev sda, sector 1855

(ksoftirqd/1,6,1):o2hb_bio_end_io:237 ERROR: IO Error -5

(o2hb-0C5DABFB2B,5734,1):o2hb_do_disk_heartbeat:768 ERROR: status = -5

分析：

开机后iscsi盘都未能自动挂载！手动挂载→ip通

问题解决：

Node0 的网线掉了。插上网线，登录iscsi设备，挂载ocfs2。没问题了！！

Problem 7、

Discription：改变卷标，使用tunefs.ocfs2 –L “label name” device 的命令来改变卷标，给出如下错误提示：

[root@xen241 ~]#tunefs.ocfs2 –L “forxen2” /dev/sda2

Tunefs.ocfs2: Trylock failed while opening device “/dev/sda2”

分析：

由lock想到是否要只能在一个卷上更改，于是，毅然将这个分区在其他节点上卸载了

(umount)。但是还是出现在样的问题，难道？？？

于是在将所有节点上的此分区都卸载，再更改，成功！

解决：

卷标不能在线更改，必须将这个分区所有挂载都umount掉才能更改！

Problem 8、

Discription：node1、2均已挂载，node0挂载出现如下问题：

[root@xen238 ~]# mount.ocfs2 /dev/sda1 /mnt/ocfs2-1

mount.ocfs2: Transport endpoint is not connected while mounting /dev/sda1 on /mnt/ocfs2-1. Check 'dmesg' for more information on this error.

[root@xen238 ~]# dmesg

(o2net,30122,0):o2net_connect_expired:1664 ERROR: no connection established with node 1 after 30.0 seconds, giving up and returning errors.

(o2net,30122,0):o2net_connect_expired:1664 ERROR: no connection established with node 2 after 30.0 seconds, giving up and returning errors.

(mount.ocfs2,31014,0):dlm_request_join:1036 ERROR: status = -107

(mount.ocfs2,31014,0):dlm_try_to_join_domain:1210 ERROR: status = -107

(mount.ocfs2,31014,0):dlm_join_domain:1488 ERROR: status = -107

(mount.ocfs2,31014,0):dlm_register_domain:1754 ERROR: status = -107

(mount.ocfs2,31014,0):ocfs2_dlm_init:2808 ERROR: status = -107

(mount.ocfs2,31014,0):ocfs2_mount_volume:1447 ERROR: status = -107

ocfs2: Unmounting device (8,1) on (node 0)

Problem 9、

Discription：node0、node2均已挂载成功两个分区，node1也开机自动挂载1个分区，但是令一个分区却不能自动挂载，手动挂在时提示如下错误：

[root@xen241 ISO]# mount /dev/sdb2 /mnt/ocfs2-2

mount.ocfs2: Unknown code B 0 while mounting /dev/sdb2 on /mnt/ocfs2-2. Check 'dmesg' for more information on this error.

[root@xen241 ISO]# dmesg

(mount.ocfs2,4420,0):dlm_join_domain:1468 Timed out joining dlm domain 8B0DD282CCE54ADBBD556D42D88B11ED after 91200 msecs

ocfs2: Unmounting device (8,18) on (node 1)

解决方案：

将不能挂载的分区在各个节点上卸载后重新挂载一次，解决了！

Problem 10、

Discription：在IBM服务器机群上安装ocfs2，两个节点，安装后启动时出现如下错误：[root@cnode036 ocfs2]# service o2cb online

Starting O2CB cluster ocfs2: Failed

Cluster ocfs2 created

Node cnode025 added

o2cb_ctl: Internal logic failure while adding node cnode036

Stopping O2CB cluster ocfs2: OK

[root@cnode036 ocfs2]#

仔细查看cluster.conf文件发现两个节点的number都是0。

解决方法：

当然是将后一个number改成1了。OK！

Problem 11、（没解决呢）

Discription：两个不同网段的电脑想要挂载同一块存储。

目前想到两个办法：

方法1（目前觉得行不通，继续测试中）：

创建两个不同的集群，但是若使用了相同的集群name，会出现冲突问题，因为在你还没挂载分区前，使用mounted.ocfs2 –f命令查看到的是另一个集群上的节点的挂载情况。

所以想到是使用不同的集群名字。但是遇到了另一个问题（problem 12）。解决这个problem12之后，使用mounted.ocfs2 –f查看到的还是另一个集群上节点的挂载情况，因为这个新的集群根本就还没有开始挂载存储的！

方法2：公网、私网IP

方法3：木头在QQ上说的防火墙

查看NAT映射？？？？

Problem 12、

Discription：为集群改另一个名字（非默认名），出现如下错误：

[root@cnode036 iscsi]# service o2cb configure

Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

without typing an answer will keep that current value. Ctrl-C

will abort.

Load O2CB driver on boot (y/n) [n]: y

Cluster stack backing O2CB [o2cb]:

Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs22

Specify heartbeat dead threshold (>=7) [301]:

Specify network idle timeout in ms (>=5000) [30000]:

Specify network keepalive delay in ms (>=1000) [2000]:

Specify network reconnect delay in ms (>=2000) [2000]:

Writing O2CB configuration: OK

Starting O2CB cluster ocfs22: Failed

o2cb_ctl: Memory allocation failed while setting cluster name

Stopping O2CB cluster ocfs22: OK

特殊字体为新的集群名字。

为集群命名！

解决办法：

Before you change a cluster name, you missed one important step, you

need to offline the cluster. So the right steps is:

1. /etc/init.d/o2cb offline

2. change /etc/ocfs2/cluster.conf

3. run /etc/init.d/o2cb configure and change the name.

Now, since you have already changed the configuration file(step 2 and

3), there are 2 different ways to get the right cluster up. One is

"umount the ocfs2 volume and reboot the box.", this is simple.

The other is that you can just renamed the cluster name(in cluster.conf

and o2cb configure) back to the name in your /config/cluster and then do

the steps I described above(1,2,3).

注意：这里需要注意cluster.conf文件中cluster项中的name和所有node项中的cluster必须一致！

Problem 13、

Discription：在IBM集群上挂载ocfs2存储

Vendor: IBM Model: 2145 Rev: 0000

Type: Direct-Access ANSI SCSI revision: 06

SCSI device sdc: 1048576000 512-byte hdwr sectors (536871 MB)

sdc: Write Protect is off

sdc: Mode Sense: 97 00 10 08

SCSI device sdc: drive cache: write through w/ FUA

SCSI device sdc: 1048576000 512-byte hdwr sectors (536871 MB)

sdc: Write Protect is off

sdc: Mode Sense: 97 00 10 08

SCSI device sdc: drive cache: write through w/ FUA

sdc: sdc1 sdc2 sdc3 sdc4 < sdc5 >

sd 16:0:0:0: Attached scsi disk sdc

sd 16:0:0:0: Attached scsi generic sg3 type 0

OCFS2 1.2.9 Mon May 19 11:42:54 PDT 2008 (build a693806cb619dd7f225004092b675ede)

(7553,0):ocfs2_initialize_super:1455 ERROR: couldn't mount because of unsupported optional features (50).

(7553,0):ocfs2_fill_super:578 ERROR: status = -22

ocfs2: Unmounting device (8,37) on (node 255)

对于第二个错，需要在格式化文件系统的时候添加其他的参数，所以这里重新执行格式化就行了。

[root@cnode025 sbin]# mkfs.ocfs2 -b 4K -C 32K -L forxen3 –N 3 /dev/sdc1 --fs-feature-level=max-compat

注意后面特殊字体部分，格式化文件系统完成后便可以挂载了！这里是因为安装的ocfs2的版本较低，需要使用这个参数。查看man mkfs.ocfs2 得到：

--fs-feature-level=feature-level

Choose from a set of pre-determined file-system features.

This option is designed to allow users to conveniently

choose a set of file system features which fits their

needs. There is no downside to trying a set of features

which your module might not support - if it won鈚mount the

new file system simply reformat at a lower level. Feature

levels can be fine-tuned via the --fs-features option. Cur-

rently, there are 3 types of feature levels:

max-compat

Chooses fewer features but ensures that the

file system can be mounted from older versions

of the OCFS2 module.