\_ 4:0:0:9 sdao 66:128 [active][ready]
\_ 3:0:0:9 sdk 8:160 [active][ready]
360a9800044336b327a24446172587972 dm-2 NETAPP,LUN
[size=10G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:13 sdac 65:192 [active][ready]
\_ 4:0:1:13 sdbg 67:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 4:0:0:13 sdar 66:176 [active][ready]
\_ 3:0:0:13 sdn 8:208 [active][ready]
360a9800044336b327a24446172587868 dm-0 NETAPP,LUN
[size=200G][features=3 queue_if_no_path pg_init_retries 50][hwhandler=1 alua][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:10 sdaa 65:160 [active][ready]
\_ 4:0:1:10 sdbe 67:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 4:0:0:10 sdap 66:144 [active][ready]
\_ 3:0:0:10 sdl 8:176 [active][ready]
查询asmlib中asm磁盘，也都可以正常识别，此时排除了存储的故障，开始定位grid的日志
[root@rac2 bin]# /etc/init.d/multipathd restart
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]
[root@rac2 bin]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@rac2 bin]# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
[root@rac2 bin]# oracleasm listdisks
ASMLB_OCR01
ASMLB_OCR02
ASMLB_OCR03
ASMLB_ORCL01
ASMLB_ORCL02
ASMLB_ORCL03
ASMLB_ORCLCTL
ASMLB_ORCLFRA
ASMLB_ORCLREDO
ASMLB_YYZF01
ASMLB_YYZF02
ASMLB_YYZF03
ASMLB_YYZFCTL
ASMLB_YYZFFRA
ASMLB_YYZREDO
重新启动crs后报，系统仍存在ASM进程，但asm无法mount磁盘,crsd依旧无法启动
alerrac2.log的日志如下
[/u01/app/11.2/product/crs_1/bin/orarootagent.bin(13382)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0
2014-05-21 13:18:09.087
[/u01/app/11.2/product/crs_1/bin/orarootagent.bin(13382)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0
2014-05-21 13:22:37.716
[ctssd(17298)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2014-05-21 13:38:09.677
[/u01/app/11.2/product/crs_1/bin/orarootagent.bin(13382)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0
2014-05-21 13:52:38.315
[ctssd(17298)]CRS-2409:The clock on host rac2 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.
2014-05-21 13:58:10.245
[/u01/app/11.2/product/crs_1/bin/orarootagent.bin(13382)]CRS-5018:(:CLSN00037:) Removed unused HAIP route: 169.254.95.0 / 255.255.255.0 / 0.0.0.0 / usb0
[gpnpd(8438)]CRS-2328:GPNPD started on node rac2.
2014-05-21 14:14:29.618
[cssd(8520)]CRS-1713:CSSD daemon is started in clustered mode
2014-05-21 14:14:31.419
[ohasd(8217)]CRS-2767:Resource state recovery not attempted for 'ora.diskmon' as its target state is OFFLINE
2014-05-21 14:14:50.409
[cssd(8520)]CRS-1707:Lease acquisition for node rac2 number 2 completed
2014-05-21 14:14:51.693
[cssd(8520)]CRS-1621:The IPMI configuration data for this node stored in the Oracle registry is incomplete; details at (:CSSNK00002:) in /u01/a