RAC上是vmware vspere的虚拟机上有两个OEL6.3的虚拟机，上面跑的库是11.2.0.4
发现节点2挂掉了
1.检查节点1的alert
[root@racnode1 racnode1]# pwd
/u01/apps/grid/gridhome/11.2.0/grid/log/racnode1
[root@racnode1 racnode1]# tail -1000 alertracnode1.log
2014-04-15 09:41:20.815:
[crsd(27311)]CRS-2765:Resource 'ora.net1.network' has failed on server 'racnode1'.
2014-04-15 09:41:42.760:
[cssd(26972)]CRS-1612:Network communication with node racnode2 (2) missing for 50% of timeout interval. Removal of this node from cluster in 15.000 seconds
2014-04-15 09:41:50.763:
[cssd(26972)]CRS-1611:Network communication with node racnode2 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.000 seconds
2014-04-15 09:41:54.764:
[cssd(26972)]CRS-1610:Network communication with node racnode2 (2) missing for 90% of timeout interval. Removal of this node from cluster in 3.000 seconds
2014-04-15 09:41:57.766:
[cssd(26972)]CRS-1607:Node racnode2 is being evicted in cluster incarnation 291818318; details at (:CSSNM00007:) in /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log.
2014-04-15 09:42:06.052:
[cssd(26972)]CRS-1625:Node racnode2, number 2, was manually shut down
2014-04-15 09:42:06.059:
[cssd(26972)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racnode1 .
2014-04-15 09:42:06.950:
[crsd(27311)]CRS-5504:Node down event reported for node 'racnode2'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'Generic'.
2014-04-15 09:42:24.882:
[crsd(27311)]CRS-2773:Server 'racnode2' has been removed from pool 'ora.pera'.
[root@racnode1 racnode1]#
节点2被驱逐时间：2014-04-15 09:41:57
2.查看ocssd.log
# more /u01/apps/grid/gridhome/11.2.0/grid/log/racnode1/cssd/ocssd.log |grep "2014-04-15 09:41"
2014-04-15 09:41:23.620: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:23.621: [ CSSD][906479360]clssnmSendingThread: sent 4 status msgs to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:28.758: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.145: [GIPCHGEN][919283456] gipchaInterfaceFail: marking interface failing 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 1, numFail 0, idxBoot 0, flags 0x184d }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sending status msg to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceFail: marking interface failing 0x7fc024040a20 { host 'racnode2', haName 'CSS_racnode-cluster', local 0x7fc01821d4c0, ip '172.168.1.12:60678', subnet '172.168.1.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 4, flags 0x6 }
2014-04-15 09:41:33.760: [ CSSD][906479360]clssnmSendingThread: sent 5 status msgs to all nodes
2014-04-15 09:41:33.760: [GIPCHGEN][920860416] gipchaInterfaceDisable: disabling interface 0x7fc01821d4c0 { host '', haName 'CSS_racnode-cluster', local (nil), ip '172.168.1.11:52955', subnet '172.168.1.0', mask '255.255.255.0', mac '00-50-56-a1-7b-e2', ifname 'eth1', numRef 0, numFail 1, idxBoot 0, flags 0x19cd }
2014-04-15 09:41:33.760: [GI