2011-11-23 18:22:22.518: [ CSSD][1106696512]clssnmvDHBValidateNCopy: node 1, dtydb1, has a disk HB, but no network HB, DHB has rcfg 216519746, wrtcnt, 1004982, LATS 1030717754, lastSeqNo 1004980, uniqueness 1321449141, timestamp 1322043742/933689044

2011-11-23 18:22:23.520: [ CSSD][1106696512]clssnmvDHBValidateNCopy: node 1, dtydb1, has a disk HB, but no network HB, DHB has rcfg 216519746, wrtcnt, 1004984, LATS 1030718754, lastSeqNo 1004982, uniqueness 1321449141, timestamp 1322043743/933690044

2011-11-23 18:22:24.140: [ CSSD][1113516352]clssnmSendingThread: sending status msg to all nodes

2011-11-23 18:22:24.141: [ CSSD][1113516352]clssnmSendingThread: sent 4 status msgs to all nodes

2011-11-23 18:22:24.523: [ CSSD][1106696512]clssnmvDHBValidateNCopy: node 1, dtydb1, has a disk HB, but no network HB, DHB has rcfg 216519746, wrtcnt, 1004986, LATS 1030719754, lastSeqNo 1004984, uniqueness 1321449141, timestamp 1322043744/933691044

2011-11-23 18:22:25.525: [ CSSD][1106696512]clssnmvDHBValidateNCopy: node 1, dtydb1, has a disk HB, but no network HB, DHB has rcfg 216519746, wrtcnt, 1004988, LATS 1030720754, lastSeqNo 1004986, uniqueness 1321449141, timestamp 1322043745/933692044

2011-11-23 18:22:26.527: [ CSSD][1106696512]clssnmvDHBValidateNCopy: node 1, dtydb1, has a disk HB, but no network HB, DHB has rcfg 216519746, wrtcnt, 1004990, LATS 1030721764, lastSeqNo 1004988, uniqueness 1321449141, timestamp 1322043746/933693044

经过部署监控脚本，ping日志

从18：21：56开始丢包（117-150包丢失）

64 bytes from 192.168.100.1: icmp_seq=114 ttl=64 time=0.342 ms

64 bytes from 192.168.100.1: icmp_seq=115 ttl=64 time=0.444 ms

64 bytes from 192.168.100.1: icmp_seq=116 ttl=64 time=0.153 ms

--- 192.168.100.1 ping statistics ---

150 packets transmitted, 116 received, 22% packet loss, time 149054ms

rtt min/avg/max/mdev = 0.084/0.246/0.485/0.099 ms

Wed Nov 23 18:22:31 CST 2011

继续分析

经过以上分析，原因基本确认为RAC节点私有网络丢包，导致一个节点主机重启；但为什么会丢包呢？在检查主机网络配置没有问题的情况下，只能请网络工程师协助解决了

网络专家通过网络抓包，发现如下现象

观察到几个现象，内容来自回复的邮件：

1. 4:02:09，192.168.100.1在e4cc这块网卡上发出的ping请求，192.168.100.2没有把回应包送到e4cc；

2. 192.168.100.2发出的ping请求数据包，没有送到192.168.100.1的e4cc这块网卡，但192.168.100.1主机肯定是收到了，因为在e4cc这块网卡上，看到了192.168.100.1给192.168.100.2的回应数据包；

3. 4:02:41，192.168.100.2的e474网卡向192.168.100.1回应了Destination unreachable (Port unreachable)，此时192.168.100.2可以正常回包，经过一段时间调整后，4:02:53起，网络恢复正常。

具体可以理解如下

1，已主机2的丢包为例，seq9-seq41丢包

64 bytes from 192.168.100.1: icmp_seq=7ttl=64 time=0.170 ms

64 bytes from 192.168.100.1: icmp_seq=8ttl=64 time=0.376 ms

64 bytes from 192.168.100.1: icmp_seq=42ttl=64 time=0.151 ms

64 bytes from 192.168.100.1: icmp_seq=43ttl=64 time=0.340 ms

2，主机2发出了seq9request