优化团队中一同事接到通知，北京一套RAC
数据库在7月1日早上9点多“宕掉了”，具体是所有的机器都宕掉了，还是其中的某一个节点宕掉了则不详，需要去排查才知。
经过对该RAC的相关的log和nmon进行分析，查明为“伪宕机”，真正原因为：第2节点对外服务的网络没有了流量，服务器被人为重启。下面为分析过程。
一、先把经过分析得出来的故障经过贴在这里，便于读者快速的了解此次过程的前因后果：
顺序
时间
现象描述
1
8:00:00
二号节点的Public网络流量突然降为0，但没断线，一直持续到二号节点关机时
2
9:39:59
二号节点服务器被关机，后经查，使用的命令是"init 0"
3
9:40:18
一号节点发现和二号节点的心跳出现了问题
4
9:40:32
将二节点从CRS中移除出去
5
9:40:32
重构CRS，CRS中只保留了第一节点
6
9:40:36
将原本属于二号节点上的两个VIP迁移并注册到了一号节点
7
9:45:13
二号节点服务器起动起来了
8
9:49:42
二节点CRS启动
9
9:50:24
CRS再次重构已经启动，将二节点加入CRS中
10
9:51:10
再次关闭二节点CRS
11
10:47:45
二节点CRS再次启动，并已经加入到CRS中来
二、详细分析过程
1、RAC基础信息
节点
主机名
public IP / VIP（eth2)
priv IP (eth3)
1
XXXXdb7
10.X.XX.80/10.X.XX.82
192.XX.XX.80
2
XXXXdb8
10.X.XX.81/10.X.XX.83/10.X.XX.84
192.XX.XX.81
2、查看第1节点（XXXXdb7）的日志信息
2.1、第1节点（XXXXdb7）的OS messages：
Jun 30 04:03:06 XXXXdb7 syslogd 1.4.1: restart.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Registering new address record for 10.X.XX.83 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Withdrawing address record for 10.X.XX.83 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Registering new address record for 10.X.XX.83 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Withdrawing address record for 10.X.XX.83 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Registering new address record for 10.X.XX.83 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Registering new address record for 10.X.XX.84 on eth2.
Jul 1 09:40:36 XXXXdb7 avahi-daemon[9246]: Withdrawing address record for 10.X.XX.84 on eth2.
分析：
（1）09:40:36，本属于第2节点的10.X.XX.83/10.X.XX.84两个VIP地址，被飘移到了第1节点的eth2网卡上，最少说明第2节点的CRS宕掉了
（2）无任何软、硬件错误日志
2.2、第1节点（XXXXdb7）CRS_alert log：
2013-07-01 09:40:18.629
[cssd(9742)]CRS-1612:Network communication with node XXXXdb8 (2) missing for 50% of timeout interval. Removal of this node from cluster in 14.290 seconds
2013-07-01 09:40:25.663
[cssd(9742)]CRS-1611:Network communication with node XXXXdb8 (2) missing for 75% of timeout interval. Removal of this node from cluster in 7.250 seconds
2013-07-01 09:40:30.673
[cssd(9742)]CRS-1610:Network communication with node XXXXdb8 (2) missing for 90% of timeout interval. Removal of this node from cluster in 2.240 seconds
2013-07-01 09:40:32.920
[cssd(9742)]CRS-1632:Node XXXXdb8 is being removed from the cluster in cluster incarnation 249439701
2013-07-01 09:40:32.940
[cssd(9742)]CRS-1601:CSSD Reconfiguration complete. Active nodes are XXXXdb7 .
2013-07-01 09:40:33.902
[crsd(10300)]CRS-5504:Node down event reported for node 'XXXXdb8'.
2013-07-01 09:40:44.683
[crsd(10300)]CRS-2773:Server 'XXXXdb8' has been removed from pool 'Generic'.
2013-07-01 09:40:44.684
[crsd(10300)]CRS-2773:Server 'XXXXdb8' has been removed from pool 'ora.XXXXc'.
2013-07-01 09:50:24.410
[cssd(9742)]CRS-1601:CSSD Reconfiguration complete. Active nodes are XXXXdb7 XXXXdb8.
2013-07-01 09:51:10.792
[cssd(9742)]CRS-1625:Node XXXXdb8, number 2, was manually shut down
2013-07-01 09:51:10.831
[cssd(9742)]CRS-1601:CSSD Reconfiguration complete. Active nodes are XXXXdb7 .
2013-07-01 10:47:45.660
[cssd(9742)]CRS-1601:C