三、分析CRS状态：
由于故障期间客户很好的保留了现场环境，因此我们得以现场分析到CRS
状态。
Name Type Target State Host
------------------------------------------------------------
ora.xxxxdb.db application ONLINE ONLINE xxdb01np5
ora....sdb1.cs application ONLINE OFFLINE
ora....db1.srv application ONLINE ONLINE xxdb01np5
ora....db2.srv application ONLINE ONLINE xxxxdb02
ora....b1.inst application ONLINE ONLINE xxdb01np5
ora....b2.inst application ONLINE ONLINE xxxxdb02
ora....P5.lsnr application ONLINE ONLINE xxdb01np5
ora....np5.gsd application ONLINE ONLINE xxdb01np5
ora....np5.ons application ONLINE ONLINE xxdb01np5
ora....np5.vip application ONLINE ONLINE xxdb01np5
ora....P5.lsnr application ONLINE OFFLINE
ora....np5.gsd application ONLINE ONLINE xxxxdb02
ora....np5.ons application ONLINE ONLINE xxxxdb02
ora....np5.vip application ONLINE ONLINE xxdb01np5
― 说明：故障期间，节点二上的数据库（实例）资源仍然在运行，Nodeapp资源正常，
唯一被OFFLINE的是一个Service和listener 资源。但是我们注意到节点二上VIP 资源
已经被节点一接管过去。这说明，节点二的VIP（xxx.xxx.xxx.4 ）在排查问题的时候是
没问题的。
四、分析数据库告警日志
Sun Apr 29 03:41:13 BEIST 2012
ALTER SYSTEM SET service_names='xxxxdb'SCOPE=MEMORY SID='xxxxdb2';
Sun Apr 29 03:41:55 BEIST 2012
ALTER SYSTEM SET service_names=''SCOPE=MEMORY SID='xxxxdb2';
Sun Apr 29 03:41:55 BEIST 2012
Immediate Kill Session#: 1659, Serial#: 2
Immediate Kill Session: sess:70000038f773bb8 OS pid: 1180972
Sun Apr 29 03:41:55 BEIST 2012
Process OS id : 1180972 alive after kill
Errors in file
Immediate Kill Session#: 1660, Serial#: 2
Immediate Kill Session: sess:7000003847623c0 OS pid: 1179690
Sun Apr 29 03:41:56 BEIST 2012
Process OS id : 1179690 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1661, Serial#: 2
Immediate Kill Session: sess:7000003837620c0 OS pid: 1300264
Sun Apr 29 03:41:56 BEIST 2012
Process OS id : 1300264 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1662, Serial#: 2
Immediate Kill Session: sess:70000038275bc80 OS pid: 1155624
Sun Apr 29 03:41:57 BEIST 2012
Process OS id : 1155624 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1663, Serial#: 2
Immediate Kill Session: sess:700000387762ed0 OS pid: 1000484
Sun Apr 29 03:41:57 BEIST 2012
Process OS id : 1000484 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1664, Serial#: 2
Immediate Kill Session: sess:7000003867923d8 OS pid: 1175586
Sun Apr 29 03:41:58 BEIST 2012
Process OS id : 1175586 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1665, Serial#: 2
Immediate Kill Session: sess:7000003857a9d30 OS pid: 1296160
Sun Apr 29 03:41:58 BEIST 2012
Process OS id : 1296160 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1666, Serial#: 3
Immediate Kill Session: sess:70000038f775130 OS pid: 1176862
Sun Apr 29 03:41:59 BEIST 2012
Process OS id : 1176862 alive after kill
Errors in file/oracle/product/10.2.0/admin/xxxxdb/udump/xxxxdb2_ora_110596.trc
Immediate Kill Session#: 1667, Serial#: 2
Immediate Kill Session: s