2013-07-16 11:30:37.400

[/u01/app/11.2.0.3/grid/bin/oraagent.bin(3981)]CRS-5016:Process "/u01/app/11.2.0.3/grid/bin/lsnrctl" spawned by agent "/u01/app/11.2.0.3/grid/bin/oraagent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0.3/grid/log/rac2/agent/crsd/oraagent_grid/oraagent_grid.log"

2013-07-16 11:30:37.413

[cssd(3335)]CRS-1654:Clean up of CRSD resources finished successfully.

2013-07-16 11:30:37.415

[cssd(3335)]CRS-1655:CSSD on node rac2 detected a problem and started to shutdown.

2013-07-16 11:30:38.049

[cssd(3335)]CRS-1660:The CSS daemon shutdown has completed

2013-07-16 11:30:38.001

[/u01/app/11.2.0.3/grid/bin/oraagent.bin(3981)]CRS-5822:Agent '/u01/app/11.2.0.3/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:2:6} in /u01/app/11.2.0.3/grid/log/rac2/agent/crsd/oraagent_grid/oraagent_grid.log.

2013-07-16 11:30:38.002

[/u01/app/11.2.0.3/grid/bin/oraagent.bin(8579)]CRS-5822:Agent '/u01/app/11.2.0.3/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:8:1180} in /u01/app/11.2.0.3/grid/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log.

2013-07-16 11:30:40.781

[/u01/app/11.2.0.3/grid/bin/orarootagent.bin(3987)]CRS-5822:Agent '/u01/app/11.2.0.3/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:1:14064} in /u01/app/11.2.0.3/grid/log/rac2/agent/crsd/orarootagent_root/orarootagent_root.log.

2013-07-16 11:30:46.076

[ohasd(2976)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac2'.

2013-07-16 11:30:51.157

[ohasd(2976)]CRS-2765:Resource 'ora.ctssd' has failed on server 'rac2'.

2013-07-16 11:30:51.182

[ohasd(2976)]CRS-2765:Resource 'ora.evmd' has failed on server 'rac2'.

2013-07-16 11:30:51.658

[ohasd(2976)]CRS-2765:Resource 'ora.cluster_interconnect.haip' has failed on server 'rac2'.

2013-07-16 11:30:53.069

[grid@rac2 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services

CRS-4530: Communications failure contacting Cluster Synchronization Services daemon

CRS-4534: Cannot communicate with Event Manager

[grid@rac2 ~]$ ps -ef |grep asm_

grid 24856 24764 0 12:15 pts/0 00:00:00 grep asm_

[grid@rac2 ~]$ ps -ef |grep ora_

grid 24863 24764 0 12:15 pts/0 00:00:00 grep ora_

5：检测节点2的eth1的IP信息：

[root@rac2 rac2]# ifconfig

eth0 Link encap:Ethernet HWaddr 08:00:27:7E:55:A2

inet addr:10.13.12.151 Bcast:10.13.12.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:fe7e:55a2/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:3233951 errors:0 dropped:0 overruns:0 frame:0

TX packets:2944115 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:206162039 (196.6 MiB) TX bytes:185034492 (176.4 MiB)

lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:16436 Metric:1

RX packets:6074681 errors:0 dropped:0 overruns:0 frame:0