一：版本信息

操作系统版本：AIX 61009

数据库版本：11.2.0.3.11(RAC)

二：错误描述

1) crsctl stat res -t命令查看crs资源状态的时候，发现"ora.LISTENER.lsnr"资源状态异常：

ora.LISTENER.lsnr
               ONLINE  OFFLINE      ****1  ##实例1                                      
               ONLINE  OFFLINE      ****2  ##实例2                                     

2)检查监听状态正常

lsnrctl       

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production on 11-NOV-2014 10:53:26

Copyright (c) 1991, 2011, Oracle.  All rights reserved.

Welcome to LSNRCTL, type "help" for information.

LSNRCTL> set password
Password: 
The command completed successfully
LSNRCTL> status
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - Production
Start Date                06-NOV-2014 17:22:05
Uptime                    4 days 17 hr. 31 min. 48 sec
Trace Level               off
Security                  ON: Password
SNMP                      OFF
Listener Parameter File   /oracle/app/11.2.0/grid/network/admin/listener.ora
Listener Log File         /oracle/app/11.2.0/grid/log/diag/tnslsnr/bidb4/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.**)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.***)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "bidb" has 1 instance(s).
  Instance "bidb2", status READY, has 1 handler(s) for this service...
The command completed successfully

3)检查监听日志，发现目前监听还能正常接受连接

su - grid

cd $ORACLE_HOME/log/diag/tnslsnr/主机名/listener/alert

host_id='bidb4'##数据库主机名
 host_addr='***.***.**.**>##数据库主机ip地址
 11-NOV-2014 10:46:02 * (CONNECT_DATA=(SERVICE_NAME=bidb)(CID=(PROGRAM=osh)(HOST=EGSNS-49-24.nd.cmgame.com)(USER=dsadm))(SERVER=dedicated)(INSTANCE_NAME=bidb2)) *
 (ADDRESS=(PROTOCOL=tcp)(HOST=***.***.**.**)(PORT=61378)) * establish * bidb * 0
                   ##连接发起的IP地址

4)检查 CRS alert日志发现有如下报错：

/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:03:21.139:
[/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "start" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:03:21.267:
[/oracle/app/11.2.0/grid/bin/oraagent.bin(7274512)]CRS-5016:Process "/oracle/app/11.2.0/grid/bin/lsnrctl" spawned by agent "/oracle/app/11.2.0/grid/bin/oraagent.bin" f
or action "check" failed: details at "(:CLSN00010:)" in "/oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/oraagent_grid.log"
2014-11-07 15:08:01.278:

##标红的部分报，监听检查报错，这个就跟我的问题联系上了。根据提示 查看oracle/app/11.2.0/grid/log/bidb4/agent/crsd/oraagent_grid/下的oraagent_grid日志(注意可能你要找的信息已经被刷出当前日志了，根据时间点查看相应日志。例如我需要的相关信息就在oraagent_grid.l02日志文件中)

5)查看oraagent_grid.l02日志