Please review the alert log for the last two shutdown / startups and the messages:
SHUTDOWN: Active sessions prevent database close operation
Please refer to
Note 434690.1 - Database Jobs Do Not Run After a Failed 'Shutdown Immediate'
Database Jobs Do Not Run After a Failed 'Shutdown Immediate' [ID 434690.1]
==>
Symptoms
After issuing a 'shutdown immediate' command, if the command does not execute after an hour, the Oracle Server automatically cancels the operation. A message will appear in the alert log as follows:
SHUTDOWN: Active sessions prevent database close operation
Unfortunately this has the added effect of shutting down all the job queue slaves as part of the shutdown process. Any jobs listed in the DBA_JOBS view will not run. This can affect a wide variety of functions, such as statistics gathering, replication propagation, materialized view refresh, etc.
Cause
This is expected behavior. Once the 'shutdown' command is issued, the shutdown must proceed; the shutdown process cannot be undone. This is because the instance is not expected to live after a shutdown is canceled.
那么是否可能是这两篇文档中提到的情况呢？客户恰恰是在3月25日对DB进行了重启，于是仔细看了那天shutdown和open的日志
Mon Mar 25 13:11:48 2013
Shutting down instance (immediate)
License high water mark = 51
Mon Mar 25 13:11:48 2013
Stopping Job queue slave processes
Mon Mar 25 13:11:53 2013
Process OS id : 14300 alive after kill
Errors in file
Mon Mar 25 13:11:58 2013
Job queue slave processes stopped
All dispatchers and shared servers shutdown
Mon Mar 25 13:12:09 2013
PMON failed to acquire latch, see PMON dump
PMON failed to acquire latch, see PMON dump
PMON failed to acquire latch, see PMON dump
PMON failed to acquire latch, see PMON dump
PMON failed to acquire latch, see PMON dump
...
Mon Mar 25 13:13:43 2013
Completed: ALTER DATABASE CLOSE NORMAL
Mon Mar 25 13:13:43 2013
ALTER DATABASE DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active <===shutdown过程没有问题
Mon Mar 25 13:40:19 2013
Starting ORACLE instance (normal)
...
Mon Mar 25 17:27:03 2013
Shutting down instance (abort)
License high water mark = 55
Instance terminated by USER, pid = 11609
Mon Mar 25 17:27:22 2013
Starting ORACLE instance (normal)
...
Mon Mar 25 17:27:33 2013
Errors in file /app/oracle/admin/neimg/udump/neimg_ora_11994.trc:
ORA-00604: error occurred at recursive SQL level 2
ORA-01013: user requested cancel of current operation ==> 启动过程中有客户有执行CANCEL的操作，并且有ora-7445报出
Mon Mar 25 17:27:33 2013
Errors in file /app/oracle/admin/neimg/udump/neimg_ora_11994.trc:
ORA-07445: exception encountered: core dump [kkttrex()+1365] [SIGSEGV] [Address not mapped to object] [0x000000074] [] []
...
Mon Mar 25 17:27:51 2013
alter database open
Mon Mar 25 17:27:51 2013
ORA-1531 signalled during: alter database open... <===正常启动成功后会有'Completed: ALTER DATABASE OPEN'的信息，但这里没有
Mon Mar 25 17:33:30 2013
Shutting down archive processes
Mon Mar 25 17:33:35 2013
ARCH shutting down
ARC2: Archival stopped
所有极有可能是在DB启动中途的那次客户手动CANCEL引起的，建议客户尝试重启DB，看能否解决这个问题。
shutdown abort
startup restrict
shutdown immediate
startup
重启DB之后问题解决。
总结：看来DB在做SHUTDOWN或者OPEN中途不能随意CTRL+C执行CANCEL，否则会导致DB状态异常，影响 系统的一些功能，出现各种各样诡异的问题。