Applies to:
Oracle Server - Enterprise Edition - Version 9.2.0.1 to 11.1.0.6 [Release 9.2 to 11.1]
z*OBSOLETE: Microsoft Windows Server 2003
Microsoft Windows Itanium (64-bit)
z*OBSOLETE: Microsoft Windows XP (64-bit AMD64 and Intel EM64T)
Microsoft Windows Server 2003 (64-bit Itanium)Microsoft Windows Server 2003Microsoft Windows XP (64-bit AMD64 and Intel EM64T)
?
Description
Periodic Database slowdowns seen in Windows 2003 environments with lots of memory, running Service Pack 1
Occurrence
Environments with a high level of concurrency, many CPU's and large amounts of physical ram are prone to this type of problem.
Symptoms
?
Symptoms
Symptoms include any or all of the following:
1. General slowdowns in the database, which may appear to be hung, even though systemstate dumps may not show evidence of hangs or locking contention. Sometimes, the database may recover from these slowdowns on its own, and begin responding with no user intervention. At other times, the database may appear to be hung for long periods of time.
2. Other symptoms may include slow disk I/O - i.e. a large increase in average I/O times may be observed when reviewing AWR reports covering the time period in question.
3. It may also be observed that network latency increases dramatically. If running in a RAC environment, you may notice an increase in cluster latency - i.e. an increase in the wait times for global cache requests and messages (again, as observed in AWR reports or in some cases as seen via real-time monitoring tools).
4. If running in a RAC environment, we may see IPC timeouts in the ipcdbg logfiles. On Oracle 10g, the ipcdbg logs are found in the $ORACLE_HOME\rdbms\log subdirectory. In Oracle 9i, the ipcdbg logs are found in C:\WINDOWS\system32\osd9i. Examples of entries in the ipcdbg logfiles that indicate a slowdown would be entries such as the following:
?
Thread(06c4): 10/25/07 05:59:56 TCP/IP IPC ErrorLog, version[ 10.2.0.2.0.99 ]
10/25/07 05:59:58 | INFO | 02dc | Initialized proc [oracle : opprodrc1 :####] from interface at#######
10/25/07 07:35:27 | ERROR | 1ae8 | IPC_TCPConnect: async connect err 10048 to [opp :####:###:#####]
10/25/07 07:38:44 | ERROR | 1c6c | IPC_TCPConnect: async connect err 10048 to [opp :###:###:#####]
10/25/07 07:43:37 | ERROR | 1414 | [ipcosd.c :####] IPC_TCPListen2: Timeout 63050 exceeds max 25000 + margin 5000
10/25/07 07:43:37 | ERROR | 1414 | [ipcosd.c :####] IPCInitPort: Timeout 63140 exceeds max 25000 + margin 5000
10/25/07 07:43:40 | ERROR | 15f0 | [ipcosd.c :####] IPC_TCPListen2: Timeout 64950 exceeds max 25000 + margin 5000
?
5. Monitoring of system resources via Perfmon or other monitoring utilities may also reveal an increase in system time, indicating a large amount of work being done by the OS kernel.
?
?
Diagnosing the problem:
This problem can be difficult to diagnose, and it is therefore advised that the solution recommended below be implemented proactively. The symptoms listed in this note will not always manifest in the same way.
In some cases, this problem may be diagnosed by verifying via Perfmon stats, AWR reports, and ipcdbg logs as noted above. In order to get confirmation of the problem, as noted above, systemstate dumps may not show any symptoms other than slow processing, so it may be necessary to run adplus and take user level dumps of the oracle.exe process, in addition to collecting the normal systemstate dumps and hanganalyze dumps. Having a combination of systemsate dumps and adplus dumps of the oracle.exe process may make this issue eas