原文出处:
刘正阳
现象
收到系统报警,查看一台机器频繁FULLGC,且该服务超时。
这是一台4核8G的机器, 使用jdk1.8.0_45-b14。
我们可以直接通过jstat等来观察。这次我先通过CPU开始。
top
查看后该java进程的运行状况为
Tasks: 161 total, 3 running, 158 sleeping, 0 stopped, 0 zombie Cpu(s): 32.1%us, 1.9%sy, 0.0%ni, 65.9%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 8059416k total, 7733088k used, 326328k free, 147536k buffers Swap: 2096440k total, 0k used, 2096440k free, 2012212k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15178 x 20 0 7314m 4.2g 10m S 98.9 55.1 4984:05 java
RES 占用4.2g CPU占用98%
top -H -p 15178
后发现
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15234 x 20 0 7314m 4.2g 10m R 70.2 55.1 781:50.32 java 15250 x 20 0 7314m 4.2g 10m S 52.9 55.1 455:12.27 java
一个PID为 15234 的轻量级进程,对应于java中的线程的本地线程号。
通过printf '%x\n' 15234
计算出16进制的 3b82
通过jstack -l 15178 然后搜索该线程,发现其为
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007ff3e4064000 nid=0x3b82 runnable
定位了确实由GC导致的系统不可用。
jinfo -flags
查看该VM参数
Non-default VM flags: -XX:CICompilerCount=3 -XX:CMSFullGCsBeforeCompaction=0 -XX:CMSInitiatingOccupancyFraction=80 -XX:+DisableExplicitGC -XX:ErrorFile=null -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=null -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=2147483648 -XX:MaxNewSize=348913664 -XX:MaxTenuringThreshold=6 -XX:MinHeapDeltaBytes=196608 -XX:NewSize=348913664 -XX:OldPLABSize=16 -XX:OldSize=1798569984 -XX:+PrintCommandLineFlags -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCMSCompactAtFullCollection -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
jmap -heap 查看统计结果
Debugger attached successfully. Server compiler detected. JVM version is 25.45-b02 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 2147483648 (2048.0MB) NewSize = 348913664 (332.75MB) MaxNewSize = 348913664 (332.75MB) OldSize = 1798569984 (1715.25MB) NewRatio = 2 SurvivorRatio = 8 MetaspaceSize = 21807104 (20.796875MB) CompressedClassSpaceSize = 1073741824 (1024.0MB) MaxMetaspaceSize = 17592186044415 MB G1HeapRegionSize = 0 (0.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 314048512 (299.5MB) used = 314048512 (299.5MB) free = 0 (0.0MB) 100.0% used Eden Space: capacity = 279183360 (266.25MB) used = 279183360 (266.25MB) free = 0 (0.0MB) 100.0% used From Space: capacity = 34865152 (33.25MB) used = 34865152 (33.25MB) free = 0 (0.0MB) 100.0% used To Space: capacity = 34865152 (33.25MB) used = 0 (0.0MB) free = 34865152 (33.25MB) 0.0% used concurrent mark-sweep generation: capacity = 1798569984 (1715.25MB) used = 2657780412087605496 (2.5346569176555684E12MB) free = 15057529128475 MB 1.4777186518907263E11% used 32778 interned Strings occupying 3879152 bytes.
使用jstat -gcutil 15178 1s
观察一段时间GC状况
jstat -gccause 15178 1s S0 S1 E O M CCS YGC YGCT FGC FGCT GCT LGCC GCC 100.00 0.00 100.00 100.00 97.78 95.59 1906 26.575 61433 217668.188 217694.763 Allocation Failure Allocation Failure 0.00 0.00 96.51 100.00 97.78 95.59 1906 26.575 61433 217672.991 217699.566 Allocation Failure No GC 100.00 0.00 100.00 100.00 97.78 95.59 1906 26.575 61434 217672.991 217699.566 Allocation Failure Allo