Java错误信息
0
昨天服务器出现很多错误hs_err_pid*****.log这样的错误信息：
#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux_x86.cpp:718), pid=18444, tid=140380183054080
#
# JRE version: (7.0_67-b01) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007facd006a000): JavaThread "Finalizer" daemon [_thread_new, id=18450, stack(0x0000000000000000,0x0000000000000000)]
Stack: [0x0000000000000000,0x0000000000000000], sp=0x00007faccef1ba00, free space=137090022510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99eb8a]
V [libjvm.so+0x49721b]
V [libjvm.so+0x8237ca]
V [libjvm.so+0x823815]
V [libjvm.so+0x952614]
V [libjvm.so+0x958dd4]
V [libjvm.so+0x81f988]
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
=>0x00007facd006a000 JavaThread "Finalizer" daemon [_thread_new, id=18450, stack(0x0000000000000000,0x0000000000000000)]
0x00007facd0068000 JavaThread "Reference Handler" daemon [_thread_blocked, id=18449, stack(0x00007faccef1d000,0x00007faccf01e000)]
0x00007facd0008800 JavaThread "main" [_thread_in_native, id=18445, stack(0x00007facd7bcd000,0x00007facd7cce000)]
Other Threads:
0x00007facd0063800 VMThread [stack: 0x00007faccf01e000,0x00007faccf11f000] [id=18448]
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap
PSYoungGen total 9216K, used 163K [0x00000000f6300000, 0x00000000f6d00000, 0x0000000100000000)
eden space 8192K, 2% used [0x00000000f6300000,0x00000000f6328fc8,0x00000000f6b00000)
from space 1024K, 0% used [0x00000000f6c00000,0x00000000f6c00000,0x00000000f6d00000)
to space 1024K, 0% used [0x00000000f6b00000,0x00000000f6b00000,0x00000000f6c00000)
ParOldGen total 19968K, used 0K [0x00000000e2a00000, 0x00000000e3d80000, 0x00000000f6300000)
object space 19968K, 0% used [0x00000000e2a00000,0x00000000e2a00000,0x00000000e3d80000)
PSPermGen total 21504K, used 855K [0x00000000dd800000, 0x00000000ded00000, 0x00000000e2a00000)
object space 21504K, 3% used [0x00000000dd800000,0x00000000dd8d5f00,0x00000000ded00000)
Card table byte_map: [0x00007facd418e000,0x00007facd42a3000] byte_map_base: 0x00007facd3aa2000
Polling page: 0x00007facd8d64000
Code Cache [0x00007facd4363000, 0x00007facd45d3000, 0x00007facd7363000)
total_blobs=72 nmethods=0 adapters=50 free_code_cache=48820Kb largest_free_block=49991616
Compilation events (0 events):
No events
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (10 events):
Event: 0.408 Thread 0x00007facd0068000 Thread added: 0x00007facd0068000
Event: 0.408 loading class 0x00007facd404b9b0
Event: 0.408 loading class 0x00007facd404b9b0 done
Event: 0.408 loading class 0x00007facd40629f0
Event: 0.408 loading class 0x00007facd40629f0 done
Event: 0.408 loading class 0x00007facd40629c0
Event: 0.408 loading class 0x00007facd40629c0 done
Event: 0.408 loading class 0x00007facd404bf60
Event: 0.409 loading class 0x00007facd404bf60 done
Event: 0.409 Thread 0x00007facd006a000 Thread added: 0x00007facd006a000
Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 6667 /usr/java/jdk1.7.0_67/bin/java
00600000-00601000 rw-p 00000000 ca:01 6667 /usr/java/jdk1.7.0_67/bin/java
018b4000-018d5000 rw-p 00000000 00:00 0 [heap]
dd800000-ded00000 rw-p 00000000 00:00 0
ded00000-e2a00000 rw-p 00000000 00:00 0
e2a00000-e3d80000 rw-p 00000000 00:00 0
e3d80000-f6300000 rw-p 00000000 00:00 0
f6300000-f6d00000 rw-p 00000000 00:00 0
f6d00000-100000000 rw-p 00000000 00:00 0
3e0c800000-3e0c820000 r-xp 00000000 ca:01 262214 /lib64/ld-2.12.so
3e0ca1f000-3e0ca20000 r--p 0001f000 ca:01 262214 /lib64/ld-2.12.so
3e0ca20000-3e0ca21000 rw-p 00020000 ca:01 262214 /lib64/ld-2.12.so
3e0ca21000-3e0ca22000 rw-p 00000000 00:00 0
3e0d000000-3e0d18b000 r-xp 00000000 ca:01 262222 /lib64/libc-2.12.so
3e0d18b000-3e0d38a000 ---p 0018b000 ca:01 262222 /lib64/libc-2.12.so
3e0d38a000-3e0d38e000 r--p 0018a000 ca:01 262222 /lib64/libc-2.12.so
3e0d38e000-3e0d38f000 rw-p 0018e000 ca:01 262222 /lib64/libc-2.12.so
3e0d38f000-3e0d394000 rw-p 00000000 00:00 0
3e0d400000-3e0d417000 r-xp 00000000 ca:01 268700 /lib64/libpthread-2.12.so
3e0d417000-3e0d617000 ---p 00017000 ca:01 268700 /lib64/libpthread-2.12.so
3e0d617000-3e0d618000 r--p 00017000 ca:01 268700 /lib64/libpthread-2.12.so
3e0d618000-3e0d619000 rw-p 00018000 ca:01 268700 /lib64/libpthread-2.12.so
3e0d619000-3e0d61d000 rw-p 00000000 00:00 0
3e0d800000-3e0d802000 r-xp 00000000 ca:01 269090 /lib64/libdl-2.12.so
3e0d802000-3e0da02000 ---p 00002000 ca:01 269090 /lib64/libdl-2.12.so
3e0da02000-3e0da03000 r--p 00002000 ca:01 269090 /lib64/libdl-2.12.so
3e0da03000-3e0da04000 rw-p 00003000 ca:01 269090 /lib64/libdl-2.12.so
3e0dc00000-3e0dc07000 r-xp 00000000 ca:01 269145 /lib64/librt-2.12.so
3e0dc07000-3e0de06000 ---p 00007000 ca:01 269145 /lib64/librt-2.12.so
3e0de06000-3e0de07000 r--p 00006000 ca:01 269145 /lib64/librt-2.12.so
3e0de07000-3e0de08000 rw-p 00007000 ca:01 269145 /lib64/librt-2.12.so
3e0e000000-3e0e083000 r-xp 00000000 ca:01 270890 /lib64/libm-2.12.so
3e0e083000-3e0e282000 ---p 00083000 ca:01 270890 /lib64/libm-2.12.so
3e0e282000-3e0e283000 r--p 00082000 ca:01 270890 /lib64/libm-2.12.so
3e0e283000-3e0e284000 rw-p 00083000 ca:01 270890 /lib64/libm-2.12.so
7facb8000000-7facb8021000 rw-p 00000000 00:00 0
7facb8021000-7facbc000000 ---p 00000000 00:00 0
7facbecbe000-7facbfdfe000 rw-p 00000000 00:00 0
7facc0000000-7facc0021000 rw-p 00000000 00:00 0
7facc0021000-7facc4000000 ---p 00000000 00:00 0
7facc4000000-7facc4021000 rw-p 00000000 00:00 0
7facc4021000-7facc8000000 ---p 00000000 00:00 0
7facc8000000-7facc8021000 rw-p 00000000 00:00 0
7facc8021000-7faccc000000 ---p 00000000 00:00 0
7faccee1c000-7faccef1d000 rw-p 00000000 00:00 0
7faccef1d000-7faccef20000 ---p 00000000 00:00 0
7faccef20000-7faccf01e000 rw-p 00000000 00:00 0
7faccf01e000-7faccf01f000 ---p 00000000 00:00 0
7faccf01f000-7faccfab5000 rw-p 00000000 00:00 0
7faccfab5000-7faccfc74000 r--s 039e5000 ca:01 9270 /usr/java/jdk1.7.0_67/jre/lib/rt.jar
7faccfc74000-7faccfdfe000 rw-p 00000000 00:00 0
7faccfdfe000-7faccfdff000 ---p 00000000 00:00 0
7faccfdff000-7faccfeff000 rw-p 00000000 00:00 0
7faccfeff000-7faccff00000 ---p 00000000 00:00 0
7faccff00000-7facd0000000 rw-p 00000000 00:00 0
7facd0000000-7facd006c000 rw-p 00000000 00:00 0
7facd006c000-7facd4000000 ---p 00000000 00:00 0
7facd4019000-7facd40d3000 rw-p 00000000 00:00 0
7facd40d3000-7facd40f1000 rw-p 00000000 00:00 0
7facd40f1000-7facd40fb000 rw-p 00000000 00:00 0
7facd40fb000-7facd418e000 rw-p 00000000 00:00 0
7facd418e000-7facd4199000 rw-p 00000000 00:00 0
7facd4199000-7facd41b7000 rw-p 00000000 00:00 0
7facd41b7000-7facd41c1000 rw-p 00000000 00:00 0
7facd41c1000-7facd4253000 rw-p 00000000 00:00 0
7facd4253000-7facd4259000 rw-p 00000000 00:00 0
7facd4259000-7facd42a2000 rw-p 00000000 00:00 0
7facd42a2000-7facd42ad000 rw-p 00000000 00:00 0
7facd42ad000-7facd4363000 rw-p 00000000 00:00 0
7facd4363000-7facd45d3000 rwxp 00000000 00:00 0
7facd45d3000-7facd7363000 rw-p 00000000 00:00 0
7facd7363000-7facd737d000 r-xp 00000000 ca:01 9077 /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd737d000-7facd757d000 ---p 0001a000 ca:01 9077 /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd757d000-7facd757e000 rw-p 0001a000 ca:01 9077 /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd757e000-7facd758a000 r-xp 00000000 ca:01 262174 /lib64/libnss_files-2.12.so
7facd758a000-7facd778a000 ---p 0000c000 ca:01 262174 /lib64/libnss_files-2.12.so
7facd778a000-7facd778b000 r--p 0000c000 ca:01 262174 /lib64/libnss_files-2.12.so
7facd778b000-7facd778c000 rw-p 0000d000 ca:01 262174 /lib64/libnss_files-2.12.so
7facd778c000-7facd7794000 rw-s 00000000 ca:01 393254 /tmp/hsperfdata_root/18444
7facd7794000-7facd77bd000 r-xp 00000000 ca:01 9047 /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd77bd000-7facd79bd000 ---p 00029000 ca:01 9047 /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd79bd000-7facd79bf000 rw-p 00029000 ca:01 9047 /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd79bf000-7facd79cc000 r-xp 00000000 ca:01 9076 /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd79cc000-7facd7bcb000 ---p 0000d000 ca:01 9076 /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd7bcb000-7facd7bcd000 rw-p 0000c000 ca:01 9076 /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd7bcd000-7facd7bd0000 ---p 00000000 00:00 0
7facd7bd0000-7facd7cce000 rw-p 00000000 00:00 0
7facd7cce000-7facd883e000 r-xp 00000000 ca:01 9081 /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd883e000-7facd8a3d000 ---p 00b70000 ca:01 9081 /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd8a3d000-7facd8b02000 rw-p 00b6f000 ca:01 9081 /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd8b02000-7facd8b45000 rw-p 00000000 00:00 0
7facd8b45000-7facd8b5b000 r-xp 00000000 ca:01 9228 /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8b5b000-7facd8d5b000 ---p 00016000 ca:01 9228 /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8d5b000-7facd8d5c000 rw-p 00016000 ca:01 9228 /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8d5c000-7facd8d5d000 rw-p 00000000 00:00 0
7facd8d63000-7facd8d64000 rw-p 00000000 00:00 0
7facd8d64000-7facd8d65000 r--p 00000000 00:00 0
7facd8d65000-7facd8d66000 rw-p 00000000 00:00 0
7fff52fd0000-7fff52fe5000 rw-p 00000000 00:00 0 [stack]
7fff52fff000-7fff53000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
java_command: org.apache.catalina.util.ServerInfo
Launcher Type: SUN_STANDARD
Environment Variables:
PATH=/sbin:/usr/sbin:/bin:/usr/bin
Signal Handlers:
SIGSEGV: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x81c060], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
--------------- S Y S T E M ---------------
OS:CentOS release 6.5 (Final)
uname:Linux 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64
libc:glibc 2.12 NPTL 2.12
rlimit: STACK 10240k, CORE 0k, NPROC 14846, NOFILE 1024, AS 1048576k
load average:0.02 0.04 0.06
/proc/meminfo:
MemTotal: 1920740 kB
MemFree: 175388 kB
Buffers: 7424 kB
Cached: 241320 kB
SwapCached: 1056 kB
Active: 1059524 kB
Inactive: 594728 kB
Active(anon): 905192 kB
Inactive(anon): 515912 kB
Active(file): 154332 kB
Inactive(file): 78816 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4047992 kB
SwapFree: 4026796 kB
Dirty: 112 kB
Writeback: 0 kB
AnonPages: 1402772 kB
Mapped: 37840 kB
Shmem: 15528 kB
Slab: 57004 kB
SReclaimable: 33020 kB
SUnreclaim: 23984 kB
KernelStack: 1720 kB
PageTables: 7692 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 5008360 kB
Committed_AS: 2002788 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 12068 kB
VmallocChunk: 34359720520 kB
HardwareCorrupted: 0 kB
AnonHugePages: 966656 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 6144 kB
DirectMap2M: 2091008 kB
CPU:total 2 (32 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, ht, tsc
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm rep_good unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips : 4600.00
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm rep_good unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips : 4599.38
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 1920740k(175388k free), swap 4047992k(4026796k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (24.65-b04) for linux-amd64 JRE (1.7.0_67-b01), built on Jul 25 2014 09:20:21 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Wed Dec 17 17:48:49 2014
elapsed time: 0 seconds
上面也列出了一些解决办法（直接用词典翻译的）：
- 减少内存系统上的负载
- 增加物理内存和交换空间
- 检查交换支持存储已满
- 使用64位Java在64位操作系统
- 减少Java堆大小(-Xmx/-Xms)
- 减少Java线程的数量
- 减少Java线程堆栈大小(-Xss)
- 设置较大的代码缓存-XX:ReservedCodeCacheSize=
但是我以前没改过配置文件，怎么会出现这个问题，研究中。
重启一下看看能不能解决了，突然发现服务器已经运行了快一年了。
13:56:21 up 341 days, 22:13, 1 user, load average: 0.00, 0.00, 0.00
依旧存在问题，想起我之前改了内核的一个参数：vm.overcommit_memory，不知道是不是这个原因，我改回去观察中，问题依旧存在。
想起前两天添加过swap分区，我估计是这个问题，但是不应该啊，os_linux_x86.cpp这个应该是一个32位的东西，我怀疑是JDK的BUG，换到了JDK8暂时未出现问题。
这是之前的JDK版本：
java version “1.7.0_67”
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)20220527批注
这个日志一般都是JVM异常结束时记录的，大概率都是内存问题。