Java错误信息

昨天服务器出现很多错误hs_err_pid*****.log这样的错误信息:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# pthread_getattr_np
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux_x86.cpp:718), pid=18444, tid=140380183054080
#
# JRE version:  (7.0_67-b01) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
 
---------------  T H R E A D  ---------------
 
Current thread (0x00007facd006a000):  JavaThread "Finalizer" daemon [_thread_new, id=18450, stack(0x0000000000000000,0x0000000000000000)]
 
Stack: [0x0000000000000000,0x0000000000000000],  sp=0x00007faccef1ba00,  free space=137090022510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x99eb8a]
V  [libjvm.so+0x49721b]
V  [libjvm.so+0x8237ca]
V  [libjvm.so+0x823815]
V  [libjvm.so+0x952614]
V  [libjvm.so+0x958dd4]
V  [libjvm.so+0x81f988]
 
 
---------------  P R O C E S S  ---------------
 
Java Threads: ( => current thread )
=>0x00007facd006a000 JavaThread "Finalizer" daemon [_thread_new, id=18450, stack(0x0000000000000000,0x0000000000000000)]
  0x00007facd0068000 JavaThread "Reference Handler" daemon [_thread_blocked, id=18449, stack(0x00007faccef1d000,0x00007faccf01e000)]
  0x00007facd0008800 JavaThread "main" [_thread_in_native, id=18445, stack(0x00007facd7bcd000,0x00007facd7cce000)]
 
Other Threads:
  0x00007facd0063800 VMThread [stack: 0x00007faccf01e000,0x00007faccf11f000] [id=18448]
 
VM state:not at safepoint (normal execution)
 
VM Mutex/Monitor currently owned by a thread: None
 
Heap
 PSYoungGen      total 9216K, used 163K [0x00000000f6300000, 0x00000000f6d00000, 0x0000000100000000)
  eden space 8192K, 2% used [0x00000000f6300000,0x00000000f6328fc8,0x00000000f6b00000)
  from space 1024K, 0% used [0x00000000f6c00000,0x00000000f6c00000,0x00000000f6d00000)
  to   space 1024K, 0% used [0x00000000f6b00000,0x00000000f6b00000,0x00000000f6c00000)
 ParOldGen       total 19968K, used 0K [0x00000000e2a00000, 0x00000000e3d80000, 0x00000000f6300000)
  object space 19968K, 0% used [0x00000000e2a00000,0x00000000e2a00000,0x00000000e3d80000)
 PSPermGen       total 21504K, used 855K [0x00000000dd800000, 0x00000000ded00000, 0x00000000e2a00000)
  object space 21504K, 3% used [0x00000000dd800000,0x00000000dd8d5f00,0x00000000ded00000)
 
Card table byte_map: [0x00007facd418e000,0x00007facd42a3000] byte_map_base: 0x00007facd3aa2000
 
Polling page: 0x00007facd8d64000
 
Code Cache  [0x00007facd4363000, 0x00007facd45d3000, 0x00007facd7363000)
 total_blobs=72 nmethods=0 adapters=50 free_code_cache=48820Kb largest_free_block=49991616
 
Compilation events (0 events):
No events
 
GC Heap History (0 events):
No events
 
Deoptimization events (0 events):
No events
 
Internal exceptions (0 events):
No events
 
Events (10 events):
Event: 0.408 Thread 0x00007facd0068000 Thread added: 0x00007facd0068000
Event: 0.408 loading class 0x00007facd404b9b0
Event: 0.408 loading class 0x00007facd404b9b0 done
Event: 0.408 loading class 0x00007facd40629f0
Event: 0.408 loading class 0x00007facd40629f0 done
Event: 0.408 loading class 0x00007facd40629c0
Event: 0.408 loading class 0x00007facd40629c0 done
Event: 0.408 loading class 0x00007facd404bf60
Event: 0.409 loading class 0x00007facd404bf60 done
Event: 0.409 Thread 0x00007facd006a000 Thread added: 0x00007facd006a000
 
 
Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 6667                               /usr/java/jdk1.7.0_67/bin/java
00600000-00601000 rw-p 00000000 ca:01 6667                               /usr/java/jdk1.7.0_67/bin/java
018b4000-018d5000 rw-p 00000000 00:00 0                                  [heap]
dd800000-ded00000 rw-p 00000000 00:00 0 
ded00000-e2a00000 rw-p 00000000 00:00 0 
e2a00000-e3d80000 rw-p 00000000 00:00 0 
e3d80000-f6300000 rw-p 00000000 00:00 0 
f6300000-f6d00000 rw-p 00000000 00:00 0 
f6d00000-100000000 rw-p 00000000 00:00 0 
3e0c800000-3e0c820000 r-xp 00000000 ca:01 262214                         /lib64/ld-2.12.so
3e0ca1f000-3e0ca20000 r--p 0001f000 ca:01 262214                         /lib64/ld-2.12.so
3e0ca20000-3e0ca21000 rw-p 00020000 ca:01 262214                         /lib64/ld-2.12.so
3e0ca21000-3e0ca22000 rw-p 00000000 00:00 0 
3e0d000000-3e0d18b000 r-xp 00000000 ca:01 262222                         /lib64/libc-2.12.so
3e0d18b000-3e0d38a000 ---p 0018b000 ca:01 262222                         /lib64/libc-2.12.so
3e0d38a000-3e0d38e000 r--p 0018a000 ca:01 262222                         /lib64/libc-2.12.so
3e0d38e000-3e0d38f000 rw-p 0018e000 ca:01 262222                         /lib64/libc-2.12.so
3e0d38f000-3e0d394000 rw-p 00000000 00:00 0 
3e0d400000-3e0d417000 r-xp 00000000 ca:01 268700                         /lib64/libpthread-2.12.so
3e0d417000-3e0d617000 ---p 00017000 ca:01 268700                         /lib64/libpthread-2.12.so
3e0d617000-3e0d618000 r--p 00017000 ca:01 268700                         /lib64/libpthread-2.12.so
3e0d618000-3e0d619000 rw-p 00018000 ca:01 268700                         /lib64/libpthread-2.12.so
3e0d619000-3e0d61d000 rw-p 00000000 00:00 0 
3e0d800000-3e0d802000 r-xp 00000000 ca:01 269090                         /lib64/libdl-2.12.so
3e0d802000-3e0da02000 ---p 00002000 ca:01 269090                         /lib64/libdl-2.12.so
3e0da02000-3e0da03000 r--p 00002000 ca:01 269090                         /lib64/libdl-2.12.so
3e0da03000-3e0da04000 rw-p 00003000 ca:01 269090                         /lib64/libdl-2.12.so
3e0dc00000-3e0dc07000 r-xp 00000000 ca:01 269145                         /lib64/librt-2.12.so
3e0dc07000-3e0de06000 ---p 00007000 ca:01 269145                         /lib64/librt-2.12.so
3e0de06000-3e0de07000 r--p 00006000 ca:01 269145                         /lib64/librt-2.12.so
3e0de07000-3e0de08000 rw-p 00007000 ca:01 269145                         /lib64/librt-2.12.so
3e0e000000-3e0e083000 r-xp 00000000 ca:01 270890                         /lib64/libm-2.12.so
3e0e083000-3e0e282000 ---p 00083000 ca:01 270890                         /lib64/libm-2.12.so
3e0e282000-3e0e283000 r--p 00082000 ca:01 270890                         /lib64/libm-2.12.so
3e0e283000-3e0e284000 rw-p 00083000 ca:01 270890                         /lib64/libm-2.12.so
7facb8000000-7facb8021000 rw-p 00000000 00:00 0 
7facb8021000-7facbc000000 ---p 00000000 00:00 0 
7facbecbe000-7facbfdfe000 rw-p 00000000 00:00 0 
7facc0000000-7facc0021000 rw-p 00000000 00:00 0 
7facc0021000-7facc4000000 ---p 00000000 00:00 0 
7facc4000000-7facc4021000 rw-p 00000000 00:00 0 
7facc4021000-7facc8000000 ---p 00000000 00:00 0 
7facc8000000-7facc8021000 rw-p 00000000 00:00 0 
7facc8021000-7faccc000000 ---p 00000000 00:00 0 
7faccee1c000-7faccef1d000 rw-p 00000000 00:00 0 
7faccef1d000-7faccef20000 ---p 00000000 00:00 0 
7faccef20000-7faccf01e000 rw-p 00000000 00:00 0 
7faccf01e000-7faccf01f000 ---p 00000000 00:00 0 
7faccf01f000-7faccfab5000 rw-p 00000000 00:00 0 
7faccfab5000-7faccfc74000 r--s 039e5000 ca:01 9270                       /usr/java/jdk1.7.0_67/jre/lib/rt.jar
7faccfc74000-7faccfdfe000 rw-p 00000000 00:00 0 
7faccfdfe000-7faccfdff000 ---p 00000000 00:00 0 
7faccfdff000-7faccfeff000 rw-p 00000000 00:00 0 
7faccfeff000-7faccff00000 ---p 00000000 00:00 0 
7faccff00000-7facd0000000 rw-p 00000000 00:00 0 
7facd0000000-7facd006c000 rw-p 00000000 00:00 0 
7facd006c000-7facd4000000 ---p 00000000 00:00 0 
7facd4019000-7facd40d3000 rw-p 00000000 00:00 0 
7facd40d3000-7facd40f1000 rw-p 00000000 00:00 0 
7facd40f1000-7facd40fb000 rw-p 00000000 00:00 0 
7facd40fb000-7facd418e000 rw-p 00000000 00:00 0 
7facd418e000-7facd4199000 rw-p 00000000 00:00 0 
7facd4199000-7facd41b7000 rw-p 00000000 00:00 0 
7facd41b7000-7facd41c1000 rw-p 00000000 00:00 0 
7facd41c1000-7facd4253000 rw-p 00000000 00:00 0 
7facd4253000-7facd4259000 rw-p 00000000 00:00 0 
7facd4259000-7facd42a2000 rw-p 00000000 00:00 0 
7facd42a2000-7facd42ad000 rw-p 00000000 00:00 0 
7facd42ad000-7facd4363000 rw-p 00000000 00:00 0 
7facd4363000-7facd45d3000 rwxp 00000000 00:00 0 
7facd45d3000-7facd7363000 rw-p 00000000 00:00 0 
7facd7363000-7facd737d000 r-xp 00000000 ca:01 9077                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd737d000-7facd757d000 ---p 0001a000 ca:01 9077                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd757d000-7facd757e000 rw-p 0001a000 ca:01 9077                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libzip.so
7facd757e000-7facd758a000 r-xp 00000000 ca:01 262174                     /lib64/libnss_files-2.12.so
7facd758a000-7facd778a000 ---p 0000c000 ca:01 262174                     /lib64/libnss_files-2.12.so
7facd778a000-7facd778b000 r--p 0000c000 ca:01 262174                     /lib64/libnss_files-2.12.so
7facd778b000-7facd778c000 rw-p 0000d000 ca:01 262174                     /lib64/libnss_files-2.12.so
7facd778c000-7facd7794000 rw-s 00000000 ca:01 393254                     /tmp/hsperfdata_root/18444
7facd7794000-7facd77bd000 r-xp 00000000 ca:01 9047                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd77bd000-7facd79bd000 ---p 00029000 ca:01 9047                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd79bd000-7facd79bf000 rw-p 00029000 ca:01 9047                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libjava.so
7facd79bf000-7facd79cc000 r-xp 00000000 ca:01 9076                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd79cc000-7facd7bcb000 ---p 0000d000 ca:01 9076                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd7bcb000-7facd7bcd000 rw-p 0000c000 ca:01 9076                       /usr/java/jdk1.7.0_67/jre/lib/amd64/libverify.so
7facd7bcd000-7facd7bd0000 ---p 00000000 00:00 0 
7facd7bd0000-7facd7cce000 rw-p 00000000 00:00 0 
7facd7cce000-7facd883e000 r-xp 00000000 ca:01 9081                       /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd883e000-7facd8a3d000 ---p 00b70000 ca:01 9081                       /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd8a3d000-7facd8b02000 rw-p 00b6f000 ca:01 9081                       /usr/java/jdk1.7.0_67/jre/lib/amd64/server/libjvm.so
7facd8b02000-7facd8b45000 rw-p 00000000 00:00 0 
7facd8b45000-7facd8b5b000 r-xp 00000000 ca:01 9228                       /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8b5b000-7facd8d5b000 ---p 00016000 ca:01 9228                       /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8d5b000-7facd8d5c000 rw-p 00016000 ca:01 9228                       /usr/java/jdk1.7.0_67/lib/amd64/jli/libjli.so
7facd8d5c000-7facd8d5d000 rw-p 00000000 00:00 0 
7facd8d63000-7facd8d64000 rw-p 00000000 00:00 0 
7facd8d64000-7facd8d65000 r--p 00000000 00:00 0 
7facd8d65000-7facd8d66000 rw-p 00000000 00:00 0 
7fff52fd0000-7fff52fe5000 rw-p 00000000 00:00 0                          [stack]
7fff52fff000-7fff53000000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
 
VM Arguments:
java_command: org.apache.catalina.util.ServerInfo
Launcher Type: SUN_STANDARD
 
Environment Variables:
PATH=/sbin:/usr/sbin:/bin:/usr/bin
 
Signal Handlers:
SIGSEGV: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x99f4a0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x81a860], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x81c060], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
 
 
---------------  S Y S T E M  ---------------
 
OS:CentOS release 6.5 (Final)
 
uname:Linux 2.6.32-431.23.3.el6.x86_64 #1 SMP Thu Jul 31 17:20:51 UTC 2014 x86_64
libc:glibc 2.12 NPTL 2.12 
rlimit: STACK 10240k, CORE 0k, NPROC 14846, NOFILE 1024, AS 1048576k
load average:0.02 0.04 0.06
 
/proc/meminfo:
MemTotal:        1920740 kB
MemFree:          175388 kB
Buffers:            7424 kB
Cached:           241320 kB
SwapCached:         1056 kB
Active:          1059524 kB
Inactive:         594728 kB
Active(anon):     905192 kB
Inactive(anon):   515912 kB
Active(file):     154332 kB
Inactive(file):    78816 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4047992 kB
SwapFree:        4026796 kB
Dirty:               112 kB
Writeback:             0 kB
AnonPages:       1402772 kB
Mapped:            37840 kB
Shmem:             15528 kB
Slab:              57004 kB
SReclaimable:      33020 kB
SUnreclaim:        23984 kB
KernelStack:        1720 kB
PageTables:         7692 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5008360 kB
Committed_AS:    2002788 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       12068 kB
VmallocChunk:   34359720520 kB
HardwareCorrupted:     0 kB
AnonHugePages:    966656 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6144 kB
DirectMap2M:     2091008 kB
 
 
CPU:total 2 (32 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, ht, tsc
 
/proc/cpuinfo:
processor   : 0
vendor_id   : GenuineIntel
cpu family : 6
model       : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping    : 7
cpu MHz        : 2300.000
cache size : 15360 KB
physical id    : 0
siblings    : 1
core id        : 0
cpu cores  : 1
apicid      : 0
initial apicid : 0
fpu     : yes
fpu_exception   : yes
cpuid level    : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm rep_good unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips    : 4600.00
clflush size   : 64
cache_alignment : 64
address sizes  : 46 bits physical, 48 bits virtual
power management:
 
processor   : 1
vendor_id   : GenuineIntel
cpu family : 6
model       : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping    : 7
cpu MHz        : 2300.000
cache size : 15360 KB
physical id    : 2
siblings    : 1
core id        : 0
cpu cores  : 1
apicid      : 2
initial apicid : 2
fpu     : yes
fpu_exception   : yes
cpuid level    : 13
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx lm rep_good unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips    : 4599.38
clflush size   : 64
cache_alignment : 64
address sizes  : 46 bits physical, 48 bits virtual
power management:
 
 
 
Memory: 4k page, physical 1920740k(175388k free), swap 4047992k(4026796k free)
 
vm_info: Java HotSpot(TM) 64-Bit Server VM (24.65-b04) for linux-amd64 JRE (1.7.0_67-b01), built on Jul 25 2014 09:20:21 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
 
time: Wed Dec 17 17:48:49 2014
elapsed time: 0 seconds

上面也列出了一些解决办法(直接用词典翻译的):

  • 减少内存系统上的负载
  • 增加物理内存和交换空间
  • 检查交换支持存储已满
  • 使用64位Java在64位操作系统
  • 减少Java堆大小(-Xmx/-Xms)
  • 减少Java线程的数量
  • 减少Java线程堆栈大小(-Xss)
  • 设置较大的代码缓存-XX:ReservedCodeCacheSize=

但是我以前没改过配置文件,怎么会出现这个问题,研究中。

重启一下看看能不能解决了,突然发现服务器已经运行了快一年了。

13:56:21 up 341 days, 22:13,  1 user,  load average: 0.00, 0.00, 0.00

依旧存在问题,想起我之前改了内核的一个参数:vm.overcommit_memory,不知道是不是这个原因,我改回去观察中,问题依旧存在。

想起前两天添加过swap分区,我估计是这个问题,但是不应该啊,os_linux_x86.cpp这个应该是一个32位的东西,我怀疑是JDK的bug, 换到了JDK8 暂时未出现问题。

这是之前的JDK版本:

java version “1.7.0_67”
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)