近期weblogic 11g永久代内存溢出，分析JVM dump文件是没有用处的，因为那只是堆内存，永久代不在里面。目前永久代设置是1G,遥想当年，只有400M，这么多年来一直在涨，现在一次full gc需要10多秒。可以增大到1.5G，但从GC日志中可以看到永久代在不断的增长，如果二周不重启weblogic，内存又不够用了，不是长久之计。java.lang.OutOfMemoryError: PermGen spaceat java.lang.Class.getDeclaredMethods0(Native Method) ~[na:1.6.0_30]at java.lang.Class.PRivateGetDeclaredMethods(Class.java:2427) ~[na:1.6.0_30]at java.lang.Class.getDeclaredMethod(Class.java:1935) ~[na:1.6.0_30]at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382) ~[na:1.6.0_30]at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52) ~[na:1.6.0_30]at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438) ~[na:1.6.0_30]at java.security.AccessController.doPrivileged(Native Method) ~[na:1.6.0_30]可以看到是方法区有问题，创建了大量的类，从GC日志上看，启动之后永久代有700M，随着时间的推移，会缓慢增长，这就是内存泄露，堆内存泄露也是这样。如何定位哪些类一直在涨呢，此时就用到两个重要的JVM参数，一个显示每个加载的类，一个显示每个卸载的类：-XX:+TraceClassLoading -XX:+TraceClassUnloadingweblogic里面会出现这种日志，需要自己写程序解析最后还有哪些类存活着。[Loaded MM.statistics.mmbillstatistics.exception.MmBillStatisticalTypeException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/mmbillstatistics/exception/MmBillStatisticalTypeException.class][Loaded MM.statistics.mmbillstatistics.appservice.impl.MmBillStatisticalTypeSes_1k61gv_IMmBillStatisticalTypeServiceImpl_1035_WLStub from file:/data/wls1035/modules/com.bea.core.utils.wrapper_1.4.0.0.jar][Loaded MM.statistics.amfmHuiXuFinance.appservice.AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf from file:/data/Domain/mm_Domain/servers/mmServer1/cache/EJBCompilerCache/1nhs7towub2hs/mm/statistics/amfmHuiXuFinance/appservice/AmfmHuiXuFinanceBizService_f879n8_IAmfmHuiXuFinanceBizServiceRIntf.class][Loaded MM.statistics.amfmHuiXuFinance.exception.AmfmHuiXuFinanceException from file:/data/Domain/mm_Domain/servers/mmServer1/stage/EAR/EAR/APP-INF/classes/mm/statistics/amfmHuiXuFinance/exception/AmfmHuiXuFinanceException.class][Unloading class mm.purchase.purchaSEOrder.model.PurchaseItemVO][Unloading class mm.inventory.reservereplenishplan.appservice.impl.ReserveReplenishPlanBizService][Unloading class mm.inventory.reservequota.dao.ReserveQuotaBatchDAO]解析程序开始：drop table load_class purge;truncate table load_class;create table load_class( nu number, action varchar2(20), class_name varchar2(1000), class_file varchar2(1000));import java.io.BufferedReader;import java.io.File;import java.io.FileInputStream;import java.io.InputStreamReader;import java.sql.Connection;import java.sql.DriverManager;import java.sql.PreparedStatement;/** * [Loaded java.lang.ClassNotFoundException from /usr/local/jdk1.6/jre/lib/rt.jar] * [Unloading class com.MM.externalinterface.fmis.impl.FmisBizService] * */public class PermAla { static final String driver_class = "Oracle.jdbc.driver.OracleDriver"; static final String connectionURL = "jdbc:oracle:thin:@10.10.15.164:1521:orcl"; static final String userID = "SPROC"; static final String userPassWord = "SPROC"; public static void readTxtFile(String filePath){ Connection con = null; String s_sql = "insert into load_class values(?,?,?,?)"; PreparedStatement pstmt = null; int i=0; try { Class.forName (driver_class).newInstance(); con = DriverManager.getConnection(connectionURL, userID, userPassword); pstmt = con.prepareStatement(s_sql); con.setAutoCommit(false); String encoding="GBK"; File file=new File(filePath); InputStreamReader read = new InputStreamReader( new FileInputStream(file),encoding);//考虑到编码格式 BufferedReader bufferedReader = new BufferedReader(read); String lineTxt = null; String[] lineTxtArray= null; while((lineTxt = bufferedReader.readLine()) != null){ if(lineTxt.indexOf("[Loaded")==0 ) { lineTxtArray = lineTxt.split(" "); pstmt.setInt(1, i); pstmt.setString(2, lineTxtArray[0]); pstmt.setString(3, lineTxtArray[1]); if(lineTxtArray.length ==4) { pstmt.setString(4, lineTxtArray[3]); }else{ System.out.println(lineTxt); } pstmt.addBatch(); i++; } else if(lineTxt.indexOf("[Unloading")==0){ lineTxtArray = lineTxt.split(" "); pstmt.setInt(1, i); pstmt.setString(2, lineTxtArray[0]); pstmt.setString(3, lineTxtArray[2]); pstmt.addBatch(); i++; } if(i % 10000 == 0){ pstmt.executeBatch(); con.commit(); } } con.commit(); read.close(); } catch (Exception e) { System.out.println("读取文件内容出错,行数："+i); e.printStackTrace(); }finally{ if(pstmt != null){ try { pstmt.close(); } catch (Exception e) { e.printStackTrace(); }finally{ pstmt = null; } } if(con != null){ try { con.close(); } catch (Exception e) { e.printStackTrace(); }finally{ con = null; } } } } public static void main(String argv[]){ String filePath = "E://永久代内存溢出//20161213//Server1.log"; readTxtFile(filePath); }}--执行java代码导入之后，处理一些特殊的格式update load_class set action = replace(action, '[', ''), class_file = replace(class_file, ']', ''), class_name = replace(class_name, ']', '');commit;update load_class set class_name = replace(class_name, 'file:', ''), class_file = replace(class_file, 'file:', '');commit;SQL> select * from load_class where rownum <10; NU ACTION CLASS_NAME CLASS_FILE--- ------- --------------------------------------- -------------------------------- 96 Loaded sun.misc.SharedSecrets /data/jdk1.6.0_45/jre/lib/rt.jar 97 Loaded sun.misc.Unsafe /data/jdk1.6.0_45/jre/lib/rt.jar 98 Loaded java.lang.IncompatibleClassChangeError /data/jdk1.6.0_45/jre/lib/rt.jar 99 Loaded java.lang.NoSuchMethodError /data/jdk1.6.0_45/jre/lib/rt.jar100 Loaded sun.reflect.Reflection /data/jdk1.6.0_45/jre/lib/rt.jar101 Loaded java.util.Collections /data/jdk1.6.0_45/jre/lib/rt.jar102 Loaded java.lang.Iterable /data/jdk1.6.0_45/jre/lib/rt.jar103 Loaded java.util.Collection /data/jdk1.6.0_45/jre/lib/rt.jar104 Loaded java.util.Set /data/jdk1.6.0_45/jre/lib/rt.jar --显示加载且没有卸载的类 drop table purge;create table aswith res as(select count(*) over(partition by action,class_name order by nu asc) rn, t.* from load_class t),res1 as (select rn,class_name from res a where a.action='Loaded' minus select rn,class_name from res b where b.action='Unloading')select b.* from res1 a, res b where a.rn= b.rn and a.class_name= b.class_name;查看数据，对数据摸底 select count(1) from where class_file='__JVM_DefineClass__';select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/lib%';select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/APP-INF/classes%' or class_file like '/data/Domain/MMDomain/servers/MMServer1/stage/EAR/EAR/ejb%';select count(1) from where class_file like '/data/wls1035/%'; select count(1) from where class_file like '/data/Domain/MMDomain/servers/MMServer1/cache/EJBCompilerCache/%'; select count(1) from where class_file like '/data/jdk1.6.0_45/jre%'; select count(1) from where class_file in ('weblogic.utils.classloaders.GenericClassLoader', 'sun.misc.Launcher$AppClassLoader', 'weblogic.utils.classloaders.ChangeAwareClassLoader','instance');select count(1) from where class_file like '/data/Domain/MMDomain/jsp_temp%'; 可以看到反射的类很多，重点研究__JVM_DefineClass__和sun.reflect.GeneratedMethodAccessor，发现是反射产生的一个问题。参考http://stackoverflow.com/questions/16130292/java-lang-outofmemoryerror-permgen-space-java-reflection
When using Java reflection, the JVM has two methods of accessing the information on the class being reflected. It can use a JNI accessor, or a Java bytecode accessor.
If it uses a Java bytecode accessor, then it needs to have its own Java class and classloader (sun/reflect/GeneratedMethodAccessor class and sun/reflect/DelegatingClassLoader). Theses classes and classloaders use native memory.
The accessor bytecode can also get JIT compiled, which will increase the native memory use even more.
If Java reflection is used frequently, this can add up to a significant amount of native memory use. The JVM will use the JNI accessor first, then after some number of accesses on the same class, will change to use the Java bytecode accessor. This is called inflation, when the JVM changes from the JNI accessor to the bytecode accessor. Fortunately, we can control this with a Java property.
The sun.reflect.inflationThreshold property tells the JVM what number of times to use the JNI accessor. If it is set to 0, then the JNI accessors are always used. Since the bytecode accessors use more native memory than the JNI ones, if we are seeing a lot of Java reflection, we will want to use the JNI accessors. To do this, we just need to set the inflationThreshold property to zero.
如果节点使用sun公司jdk，配置-Dsun.reflect.inflationThreshold=2147483647 如果是IBM的jdk，配置-Dsun.reflect.inflationThreshold=0
以下是配置的前后类数量对比，反射类有明显的下降，从gc日志上看也有回收的迹象：
| 加载类的出处 | Server1 | server2 |
| 加参数前 | 加参数后 | 加参数前 | 加参数后 |
| __JVM_DefineClass__(反射产生的类) | 42372 | 3632 | 19961 | 2822 |
| EAR/APP-INF/lib | 25552 | 23162 | 23613 | 23579 |
| EAR/APP-INF/classes | 21531 | 15684 | 15167 | 15176 |
| wls1035 | 19017 | 19420 | 19661 | 19168 |
| MMServer1/cache/EJBCompilerCache | 5996 | 5996 | 5996 | 5996 |
| /data/jdk1.6.0_45/jre | 3494 | 3493 | 3175 | 3458 |
| weblogic.utils.classloaders.GenericClassLoadersun.misc.Launcher$AppClassLoaderweblogic.utils.classloaders.ChangeAwareClassLoader | 2550 | 2574 | 2508 | 2516 |
| jsp_temp | 843 | 1395 | 843 | 849 |
当然我遇到的情况是反射类引起的问题，如果你的永久代设置太小，不用纠结，直接把永久代内存加大；如果是类太多导致（不是反射类），那就想办法瘦身，想办法 改造去掉一些依赖的jar包，可能是一个大工程。
