Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.1.1
    • Fix Version/s: None
    • Component/s: orb
    • Labels:
      None
    • Environment:

      windows 7, 64bit

      Description

      Redeployment causes memory leaks, using jmap and jhat can resolve for example: org.glassfish.javaee.full.deployment.EarClassLoader this class has instance each deploy and undeploy even if app is undeployed

      1. keepalive-ref.tiff
        269 kB
        Cheng Fang
      2. runtest.sh
        0.4 kB
        Tim Quinn

        Activity

        Hide
        Hong Zhang added a comment -

        Assign to tim to take a look as he has done a lot of work of tracking down memory leak in 3.1.1.

        Show
        Hong Zhang added a comment - Assign to tim to take a look as he has done a lot of work of tracking down memory leak in 3.1.1.
        Hide
        Tim Quinn added a comment -

        I was able to reproduce the problem.

        I deployed a simple EAR (containing an EJB) 100 times, and saw 100 EarClassLoader instances remaining live, even after forcing GCs using jconsole.

        (I will attach the simple shell script I used.)

        Run the script, then browse to http://localhost:7000.

        At least part of the problem seems to be caused by EJB monitoring retaining a reference to each StatelessSessionContainer, which in turn has a reference to EarClassLoader.

        I am transferring this to the EJB team for investigation into cleaning up the references from monitoring when a container is retired.

        Show
        Tim Quinn added a comment - I was able to reproduce the problem. I deployed a simple EAR (containing an EJB) 100 times, and saw 100 EarClassLoader instances remaining live, even after forcing GCs using jconsole. (I will attach the simple shell script I used.) Run the script, then browse to http://localhost:7000 . At least part of the problem seems to be caused by EJB monitoring retaining a reference to each StatelessSessionContainer, which in turn has a reference to EarClassLoader. I am transferring this to the EJB team for investigation into cleaning up the references from monitoring when a container is retired.
        Hide
        Tim Quinn added a comment -

        Attached runtest.sh for repeatedly deploying an app and monitoring memory usage.

        Show
        Tim Quinn added a comment - Attached runtest.sh for repeatedly deploying an app and monitoring memory usage.
        Hide
        jelinj14 added a comment -

        When the application is bigger, (I have about 70 stateless session beans on EJB side) it causes big memory leak, every deploy is about 90MB leaks. But I'm not sure whether is it only EarClassLoader.
        Thanks for resolve.

        Show
        jelinj14 added a comment - When the application is bigger, (I have about 70 stateless session beans on EJB side) it causes big memory leak, every deploy is about 90MB leaks. But I'm not sure whether is it only EarClassLoader. Thanks for resolve.
        Hide
        Tim Quinn added a comment -

        There might well be other issues besides the one I found so far. Once that one is resolved we will look into this again for other leaks.

        Show
        Tim Quinn added a comment - There might well be other issues besides the one I found so far. Once that one is resolved we will look into this again for other leaks.
        Hide
        Cheng Fang added a comment -

        Hi Tim, I tried your script with my test EAR on trunk build, and had different results. After finishing your script (100 deploy/undeploy), the result page shows only 3 instances of EarClassLoader.

        I then tried deployed/undeploy one by one, and also shows any time after GC, the count is at most 3. The small number of left over instances could be related to annotation processing tasks.

        Show
        Cheng Fang added a comment - Hi Tim, I tried your script with my test EAR on trunk build, and had different results. After finishing your script (100 deploy/undeploy), the result page shows only 3 instances of EarClassLoader. I then tried deployed/undeploy one by one, and also shows any time after GC, the count is at most 3. The small number of left over instances could be related to annotation processing tasks.
        Hide
        Tim Quinn added a comment -

        Cheng,

        Perhaps something has changed between 3.1.1 and 4.0 (trunk) then. (I do not have a chance right now to try it myself on the trunk.)

        Can whatever the change is be back-ported to 3.1.2? (I realize that requires understanding what change(s) are responsible for the better behavior in 4.0).

        Show
        Tim Quinn added a comment - Cheng, Perhaps something has changed between 3.1.1 and 4.0 (trunk) then. (I do not have a chance right now to try it myself on the trunk.) Can whatever the change is be back-ported to 3.1.2? (I realize that requires understanding what change(s) are responsible for the better behavior in 4.0).
        Hide
        Cheng Fang added a comment -

        Related to http://java.net/jira/browse/GLASSFISH-17468 (WebappClassLoader leak after undeployment)

        Show
        Cheng Fang added a comment - Related to http://java.net/jira/browse/GLASSFISH-17468 (WebappClassLoader leak after undeployment)
        Hide
        Cheng Fang added a comment -

        Once issue 17468 is resolved, it will help eliminate some causes of leaks.

        Another source of leaks, when deploying apps with remote ejb, is from orb layer. An instance of com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive holds a reference to the EarClassLoader after undeploy.

        Show
        Cheng Fang added a comment - Once issue 17468 is resolved, it will help eliminate some causes of leaks. Another source of leaks, when deploying apps with remote ejb, is from orb layer. An instance of com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive holds a reference to the EarClassLoader after undeploy.
        Hide
        Cheng Fang added a comment -

        Attached a screen shot of reference from com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive

        Show
        Cheng Fang added a comment - Attached a screen shot of reference from com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive
        Hide
        Cheng Fang added a comment -

        re-assign to orb team to evaluate if we can reset the context class loader when creating the KeepAlive thread, to avoid inheriting the app class loader from the parent thread.

        Show
        Cheng Fang added a comment - re-assign to orb team to evaluate if we can reset the context class loader when creating the KeepAlive thread, to avoid inheriting the app class loader from the parent thread.
        Hide
        notabenem added a comment -

        Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected.
        This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)

        Show
        notabenem added a comment - Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected. This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)
        Hide
        electricsam added a comment -

        I too can confirm that this is occurring in Glassfish 4.0

        Show
        electricsam added a comment - I too can confirm that this is occurring in Glassfish 4.0
        Hide
        electricsam added a comment - - edited

        I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations:

        1. contextClassLoader in various threads
        2. Direct references in ThreadLocals in varous threads
        3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector
        4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class
        5. Lots of indirect references in the ThreadLocals in the admin-listener threads

        The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed).

        For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue.

        For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point.

        Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then.

        Bar.java
        package test;
        
        import java.lang.reflect.Array;
        import java.lang.reflect.Field;
        import java.lang.reflect.InvocationTargetException;
        import java.lang.reflect.Method;
        import java.util.logging.Level;
        import java.util.logging.Logger;
        import javax.annotation.PreDestroy;
        import javax.ejb.Singleton;
        import javax.ejb.Startup;
        
        @Singleton
        @Startup
        public abstract class ClassLoaderCleaner {
        
            private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName());
        
            private ClassLoader loader = null;
        
            @PreDestroy
            protected void destroy() {
                try {
                    loader = getClass().getClassLoader();
                    cleanUp();
                } catch (Throwable e) {
                    logger.log(Level.SEVERE, null, e);
                }
            }
        
            private void cleanUp() {
                Thread[] threads = getThreads();
                for (Thread thread : threads) {
                    if (thread != null) {
                        cleanContextClassLoader(thread);
                        cleanOrb(thread);
                        cleanThreadLocal(thread);
        
                    }
        
                }
            }
            
                private Thread[] getThreads() {
                ThreadGroup rootGroup = Thread.currentThread().getThreadGroup();
                ThreadGroup parentGroup;
                while ((parentGroup = rootGroup.getParent()) != null) {
                    rootGroup = parentGroup;
                }
        
                Thread[] threads = new Thread[rootGroup.activeCount()];
                while (rootGroup.enumerate(threads, true) == threads.length) {
                    threads = new Thread[threads.length * 2];
                }
                return threads;
            }
        
            private boolean loaderRemovable(ClassLoader cl) {
                if (cl == null) {
                    return false;
                }
                Object isDoneCalled = getObject(cl, "doneCalled");
                if (cl.getClass().getName().equals(loader.getClass().getName())
                        && isDoneCalled instanceof Boolean && (Boolean) isDoneCalled) {
                    return true;
                }
                return loader == cl;
            }
        
            private Field getField(Class clazz, String fieldName) {
                Field f = null;
                try {
                    f = clazz.getDeclaredField(fieldName);
                } catch (NoSuchFieldException ex) {
        
                } catch (SecurityException ex) {
                    logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex);
                }
        
                if (f == null) {
                    Class parent = clazz.getSuperclass();
                    if (parent != null) {
                        f = getField(parent, fieldName);
                    }
                }
                if (f != null) {
                    f.setAccessible(true);
                }
                return f;
            }
        
            private Object getObject(Object instance, String fieldName) {
                Class clazz = instance.getClass();
                Field f = getField(clazz, fieldName);
                if (f != null) {
                    try {
                        return f.get(instance);
                    } catch (IllegalArgumentException | IllegalAccessException ex) {
                        logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex);
                    }
                }
                return null;
            }
        
            private void cleanContextClassLoader(Thread thread) {
                if (loaderRemovable(thread.getContextClassLoader())) {
                    thread.setContextClassLoader(null);
                    logger.log(Level.INFO, "Cleaned context classloader {0}", thread.getName());
                }
            }
        
            private void cleanOrb(Thread thread) {
                Object currentWork = getObject(thread, "currentWork");
                if (currentWork != null) {
                    Object orb = getObject(currentWork, "orb");
                    if (orb != null) {
                        Object transportManager = getObject(orb, "transportManager");
                        if (transportManager != null) {
                            Thread selector = (Thread) getObject(transportManager, "selector");
                            if (selector != null && loaderRemovable(selector.getContextClassLoader())) {
                                selector.setContextClassLoader(null);
                                logger.log(Level.INFO, "Cleaned orb ref {0}", thread.getName());
                            }
                        }
                    }
                }
            }
        
            private void removeThreadLocal(Object entry, Object threadLocals, Thread thread) {
                ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent");
                if (threadLocal != null) {
                    Class clazz = null;
                    try {
                        clazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
                    } catch (ClassNotFoundException ex) {
                        logger.log(Level.WARNING, null, ex);
                    }
                    if (clazz != null) {
                        Method removeMethod = null;
                        Method[] methods = clazz.getDeclaredMethods();
                        if (methods != null) {
                            for (Method method : methods) {
                                if (method.getName().equals("remove")) {
                                    removeMethod = method;
                                    removeMethod.setAccessible(true);
                                    break;
                                }
                            }
                        }
                        if (removeMethod != null) {
                            try {
                                removeMethod.invoke(threadLocals, threadLocal);
                                logger.log(Level.INFO, "Cleaned threadlocal {0}", thread.getName());
                            } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                                logger.log(Level.SEVERE, null, ex);
                            }
                        }
        
                    }
        
                }
            }
        
            private void cleanThreadLocal(Thread thread) {
                Object threadLocals = getObject(thread, "threadLocals");
                if (threadLocals != null) {
                    Object table = getObject(threadLocals, "table");
                    if (table != null) {
                        int size = Array.getLength(table);
                        for (int i = 0; i < size; i++) {
                            Object entry = Array.get(table, i);
                            if (entry != null) {
                                Field valueField = getField(entry.getClass(), "value");
                                if (valueField != null) {
                                    try {
                                        Object value = valueField.get(entry);
                                        if (value != null && value instanceof ClassLoader && loaderRemovable((ClassLoader) value)) {
                                            removeThreadLocal(entry, threadLocals, thread);
                                        }
                                    } catch (IllegalArgumentException | IllegalAccessException ex) {
                                        logger.log(Level.WARNING, "Unable to get threadlocal value", ex);
                                    }
        
                                }
                            }
        
                        }
                    }
                }
            }
        
        }
        
        Show
        electricsam added a comment - - edited I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations: 1. contextClassLoader in various threads 2. Direct references in ThreadLocals in varous threads 3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector 4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class 5. Lots of indirect references in the ThreadLocals in the admin-listener threads The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed). For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue. For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point. Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then. Bar.java package test; import java.lang.reflect.Array; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; import java.util.logging.Level; import java.util.logging.Logger; import javax.annotation.PreDestroy; import javax.ejb.Singleton; import javax.ejb.Startup; @Singleton @Startup public abstract class ClassLoaderCleaner { private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName()); private ClassLoader loader = null ; @PreDestroy protected void destroy() { try { loader = getClass().getClassLoader(); cleanUp(); } catch (Throwable e) { logger.log(Level.SEVERE, null , e); } } private void cleanUp() { Thread [] threads = getThreads(); for ( Thread thread : threads) { if (thread != null ) { cleanContextClassLoader(thread); cleanOrb(thread); cleanThreadLocal(thread); } } } private Thread [] getThreads() { ThreadGroup rootGroup = Thread .currentThread().getThreadGroup(); ThreadGroup parentGroup; while ((parentGroup = rootGroup.getParent()) != null ) { rootGroup = parentGroup; } Thread [] threads = new Thread [rootGroup.activeCount()]; while (rootGroup.enumerate(threads, true ) == threads.length) { threads = new Thread [threads.length * 2]; } return threads; } private boolean loaderRemovable( ClassLoader cl) { if (cl == null ) { return false ; } Object isDoneCalled = getObject(cl, "doneCalled" ); if (cl.getClass().getName().equals(loader.getClass().getName()) && isDoneCalled instanceof Boolean && ( Boolean ) isDoneCalled) { return true ; } return loader == cl; } private Field getField( Class clazz, String fieldName) { Field f = null ; try { f = clazz.getDeclaredField(fieldName); } catch (NoSuchFieldException ex) { } catch (SecurityException ex) { logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex); } if (f == null ) { Class parent = clazz.getSuperclass(); if (parent != null ) { f = getField(parent, fieldName); } } if (f != null ) { f.setAccessible( true ); } return f; } private Object getObject( Object instance, String fieldName) { Class clazz = instance.getClass(); Field f = getField(clazz, fieldName); if (f != null ) { try { return f.get(instance); } catch (IllegalArgumentException | IllegalAccessException ex) { logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex); } } return null ; } private void cleanContextClassLoader( Thread thread) { if (loaderRemovable(thread.getContextClassLoader())) { thread.setContextClassLoader( null ); logger.log(Level.INFO, "Cleaned context classloader {0}" , thread.getName()); } } private void cleanOrb( Thread thread) { Object currentWork = getObject(thread, "currentWork" ); if (currentWork != null ) { Object orb = getObject(currentWork, "orb" ); if (orb != null ) { Object transportManager = getObject(orb, "transportManager" ); if (transportManager != null ) { Thread selector = ( Thread ) getObject(transportManager, "selector" ); if (selector != null && loaderRemovable(selector.getContextClassLoader())) { selector.setContextClassLoader( null ); logger.log(Level.INFO, "Cleaned orb ref {0}" , thread.getName()); } } } } } private void removeThreadLocal( Object entry, Object threadLocals, Thread thread) { ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent" ); if (threadLocal != null ) { Class clazz = null ; try { clazz = Class .forName( "java.lang.ThreadLocal$ThreadLocalMap" ); } catch (ClassNotFoundException ex) { logger.log(Level.WARNING, null , ex); } if (clazz != null ) { Method removeMethod = null ; Method[] methods = clazz.getDeclaredMethods(); if (methods != null ) { for (Method method : methods) { if (method.getName().equals( "remove" )) { removeMethod = method; removeMethod.setAccessible( true ); break ; } } } if (removeMethod != null ) { try { removeMethod.invoke(threadLocals, threadLocal); logger.log(Level.INFO, "Cleaned threadlocal {0}" , thread.getName()); } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) { logger.log(Level.SEVERE, null , ex); } } } } } private void cleanThreadLocal( Thread thread) { Object threadLocals = getObject(thread, "threadLocals" ); if (threadLocals != null ) { Object table = getObject(threadLocals, "table" ); if (table != null ) { int size = Array.getLength(table); for ( int i = 0; i < size; i++) { Object entry = Array.get(table, i); if (entry != null ) { Field valueField = getField(entry.getClass(), "value" ); if (valueField != null ) { try { Object value = valueField.get(entry); if (value != null && value instanceof ClassLoader && loaderRemovable(( ClassLoader ) value)) { removeThreadLocal(entry, threadLocals, thread); } } catch (IllegalArgumentException | IllegalAccessException ex) { logger.log(Level.WARNING, "Unable to get threadlocal value" , ex); } } } } } } } }
        Hide
        Ed Bratt added a comment -

        Assigned FYI ...

        Show
        Ed Bratt added a comment - Assigned FYI ...
        Hide
        russgold added a comment -

        If, as notabenem says, it is the KeepAlive objects preventing the garbage collection, that suggests that deploying the EAR is calling javax.rmi.CORBA.registerTarget, but undeploying is not calling javax.rmi/CORBA.unexportObject for each of the remote objects.

        I'm going to look through the source to see where this is being called.

        Show
        russgold added a comment - If, as notabenem says, it is the KeepAlive objects preventing the garbage collection, that suggests that deploying the EAR is calling javax.rmi.CORBA.registerTarget, but undeploying is not calling javax.rmi/CORBA.unexportObject for each of the remote objects. I'm going to look through the source to see where this is being called.

          People

          • Assignee:
            russgold
            Reporter:
            jelinj14
          • Votes:
            12 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1 day
              1d
              Remaining:
              Remaining Estimate - 1 day
              1d
              Logged:
              Time Spent - Not Specified
              Not Specified