Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.1.1
    • Fix Version/s: None
    • Component/s: orb
    • Labels:
      None
    • Environment:

      windows 7, 64bit

      Description

      Redeployment causes memory leaks, using jmap and jhat can resolve for example: org.glassfish.javaee.full.deployment.EarClassLoader this class has instance each deploy and undeploy even if app is undeployed

      1. keepalive-ref.tiff
        269 kB
        Cheng Fang
      2. runtest.sh
        0.4 kB
        Tim Quinn

        Activity

        Hide
        russgold added a comment -

        If, as notabenem says, it is the KeepAlive objects preventing the garbage collection, that suggests that deploying the EAR is calling javax.rmi.CORBA.registerTarget, but undeploying is not calling javax.rmi/CORBA.unexportObject for each of the remote objects.

        I'm going to look through the source to see where this is being called.

        Show
        russgold added a comment - If, as notabenem says, it is the KeepAlive objects preventing the garbage collection, that suggests that deploying the EAR is calling javax.rmi.CORBA.registerTarget, but undeploying is not calling javax.rmi/CORBA.unexportObject for each of the remote objects. I'm going to look through the source to see where this is being called.
        Hide
        Ed Bratt added a comment -

        Assigned FYI ...

        Show
        Ed Bratt added a comment - Assigned FYI ...
        Hide
        electricsam added a comment - - edited

        I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations:

        1. contextClassLoader in various threads
        2. Direct references in ThreadLocals in varous threads
        3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector
        4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class
        5. Lots of indirect references in the ThreadLocals in the admin-listener threads

        The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed).

        For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue.

        For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point.

        Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then.

        Bar.java
        package test;
        
        import java.lang.reflect.Array;
        import java.lang.reflect.Field;
        import java.lang.reflect.InvocationTargetException;
        import java.lang.reflect.Method;
        import java.util.logging.Level;
        import java.util.logging.Logger;
        import javax.annotation.PreDestroy;
        import javax.ejb.Singleton;
        import javax.ejb.Startup;
        
        @Singleton
        @Startup
        public abstract class ClassLoaderCleaner {
        
            private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName());
        
            private ClassLoader loader = null;
        
            @PreDestroy
            protected void destroy() {
                try {
                    loader = getClass().getClassLoader();
                    cleanUp();
                } catch (Throwable e) {
                    logger.log(Level.SEVERE, null, e);
                }
            }
        
            private void cleanUp() {
                Thread[] threads = getThreads();
                for (Thread thread : threads) {
                    if (thread != null) {
                        cleanContextClassLoader(thread);
                        cleanOrb(thread);
                        cleanThreadLocal(thread);
        
                    }
        
                }
            }
            
                private Thread[] getThreads() {
                ThreadGroup rootGroup = Thread.currentThread().getThreadGroup();
                ThreadGroup parentGroup;
                while ((parentGroup = rootGroup.getParent()) != null) {
                    rootGroup = parentGroup;
                }
        
                Thread[] threads = new Thread[rootGroup.activeCount()];
                while (rootGroup.enumerate(threads, true) == threads.length) {
                    threads = new Thread[threads.length * 2];
                }
                return threads;
            }
        
            private boolean loaderRemovable(ClassLoader cl) {
                if (cl == null) {
                    return false;
                }
                Object isDoneCalled = getObject(cl, "doneCalled");
                if (cl.getClass().getName().equals(loader.getClass().getName())
                        && isDoneCalled instanceof Boolean && (Boolean) isDoneCalled) {
                    return true;
                }
                return loader == cl;
            }
        
            private Field getField(Class clazz, String fieldName) {
                Field f = null;
                try {
                    f = clazz.getDeclaredField(fieldName);
                } catch (NoSuchFieldException ex) {
        
                } catch (SecurityException ex) {
                    logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex);
                }
        
                if (f == null) {
                    Class parent = clazz.getSuperclass();
                    if (parent != null) {
                        f = getField(parent, fieldName);
                    }
                }
                if (f != null) {
                    f.setAccessible(true);
                }
                return f;
            }
        
            private Object getObject(Object instance, String fieldName) {
                Class clazz = instance.getClass();
                Field f = getField(clazz, fieldName);
                if (f != null) {
                    try {
                        return f.get(instance);
                    } catch (IllegalArgumentException | IllegalAccessException ex) {
                        logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex);
                    }
                }
                return null;
            }
        
            private void cleanContextClassLoader(Thread thread) {
                if (loaderRemovable(thread.getContextClassLoader())) {
                    thread.setContextClassLoader(null);
                    logger.log(Level.INFO, "Cleaned context classloader {0}", thread.getName());
                }
            }
        
            private void cleanOrb(Thread thread) {
                Object currentWork = getObject(thread, "currentWork");
                if (currentWork != null) {
                    Object orb = getObject(currentWork, "orb");
                    if (orb != null) {
                        Object transportManager = getObject(orb, "transportManager");
                        if (transportManager != null) {
                            Thread selector = (Thread) getObject(transportManager, "selector");
                            if (selector != null && loaderRemovable(selector.getContextClassLoader())) {
                                selector.setContextClassLoader(null);
                                logger.log(Level.INFO, "Cleaned orb ref {0}", thread.getName());
                            }
                        }
                    }
                }
            }
        
            private void removeThreadLocal(Object entry, Object threadLocals, Thread thread) {
                ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent");
                if (threadLocal != null) {
                    Class clazz = null;
                    try {
                        clazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
                    } catch (ClassNotFoundException ex) {
                        logger.log(Level.WARNING, null, ex);
                    }
                    if (clazz != null) {
                        Method removeMethod = null;
                        Method[] methods = clazz.getDeclaredMethods();
                        if (methods != null) {
                            for (Method method : methods) {
                                if (method.getName().equals("remove")) {
                                    removeMethod = method;
                                    removeMethod.setAccessible(true);
                                    break;
                                }
                            }
                        }
                        if (removeMethod != null) {
                            try {
                                removeMethod.invoke(threadLocals, threadLocal);
                                logger.log(Level.INFO, "Cleaned threadlocal {0}", thread.getName());
                            } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                                logger.log(Level.SEVERE, null, ex);
                            }
                        }
        
                    }
        
                }
            }
        
            private void cleanThreadLocal(Thread thread) {
                Object threadLocals = getObject(thread, "threadLocals");
                if (threadLocals != null) {
                    Object table = getObject(threadLocals, "table");
                    if (table != null) {
                        int size = Array.getLength(table);
                        for (int i = 0; i < size; i++) {
                            Object entry = Array.get(table, i);
                            if (entry != null) {
                                Field valueField = getField(entry.getClass(), "value");
                                if (valueField != null) {
                                    try {
                                        Object value = valueField.get(entry);
                                        if (value != null && value instanceof ClassLoader && loaderRemovable((ClassLoader) value)) {
                                            removeThreadLocal(entry, threadLocals, thread);
                                        }
                                    } catch (IllegalArgumentException | IllegalAccessException ex) {
                                        logger.log(Level.WARNING, "Unable to get threadlocal value", ex);
                                    }
        
                                }
                            }
        
                        }
                    }
                }
            }
        
        }
        
        Show
        electricsam added a comment - - edited I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations: 1. contextClassLoader in various threads 2. Direct references in ThreadLocals in varous threads 3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector 4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class 5. Lots of indirect references in the ThreadLocals in the admin-listener threads The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed). For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue. For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point. Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then. Bar.java package test; import java.lang.reflect.Array; import java.lang.reflect.Field; import java.lang.reflect.InvocationTargetException; import java.lang.reflect.Method; import java.util.logging.Level; import java.util.logging.Logger; import javax.annotation.PreDestroy; import javax.ejb.Singleton; import javax.ejb.Startup; @Singleton @Startup public abstract class ClassLoaderCleaner { private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName()); private ClassLoader loader = null ; @PreDestroy protected void destroy() { try { loader = getClass().getClassLoader(); cleanUp(); } catch (Throwable e) { logger.log(Level.SEVERE, null , e); } } private void cleanUp() { Thread [] threads = getThreads(); for ( Thread thread : threads) { if (thread != null ) { cleanContextClassLoader(thread); cleanOrb(thread); cleanThreadLocal(thread); } } } private Thread [] getThreads() { ThreadGroup rootGroup = Thread .currentThread().getThreadGroup(); ThreadGroup parentGroup; while ((parentGroup = rootGroup.getParent()) != null ) { rootGroup = parentGroup; } Thread [] threads = new Thread [rootGroup.activeCount()]; while (rootGroup.enumerate(threads, true ) == threads.length) { threads = new Thread [threads.length * 2]; } return threads; } private boolean loaderRemovable( ClassLoader cl) { if (cl == null ) { return false ; } Object isDoneCalled = getObject(cl, "doneCalled" ); if (cl.getClass().getName().equals(loader.getClass().getName()) && isDoneCalled instanceof Boolean && ( Boolean ) isDoneCalled) { return true ; } return loader == cl; } private Field getField( Class clazz, String fieldName) { Field f = null ; try { f = clazz.getDeclaredField(fieldName); } catch (NoSuchFieldException ex) { } catch (SecurityException ex) { logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex); } if (f == null ) { Class parent = clazz.getSuperclass(); if (parent != null ) { f = getField(parent, fieldName); } } if (f != null ) { f.setAccessible( true ); } return f; } private Object getObject( Object instance, String fieldName) { Class clazz = instance.getClass(); Field f = getField(clazz, fieldName); if (f != null ) { try { return f.get(instance); } catch (IllegalArgumentException | IllegalAccessException ex) { logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex); } } return null ; } private void cleanContextClassLoader( Thread thread) { if (loaderRemovable(thread.getContextClassLoader())) { thread.setContextClassLoader( null ); logger.log(Level.INFO, "Cleaned context classloader {0}" , thread.getName()); } } private void cleanOrb( Thread thread) { Object currentWork = getObject(thread, "currentWork" ); if (currentWork != null ) { Object orb = getObject(currentWork, "orb" ); if (orb != null ) { Object transportManager = getObject(orb, "transportManager" ); if (transportManager != null ) { Thread selector = ( Thread ) getObject(transportManager, "selector" ); if (selector != null && loaderRemovable(selector.getContextClassLoader())) { selector.setContextClassLoader( null ); logger.log(Level.INFO, "Cleaned orb ref {0}" , thread.getName()); } } } } } private void removeThreadLocal( Object entry, Object threadLocals, Thread thread) { ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent" ); if (threadLocal != null ) { Class clazz = null ; try { clazz = Class .forName( "java.lang.ThreadLocal$ThreadLocalMap" ); } catch (ClassNotFoundException ex) { logger.log(Level.WARNING, null , ex); } if (clazz != null ) { Method removeMethod = null ; Method[] methods = clazz.getDeclaredMethods(); if (methods != null ) { for (Method method : methods) { if (method.getName().equals( "remove" )) { removeMethod = method; removeMethod.setAccessible( true ); break ; } } } if (removeMethod != null ) { try { removeMethod.invoke(threadLocals, threadLocal); logger.log(Level.INFO, "Cleaned threadlocal {0}" , thread.getName()); } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) { logger.log(Level.SEVERE, null , ex); } } } } } private void cleanThreadLocal( Thread thread) { Object threadLocals = getObject(thread, "threadLocals" ); if (threadLocals != null ) { Object table = getObject(threadLocals, "table" ); if (table != null ) { int size = Array.getLength(table); for ( int i = 0; i < size; i++) { Object entry = Array.get(table, i); if (entry != null ) { Field valueField = getField(entry.getClass(), "value" ); if (valueField != null ) { try { Object value = valueField.get(entry); if (value != null && value instanceof ClassLoader && loaderRemovable(( ClassLoader ) value)) { removeThreadLocal(entry, threadLocals, thread); } } catch (IllegalArgumentException | IllegalAccessException ex) { logger.log(Level.WARNING, "Unable to get threadlocal value" , ex); } } } } } } } }
        Hide
        electricsam added a comment -

        I too can confirm that this is occurring in Glassfish 4.0

        Show
        electricsam added a comment - I too can confirm that this is occurring in Glassfish 4.0
        Hide
        notabenem added a comment -

        Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected.
        This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)

        Show
        notabenem added a comment - Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected. This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)

          People

          • Assignee:
            russgold
            Reporter:
            jelinj14
          • Votes:
            12 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 1 day
              1d
              Remaining:
              Remaining Estimate - 1 day
              1d
              Logged:
              Time Spent - Not Specified
              Not Specified