[GLASSFISH-17449] Redeploy memory leak Created: 20/Oct/11  Updated: 04/May/16

Status: Open
Project: glassfish
Component/s: orb
Affects Version/s: 3.1.1
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: jelinj14 Assignee: russgold
Resolution: Unresolved Votes: 13
Labels: None
Remaining Estimate: 1 day
Time Spent: Not Specified
Original Estimate: 1 day
Environment:

windows 7, 64bit


Attachments: File keepalive-ref.tiff     File runtest.sh    
Tags: 3_1_2-exclude, 4_0_1-approved, 4_0_1-evangelists, memoryleak, orb-review, redeploy

 Description   

Redeployment causes memory leaks, using jmap and jhat can resolve for example: org.glassfish.javaee.full.deployment.EarClassLoader this class has instance each deploy and undeploy even if app is undeployed



 Comments   
Comment by Hong Zhang [ 21/Oct/11 ]

Assign to tim to take a look as he has done a lot of work of tracking down memory leak in 3.1.1.

Comment by Tim Quinn [ 21/Oct/11 ]

I was able to reproduce the problem.

I deployed a simple EAR (containing an EJB) 100 times, and saw 100 EarClassLoader instances remaining live, even after forcing GCs using jconsole.

(I will attach the simple shell script I used.)

Run the script, then browse to http://localhost:7000.

At least part of the problem seems to be caused by EJB monitoring retaining a reference to each StatelessSessionContainer, which in turn has a reference to EarClassLoader.

I am transferring this to the EJB team for investigation into cleaning up the references from monitoring when a container is retired.

Comment by Tim Quinn [ 21/Oct/11 ]

Attached runtest.sh for repeatedly deploying an app and monitoring memory usage.

Comment by jelinj14 [ 21/Oct/11 ]

When the application is bigger, (I have about 70 stateless session beans on EJB side) it causes big memory leak, every deploy is about 90MB leaks. But I'm not sure whether is it only EarClassLoader.
Thanks for resolve.

Comment by Tim Quinn [ 21/Oct/11 ]

There might well be other issues besides the one I found so far. Once that one is resolved we will look into this again for other leaks.

Comment by Cheng Fang [ 25/Oct/11 ]

Hi Tim, I tried your script with my test EAR on trunk build, and had different results. After finishing your script (100 deploy/undeploy), the result page shows only 3 instances of EarClassLoader.

I then tried deployed/undeploy one by one, and also shows any time after GC, the count is at most 3. The small number of left over instances could be related to annotation processing tasks.

Comment by Tim Quinn [ 25/Oct/11 ]

Cheng,

Perhaps something has changed between 3.1.1 and 4.0 (trunk) then. (I do not have a chance right now to try it myself on the trunk.)

Can whatever the change is be back-ported to 3.1.2? (I realize that requires understanding what change(s) are responsible for the better behavior in 4.0).

Comment by Cheng Fang [ 25/Oct/11 ]

Related to http://java.net/jira/browse/GLASSFISH-17468 (WebappClassLoader leak after undeployment)

Comment by Cheng Fang [ 03/Nov/11 ]

Once issue 17468 is resolved, it will help eliminate some causes of leaks.

Another source of leaks, when deploying apps with remote ejb, is from orb layer. An instance of com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive holds a reference to the EarClassLoader after undeploy.

Comment by Cheng Fang [ 03/Nov/11 ]

Attached a screen shot of reference from com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive

Comment by Cheng Fang [ 03/Nov/11 ]

re-assign to orb team to evaluate if we can reset the context class loader when creating the KeepAlive thread, to avoid inheriting the app class loader from the parent thread.

Comment by notabenem [ 02/Dec/13 ]

Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected.
This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)

Comment by electricsam [ 15/Jan/14 ]

I too can confirm that this is occurring in Glassfish 4.0

Comment by electricsam [ 30/Jan/14 ]

I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations:

1. contextClassLoader in various threads
2. Direct references in ThreadLocals in varous threads
3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector
4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class
5. Lots of indirect references in the ThreadLocals in the admin-listener threads

The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed).

For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue.

For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point.

Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then.

Bar.java
package test;

import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.annotation.PreDestroy;
import javax.ejb.Singleton;
import javax.ejb.Startup;

@Singleton
@Startup
public abstract class ClassLoaderCleaner {

    private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName());

    private ClassLoader loader = null;

    @PreDestroy
    protected void destroy() {
        try {
            loader = getClass().getClassLoader();
            cleanUp();
        } catch (Throwable e) {
            logger.log(Level.SEVERE, null, e);
        }
    }

    private void cleanUp() {
        Thread[] threads = getThreads();
        for (Thread thread : threads) {
            if (thread != null) {
                cleanContextClassLoader(thread);
                cleanOrb(thread);
                cleanThreadLocal(thread);

            }

        }
    }
    
        private Thread[] getThreads() {
        ThreadGroup rootGroup = Thread.currentThread().getThreadGroup();
        ThreadGroup parentGroup;
        while ((parentGroup = rootGroup.getParent()) != null) {
            rootGroup = parentGroup;
        }

        Thread[] threads = new Thread[rootGroup.activeCount()];
        while (rootGroup.enumerate(threads, true) == threads.length) {
            threads = new Thread[threads.length * 2];
        }
        return threads;
    }

    private boolean loaderRemovable(ClassLoader cl) {
        if (cl == null) {
            return false;
        }
        Object isDoneCalled = getObject(cl, "doneCalled");
        if (cl.getClass().getName().equals(loader.getClass().getName())
                && isDoneCalled instanceof Boolean && (Boolean) isDoneCalled) {
            return true;
        }
        return loader == cl;
    }

    private Field getField(Class clazz, String fieldName) {
        Field f = null;
        try {
            f = clazz.getDeclaredField(fieldName);
        } catch (NoSuchFieldException ex) {

        } catch (SecurityException ex) {
            logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex);
        }

        if (f == null) {
            Class parent = clazz.getSuperclass();
            if (parent != null) {
                f = getField(parent, fieldName);
            }
        }
        if (f != null) {
            f.setAccessible(true);
        }
        return f;
    }

    private Object getObject(Object instance, String fieldName) {
        Class clazz = instance.getClass();
        Field f = getField(clazz, fieldName);
        if (f != null) {
            try {
                return f.get(instance);
            } catch (IllegalArgumentException | IllegalAccessException ex) {
                logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex);
            }
        }
        return null;
    }

    private void cleanContextClassLoader(Thread thread) {
        if (loaderRemovable(thread.getContextClassLoader())) {
            thread.setContextClassLoader(null);
            logger.log(Level.INFO, "Cleaned context classloader {0}", thread.getName());
        }
    }

    private void cleanOrb(Thread thread) {
        Object currentWork = getObject(thread, "currentWork");
        if (currentWork != null) {
            Object orb = getObject(currentWork, "orb");
            if (orb != null) {
                Object transportManager = getObject(orb, "transportManager");
                if (transportManager != null) {
                    Thread selector = (Thread) getObject(transportManager, "selector");
                    if (selector != null && loaderRemovable(selector.getContextClassLoader())) {
                        selector.setContextClassLoader(null);
                        logger.log(Level.INFO, "Cleaned orb ref {0}", thread.getName());
                    }
                }
            }
        }
    }

    private void removeThreadLocal(Object entry, Object threadLocals, Thread thread) {
        ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent");
        if (threadLocal != null) {
            Class clazz = null;
            try {
                clazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
            } catch (ClassNotFoundException ex) {
                logger.log(Level.WARNING, null, ex);
            }
            if (clazz != null) {
                Method removeMethod = null;
                Method[] methods = clazz.getDeclaredMethods();
                if (methods != null) {
                    for (Method method : methods) {
                        if (method.getName().equals("remove")) {
                            removeMethod = method;
                            removeMethod.setAccessible(true);
                            break;
                        }
                    }
                }
                if (removeMethod != null) {
                    try {
                        removeMethod.invoke(threadLocals, threadLocal);
                        logger.log(Level.INFO, "Cleaned threadlocal {0}", thread.getName());
                    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                        logger.log(Level.SEVERE, null, ex);
                    }
                }

            }

        }
    }

    private void cleanThreadLocal(Thread thread) {
        Object threadLocals = getObject(thread, "threadLocals");
        if (threadLocals != null) {
            Object table = getObject(threadLocals, "table");
            if (table != null) {
                int size = Array.getLength(table);
                for (int i = 0; i < size; i++) {
                    Object entry = Array.get(table, i);
                    if (entry != null) {
                        Field valueField = getField(entry.getClass(), "value");
                        if (valueField != null) {
                            try {
                                Object value = valueField.get(entry);
                                if (value != null && value instanceof ClassLoader && loaderRemovable((ClassLoader) value)) {
                                    removeThreadLocal(entry, threadLocals, thread);
                                }
                            } catch (IllegalArgumentException | IllegalAccessException ex) {
                                logger.log(Level.WARNING, "Unable to get threadlocal value", ex);
                            }

                        }
                    }

                }
            }
        }
    }

}
Comment by Ed Bratt [ 20/May/14 ]

Assigned FYI ...

Comment by russgold [ 11/Jun/14 ]

If, as notabenem says, it is the KeepAlive objects preventing the garbage collection, that suggests that deploying the EAR is calling javax.rmi.CORBA.registerTarget, but undeploying is not calling javax.rmi/CORBA.unexportObject for each of the remote objects.

I'm going to look through the source to see where this is being called.

Comment by bebbo [ 04/May/16 ]

It's still present in glassfish 4.1.1.

I am using a ContextListener do null out references if contextDestroyed is invoked. For the ORB I am using

    private void fixORB() {
        try {
            // fix the ORB
            final Object globalPM = getMember(null, Class.forName("com.sun.corba.ee.spi.orb.ORB"), "globalPM");
            final Object weakCache = getMember(globalPM, "classToClassData");
            final Map<?, ?> classToClassData = (Map<?, ?>) getMember(weakCache, "map");

            LOG.info("clearing classToclassData map");
            classToClassData.clear();

        } catch (Exception ex) {
            LOG.error(ex, ex);
        }
    }

I am also patching way more locations to get a reliable undeployment/redeployment. See the full code of my ContextListener here: http://franke.ms/glassfish4_1_1_memory_leak.wiki

Generated at Wed Jan 18 08:59:31 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.