Issue Details (XML | Word | Printable)

Key: GLASSFISH-17449
Type: Bug Bug
Status: Open Open
Priority: Critical Critical
Assignee: Harshad Vilekar
Reporter: jelinj14
Votes: 11
Watchers: 9
Operations

If you were logged in you would be able to see more operations.
glassfish

Redeploy memory leak

Created: 20/Oct/11 11:16 AM   Updated: 01/Apr/14 10:50 PM
Component/s: orb
Affects Version/s: 3.1.1
Fix Version/s: None

Time Tracking:
Original Estimate: 1 day
Original Estimate - 1 day
Remaining Estimate: 1 day
Remaining Estimate - 1 day
Time Spent: Not Specified
Time Spent - Not Specified

File Attachments: 1. File keepalive-ref.tiff (269 kB) 03/Nov/11 06:18 PM - Cheng Fang
2. File runtest.sh (0.4 kB) 21/Oct/11 05:17 PM - Tim Quinn

Environment:

windows 7, 64bit


Tags: 3_1_2-exclude 4_0_1-evangelists 4_0_1-review memoryleak orb-review redeploy
Participants: Cheng Fang, electricsam, Harshad Vilekar, Hong Zhang, jelinj14, notabenem and Tim Quinn


 Description  « Hide

Redeployment causes memory leaks, using jmap and jhat can resolve for example: org.glassfish.javaee.full.deployment.EarClassLoader this class has instance each deploy and undeploy even if app is undeployed



electricsam added a comment - 30/Jan/14 05:08 PM - edited

I've investigated a workaround until this is fixed. I found references to the current EarClassLoader (to be undeployed) in the following locations:

1. contextClassLoader in various threads
2. Direct references in ThreadLocals in varous threads
3. A contextClassLoader reference in the selector thread of the following field heirerchy: thread.orb.transportManager.selector
4. A contextClassLoader reference in the static resyncThread field of the com.sun.jts.CosTransactions.RecoveryManager class
5. Lots of indirect references in the ThreadLocals in the admin-listener threads

The below listed code will attempt to clean up items 1-3 when an app is undeployed (or redeployed).

For item 4, I was unable to get a reference to the RecoveryManager that had a contextClassLoader reference, but it is static and not as bad as a thread pool issue.

For item 5, the number of references and the object hierarchy depth made a workaround difficult. My best attempt was to limit the pool size for the admin-thread-pool to 5 with a min of 0 and a timeout of 0 (Although, I never see the pool shrink after it reaches capacity). There is still a leak here, but does not seem to grow past a certain point.

Hopefully this will help the GF dev that looks at this issue or any GF users with the same problem until then.

Bar.java
package test;

import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.logging.Level;
import java.util.logging.Logger;
import javax.annotation.PreDestroy;
import javax.ejb.Singleton;
import javax.ejb.Startup;

@Singleton
@Startup
public abstract class ClassLoaderCleaner {

    private static final Logger logger = Logger.getLogger(ClassLoaderCleaner.class.getName());

    private ClassLoader loader = null;

    @PreDestroy
    protected void destroy() {
        try {
            loader = getClass().getClassLoader();
            cleanUp();
        } catch (Throwable e) {
            logger.log(Level.SEVERE, null, e);
        }
    }

    private void cleanUp() {
        Thread[] threads = getThreads();
        for (Thread thread : threads) {
            if (thread != null) {
                cleanContextClassLoader(thread);
                cleanOrb(thread);
                cleanThreadLocal(thread);

            }

        }
    }
    
        private Thread[] getThreads() {
        ThreadGroup rootGroup = Thread.currentThread().getThreadGroup();
        ThreadGroup parentGroup;
        while ((parentGroup = rootGroup.getParent()) != null) {
            rootGroup = parentGroup;
        }

        Thread[] threads = new Thread[rootGroup.activeCount()];
        while (rootGroup.enumerate(threads, true) == threads.length) {
            threads = new Thread[threads.length * 2];
        }
        return threads;
    }

    private boolean loaderRemovable(ClassLoader cl) {
        if (cl == null) {
            return false;
        }
        Object isDoneCalled = getObject(cl, "doneCalled");
        if (cl.getClass().getName().equals(loader.getClass().getName())
                && isDoneCalled instanceof Boolean && (Boolean) isDoneCalled) {
            return true;
        }
        return loader == cl;
    }

    private Field getField(Class clazz, String fieldName) {
        Field f = null;
        try {
            f = clazz.getDeclaredField(fieldName);
        } catch (NoSuchFieldException ex) {

        } catch (SecurityException ex) {
            logger.log(Level.WARNING, "Unable to get field " + fieldName + " on " + clazz.getName(), ex);
        }

        if (f == null) {
            Class parent = clazz.getSuperclass();
            if (parent != null) {
                f = getField(parent, fieldName);
            }
        }
        if (f != null) {
            f.setAccessible(true);
        }
        return f;
    }

    private Object getObject(Object instance, String fieldName) {
        Class clazz = instance.getClass();
        Field f = getField(clazz, fieldName);
        if (f != null) {
            try {
                return f.get(instance);
            } catch (IllegalArgumentException | IllegalAccessException ex) {
                logger.log(Level.WARNING, "Unable to get " + fieldName + " on " + clazz.getName(), ex);
            }
        }
        return null;
    }

    private void cleanContextClassLoader(Thread thread) {
        if (loaderRemovable(thread.getContextClassLoader())) {
            thread.setContextClassLoader(null);
            logger.log(Level.INFO, "Cleaned context classloader {0}", thread.getName());
        }
    }

    private void cleanOrb(Thread thread) {
        Object currentWork = getObject(thread, "currentWork");
        if (currentWork != null) {
            Object orb = getObject(currentWork, "orb");
            if (orb != null) {
                Object transportManager = getObject(orb, "transportManager");
                if (transportManager != null) {
                    Thread selector = (Thread) getObject(transportManager, "selector");
                    if (selector != null && loaderRemovable(selector.getContextClassLoader())) {
                        selector.setContextClassLoader(null);
                        logger.log(Level.INFO, "Cleaned orb ref {0}", thread.getName());
                    }
                }
            }
        }
    }

    private void removeThreadLocal(Object entry, Object threadLocals, Thread thread) {
        ThreadLocal threadLocal = (ThreadLocal) getObject(entry, "referent");
        if (threadLocal != null) {
            Class clazz = null;
            try {
                clazz = Class.forName("java.lang.ThreadLocal$ThreadLocalMap");
            } catch (ClassNotFoundException ex) {
                logger.log(Level.WARNING, null, ex);
            }
            if (clazz != null) {
                Method removeMethod = null;
                Method[] methods = clazz.getDeclaredMethods();
                if (methods != null) {
                    for (Method method : methods) {
                        if (method.getName().equals("remove")) {
                            removeMethod = method;
                            removeMethod.setAccessible(true);
                            break;
                        }
                    }
                }
                if (removeMethod != null) {
                    try {
                        removeMethod.invoke(threadLocals, threadLocal);
                        logger.log(Level.INFO, "Cleaned threadlocal {0}", thread.getName());
                    } catch (IllegalAccessException | IllegalArgumentException | InvocationTargetException ex) {
                        logger.log(Level.SEVERE, null, ex);
                    }
                }

            }

        }
    }

    private void cleanThreadLocal(Thread thread) {
        Object threadLocals = getObject(thread, "threadLocals");
        if (threadLocals != null) {
            Object table = getObject(threadLocals, "table");
            if (table != null) {
                int size = Array.getLength(table);
                for (int i = 0; i < size; i++) {
                    Object entry = Array.get(table, i);
                    if (entry != null) {
                        Field valueField = getField(entry.getClass(), "value");
                        if (valueField != null) {
                            try {
                                Object value = valueField.get(entry);
                                if (value != null && value instanceof ClassLoader && loaderRemovable((ClassLoader) value)) {
                                    removeThreadLocal(entry, threadLocals, thread);
                                }
                            } catch (IllegalArgumentException | IllegalAccessException ex) {
                                logger.log(Level.WARNING, "Unable to get threadlocal value", ex);
                            }

                        }
                    }

                }
            }
        }
    }

}

electricsam added a comment - 15/Jan/14 12:00 PM

I too can confirm that this is occurring in Glassfish 4.0


notabenem added a comment - 02/Dec/13 12:10 PM

Just run into this on Glassfish 4: com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive prevents the EAR from being garbage collected.
This pretty much eliminates the possibility of a reliable reloading mechanism (Continuous delivery)


Cheng Fang added a comment - 03/Nov/11 06:21 PM

re-assign to orb team to evaluate if we can reset the context class loader when creating the KeepAlive thread, to avoid inheriting the app class loader from the parent thread.


Cheng Fang added a comment - 03/Nov/11 06:18 PM

Attached a screen shot of reference from com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive


Cheng Fang added a comment - 03/Nov/11 06:15 PM

Once issue 17468 is resolved, it will help eliminate some causes of leaks.

Another source of leaks, when deploying apps with remote ejb, is from orb layer. An instance of com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive holds a reference to the EarClassLoader after undeploy.


Cheng Fang added a comment - 25/Oct/11 09:48 PM

Related to http://java.net/jira/browse/GLASSFISH-17468 (WebappClassLoader leak after undeployment)


Tim Quinn added a comment - 25/Oct/11 09:15 PM

Cheng,

Perhaps something has changed between 3.1.1 and 4.0 (trunk) then. (I do not have a chance right now to try it myself on the trunk.)

Can whatever the change is be back-ported to 3.1.2? (I realize that requires understanding what change(s) are responsible for the better behavior in 4.0).


Cheng Fang added a comment - 25/Oct/11 07:53 PM

Hi Tim, I tried your script with my test EAR on trunk build, and had different results. After finishing your script (100 deploy/undeploy), the result page shows only 3 instances of EarClassLoader.

I then tried deployed/undeploy one by one, and also shows any time after GC, the count is at most 3. The small number of left over instances could be related to annotation processing tasks.


Tim Quinn added a comment - 21/Oct/11 05:58 PM

There might well be other issues besides the one I found so far. Once that one is resolved we will look into this again for other leaks.


jelinj14 added a comment - 21/Oct/11 05:44 PM

When the application is bigger, (I have about 70 stateless session beans on EJB side) it causes big memory leak, every deploy is about 90MB leaks. But I'm not sure whether is it only EarClassLoader.
Thanks for resolve.


Tim Quinn added a comment - 21/Oct/11 05:17 PM

Attached runtest.sh for repeatedly deploying an app and monitoring memory usage.


Tim Quinn added a comment - 21/Oct/11 05:16 PM

I was able to reproduce the problem.

I deployed a simple EAR (containing an EJB) 100 times, and saw 100 EarClassLoader instances remaining live, even after forcing GCs using jconsole.

(I will attach the simple shell script I used.)

Run the script, then browse to http://localhost:7000.

At least part of the problem seems to be caused by EJB monitoring retaining a reference to each StatelessSessionContainer, which in turn has a reference to EarClassLoader.

I am transferring this to the EJB team for investigation into cleaning up the references from monitoring when a container is retired.


Hong Zhang added a comment - 21/Oct/11 02:38 PM

Assign to tim to take a look as he has done a lot of work of tracking down memory leak in 3.1.1.