glassfish
  1. glassfish
  2. GLASSFISH-17044

[PERF] gmbal objects consuming large part of heap

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1, 3.1.1
    • Fix Version/s: None
    • Component/s: monitoring
    • Labels:
      None

      Description

      The gmbal related classes added in 3.x have contributed significantly to the heap regression usage for larger apps between 2.x and 3.x. In fact, now that other issues (notably 16747) have been fixed, these classes constitute almost all of the remaining regression. In a standard domain with specj deployed, the gmbal classes retain some 33MB of heap space (and the entire consumed heap space after startup is only 130MB).

        Issue Links

          Activity

          Hide
          scatari added a comment -

          Targeting to be fixed in the patch release post 3.1.1.

          Show
          scatari added a comment - Targeting to be fixed in the patch release post 3.1.1.
          Hide
          Tom Mueller added a comment -

          Scott, can you please provide details on how to recreate this issue?

          Is monitoring turned on during the test?
          Do you see a relatively significant amount of memory consumed by gmbal objects with a smaller application?
          How did you determine the values that are quoted in the description?

          Show
          Tom Mueller added a comment - Scott, can you please provide details on how to recreate this issue? Is monitoring turned on during the test? Do you see a relatively significant amount of memory consumed by gmbal objects with a smaller application? How did you determine the values that are quoted in the description?
          Hide
          Scott Oaks added a comment -

          The numbers I quoted come from examining the heap dump taken after the domain has started (but not accessed); the 33MB is the size of the memory retained by the 5,619 org.glassfish.gmbal.typelib.DeclarationFactory$EvaluatedClassDeclarationImpl objects.

          Monitoring options are out-of-the box settings; the only changes to the domain are to add the necessary JDBC and JMS resources for the app (which in this case is specjappserver). My understanding from Ken is that although there is a way to disable gmbal monitoring, the necessary code is not implemented at the glassfish level (it means using a different gmbal factory to get a no-op gmbal manager). Allowing that might be a good option.

          We have only observed this on ejb-related deployments, not on web-only deployments. I'll have to see if we can get measurements from other apps.

          Show
          Scott Oaks added a comment - The numbers I quoted come from examining the heap dump taken after the domain has started (but not accessed); the 33MB is the size of the memory retained by the 5,619 org.glassfish.gmbal.typelib.DeclarationFactory$EvaluatedClassDeclarationImpl objects. Monitoring options are out-of-the box settings; the only changes to the domain are to add the necessary JDBC and JMS resources for the app (which in this case is specjappserver). My understanding from Ken is that although there is a way to disable gmbal monitoring, the necessary code is not implemented at the glassfish level (it means using a different gmbal factory to get a no-op gmbal manager). Allowing that might be a good option. We have only observed this on ejb-related deployments, not on web-only deployments. I'll have to see if we can get measurements from other apps.
          Hide
          Jennifer Chou added a comment - - edited
          • If monitoring is disabled - setting mbean-enabled=false will make no difference.
          • If it is the ManagedObjectManagerFactory that is causing problems, there is only 2 places it is referenced - monitoring and web services. Since the monitoring is disabled it will not go through the code path that references ManagedObjectManagerFactory.

          I tried to reproduce the issue, but was unsuccessful.

          1. Download glassfish 3.1.1 open source edition
          2. asadmin start-domain
          3. jconsole <gf pid>
          4. MBeans > com.sun.management > Operations
          a. enter 'heap.dump.out' in p0
          b. clicked on dumpHeap
          5. Open 'heap.dump.out' in NetBeans.

          I couldn't find any gmbal classes listed under Classes. I searched for 'DeclarationFactory' and 'gmbal'.

          What am I missing? Do I need to have specj deployed?

          Show
          Jennifer Chou added a comment - - edited If monitoring is disabled - setting mbean-enabled=false will make no difference. If it is the ManagedObjectManagerFactory that is causing problems, there is only 2 places it is referenced - monitoring and web services. Since the monitoring is disabled it will not go through the code path that references ManagedObjectManagerFactory. I tried to reproduce the issue, but was unsuccessful. 1. Download glassfish 3.1.1 open source edition 2. asadmin start-domain 3. jconsole <gf pid> 4. MBeans > com.sun.management > Operations a. enter 'heap.dump.out' in p0 b. clicked on dumpHeap 5. Open 'heap.dump.out' in NetBeans. I couldn't find any gmbal classes listed under Classes. I searched for 'DeclarationFactory' and 'gmbal'. What am I missing? Do I need to have specj deployed?
          Hide
          Scott Oaks added a comment -

          You don't need specj per se, but you need some application with EJBs deployed.

          Show
          Scott Oaks added a comment - You don't need specj per se, but you need some application with EJBs deployed.
          Hide
          Jennifer Chou added a comment -

          After deploying attached simple EJB app (with a stateless session bean), the gmbal classes can be seen in the screenshot of the heap dump list.

          Show
          Jennifer Chou added a comment - After deploying attached simple EJB app (with a stateless session bean), the gmbal classes can be seen in the screenshot of the heap dump list.
          Hide
          Jennifer Chou added a comment -

          After replacing gmbal.jar with gmbal-api-only.jar, the size and number of instances is greatly reduced. See attached screenshot - heap-dump-gmbal-api-only.

          gmbal-api-only.jar is downloaded from http://download.java.net/maven/2/org/glassfish/gmbal/gmbal-api-only/3.1.0-b001/gmbal-api-only-3.1.0-b001.jar

          Show
          Jennifer Chou added a comment - After replacing gmbal.jar with gmbal-api-only.jar, the size and number of instances is greatly reduced. See attached screenshot - heap-dump-gmbal-api-only. gmbal-api-only.jar is downloaded from http://download.java.net/maven/2/org/glassfish/gmbal/gmbal-api-only/3.1.0-b001/gmbal-api-only-3.1.0-b001.jar
          Hide
          Jennifer Chou added a comment -

          From Scott:

          The gmbal instances are all held by the org.glassfish.gmbal.impl.ManagedObjectManagerImpl object that is held in the ORB.

          There is a factory that produces a "null" maanged object manager impl instead of that ManagedObjectManagerImpl, so if we could arrange for the ORB to use that factory when we don't want the overhead of gmbal, that would solve the issue.

          Show
          Jennifer Chou added a comment - From Scott: The gmbal instances are all held by the org.glassfish.gmbal.impl.ManagedObjectManagerImpl object that is held in the ORB. There is a factory that produces a "null" maanged object manager impl instead of that ManagedObjectManagerImpl, so if we could arrange for the ORB to use that factory when we don't want the overhead of gmbal, that would solve the issue.
          Hide
          Jennifer Chou added a comment - - edited

          The fix should be in ORB to defer the gmbal API calls until there is a JMX client connection.

          http://java.net/jira/browse/GLASSFISH_CORBA-5

          Show
          Jennifer Chou added a comment - - edited The fix should be in ORB to defer the gmbal API calls until there is a JMX client connection. http://java.net/jira/browse/GLASSFISH_CORBA-5
          Hide
          Jennifer Chou added a comment -

          The fix should be in metro and WebServicesContainer to defer the gmbal API call until there is a JMX client connection.

          http://java.net/jira/browse/METRO-17

          Show
          Jennifer Chou added a comment - The fix should be in metro and WebServicesContainer to defer the gmbal API call until there is a JMX client connection. http://java.net/jira/browse/METRO-17
          Hide
          Jennifer Chou added a comment -

          Transfer to Scott Oaks. This is an umbrella bug to track the 2 issues in ORB and Metro.

          Show
          Jennifer Chou added a comment - Transfer to Scott Oaks. This is an umbrella bug to track the 2 issues in ORB and Metro.
          Hide
          Joe Di Pol added a comment -

          We've done all we plan on doing for 3.1.2 (See linked Metro bug). The ORB fix will have to wait for a subsequent release.

          Show
          Joe Di Pol added a comment - We've done all we plan on doing for 3.1.2 (See linked Metro bug). The ORB fix will have to wait for a subsequent release.

            People

            • Assignee:
              Scott Oaks
              Reporter:
              Scott Oaks
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: