glassfish
  1. glassfish
  2. GLASSFISH-17044

[PERF] gmbal objects consuming large part of heap

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1, 3.1.1
    • Fix Version/s: None
    • Component/s: monitoring
    • Labels:
      None

      Description

      The gmbal related classes added in 3.x have contributed significantly to the heap regression usage for larger apps between 2.x and 3.x. In fact, now that other issues (notably 16747) have been fixed, these classes constitute almost all of the remaining regression. In a standard domain with specj deployed, the gmbal classes retain some 33MB of heap space (and the entire consumed heap space after startup is only 130MB).

        Issue Links

          Activity

          Scott Oaks created issue -
          Scott Oaks made changes -
          Field Original Value New Value
          Link This issue blocks GLASSFISH-16355 [ GLASSFISH-16355 ]
          scatari made changes -
          Tags 3_1-next 3_1_1-scrubbed
          Hide
          scatari added a comment -

          Targeting to be fixed in the patch release post 3.1.1.

          Show
          scatari added a comment - Targeting to be fixed in the patch release post 3.1.1.
          Hide
          Tom Mueller added a comment -

          Scott, can you please provide details on how to recreate this issue?

          Is monitoring turned on during the test?
          Do you see a relatively significant amount of memory consumed by gmbal objects with a smaller application?
          How did you determine the values that are quoted in the description?

          Show
          Tom Mueller added a comment - Scott, can you please provide details on how to recreate this issue? Is monitoring turned on during the test? Do you see a relatively significant amount of memory consumed by gmbal objects with a smaller application? How did you determine the values that are quoted in the description?
          Hide
          Scott Oaks added a comment -

          The numbers I quoted come from examining the heap dump taken after the domain has started (but not accessed); the 33MB is the size of the memory retained by the 5,619 org.glassfish.gmbal.typelib.DeclarationFactory$EvaluatedClassDeclarationImpl objects.

          Monitoring options are out-of-the box settings; the only changes to the domain are to add the necessary JDBC and JMS resources for the app (which in this case is specjappserver). My understanding from Ken is that although there is a way to disable gmbal monitoring, the necessary code is not implemented at the glassfish level (it means using a different gmbal factory to get a no-op gmbal manager). Allowing that might be a good option.

          We have only observed this on ejb-related deployments, not on web-only deployments. I'll have to see if we can get measurements from other apps.

          Show
          Scott Oaks added a comment - The numbers I quoted come from examining the heap dump taken after the domain has started (but not accessed); the 33MB is the size of the memory retained by the 5,619 org.glassfish.gmbal.typelib.DeclarationFactory$EvaluatedClassDeclarationImpl objects. Monitoring options are out-of-the box settings; the only changes to the domain are to add the necessary JDBC and JMS resources for the app (which in this case is specjappserver). My understanding from Ken is that although there is a way to disable gmbal monitoring, the necessary code is not implemented at the glassfish level (it means using a different gmbal factory to get a no-op gmbal manager). Allowing that might be a good option. We have only observed this on ejb-related deployments, not on web-only deployments. I'll have to see if we can get measurements from other apps.
          Tom Mueller made changes -
          Assignee dochez [ dochez ] Jennifer Chou [ jc129909 ]
          Component/s monitoring [ 10609 ]
          Component/s other [ 10611 ]
          Hide
          Jennifer Chou added a comment - - edited
          • If monitoring is disabled - setting mbean-enabled=false will make no difference.
          • If it is the ManagedObjectManagerFactory that is causing problems, there is only 2 places it is referenced - monitoring and web services. Since the monitoring is disabled it will not go through the code path that references ManagedObjectManagerFactory.

          I tried to reproduce the issue, but was unsuccessful.

          1. Download glassfish 3.1.1 open source edition
          2. asadmin start-domain
          3. jconsole <gf pid>
          4. MBeans > com.sun.management > Operations
          a. enter 'heap.dump.out' in p0
          b. clicked on dumpHeap
          5. Open 'heap.dump.out' in NetBeans.

          I couldn't find any gmbal classes listed under Classes. I searched for 'DeclarationFactory' and 'gmbal'.

          What am I missing? Do I need to have specj deployed?

          Show
          Jennifer Chou added a comment - - edited If monitoring is disabled - setting mbean-enabled=false will make no difference. If it is the ManagedObjectManagerFactory that is causing problems, there is only 2 places it is referenced - monitoring and web services. Since the monitoring is disabled it will not go through the code path that references ManagedObjectManagerFactory. I tried to reproduce the issue, but was unsuccessful. 1. Download glassfish 3.1.1 open source edition 2. asadmin start-domain 3. jconsole <gf pid> 4. MBeans > com.sun.management > Operations a. enter 'heap.dump.out' in p0 b. clicked on dumpHeap 5. Open 'heap.dump.out' in NetBeans. I couldn't find any gmbal classes listed under Classes. I searched for 'DeclarationFactory' and 'gmbal'. What am I missing? Do I need to have specj deployed?
          Hide
          Scott Oaks added a comment -

          You don't need specj per se, but you need some application with EJBs deployed.

          Show
          Scott Oaks added a comment - You don't need specj per se, but you need some application with EJBs deployed.
          Jennifer Chou made changes -
          Attachment MyEjb.ear [ 47645 ]
          Hide
          Jennifer Chou added a comment -

          After deploying attached simple EJB app (with a stateless session bean), the gmbal classes can be seen in the screenshot of the heap dump list.

          Show
          Jennifer Chou added a comment - After deploying attached simple EJB app (with a stateless session bean), the gmbal classes can be seen in the screenshot of the heap dump list.
          Jennifer Chou made changes -
          Attachment HeapDump-gmbal-classes.jpg [ 47646 ]
          Hide
          Jennifer Chou added a comment -

          After replacing gmbal.jar with gmbal-api-only.jar, the size and number of instances is greatly reduced. See attached screenshot - heap-dump-gmbal-api-only.

          gmbal-api-only.jar is downloaded from http://download.java.net/maven/2/org/glassfish/gmbal/gmbal-api-only/3.1.0-b001/gmbal-api-only-3.1.0-b001.jar

          Show
          Jennifer Chou added a comment - After replacing gmbal.jar with gmbal-api-only.jar, the size and number of instances is greatly reduced. See attached screenshot - heap-dump-gmbal-api-only. gmbal-api-only.jar is downloaded from http://download.java.net/maven/2/org/glassfish/gmbal/gmbal-api-only/3.1.0-b001/gmbal-api-only-3.1.0-b001.jar
          Jennifer Chou made changes -
          Attachment heap-dump-gmbal-api-only.jpg [ 47649 ]
          Hide
          Jennifer Chou added a comment -

          From Scott:

          The gmbal instances are all held by the org.glassfish.gmbal.impl.ManagedObjectManagerImpl object that is held in the ORB.

          There is a factory that produces a "null" maanged object manager impl instead of that ManagedObjectManagerImpl, so if we could arrange for the ORB to use that factory when we don't want the overhead of gmbal, that would solve the issue.

          Show
          Jennifer Chou added a comment - From Scott: The gmbal instances are all held by the org.glassfish.gmbal.impl.ManagedObjectManagerImpl object that is held in the ORB. There is a factory that produces a "null" maanged object manager impl instead of that ManagedObjectManagerImpl, so if we could arrange for the ORB to use that factory when we don't want the overhead of gmbal, that would solve the issue.
          Joe Di Pol made changes -
          Tags 3_1-next 3_1_1-scrubbed 3_1-next 3_1_1-scrubbed 3_1_2-review
          Hide
          Jennifer Chou added a comment - - edited

          The fix should be in ORB to defer the gmbal API calls until there is a JMX client connection.

          http://java.net/jira/browse/GLASSFISH_CORBA-5

          Show
          Jennifer Chou added a comment - - edited The fix should be in ORB to defer the gmbal API calls until there is a JMX client connection. http://java.net/jira/browse/GLASSFISH_CORBA-5
          Jennifer Chou made changes -
          Link This issue depends on GLASSFISH_CORBA-5 [ GLASSFISH_CORBA-5 ]
          Hide
          Jennifer Chou added a comment -

          The fix should be in metro and WebServicesContainer to defer the gmbal API call until there is a JMX client connection.

          http://java.net/jira/browse/METRO-17

          Show
          Jennifer Chou added a comment - The fix should be in metro and WebServicesContainer to defer the gmbal API call until there is a JMX client connection. http://java.net/jira/browse/METRO-17
          Jennifer Chou made changes -
          Link This issue depends on METRO-17 [ METRO-17 ]
          Hide
          Jennifer Chou added a comment -

          Transfer to Scott Oaks. This is an umbrella bug to track the 2 issues in ORB and Metro.

          Show
          Jennifer Chou added a comment - Transfer to Scott Oaks. This is an umbrella bug to track the 2 issues in ORB and Metro.
          Jennifer Chou made changes -
          Assignee Jennifer Chou [ jc129909 ] Scott Oaks [ sdo ]
          Hide
          Joe Di Pol added a comment -

          We've done all we plan on doing for 3.1.2 (See linked Metro bug). The ORB fix will have to wait for a subsequent release.

          Show
          Joe Di Pol added a comment - We've done all we plan on doing for 3.1.2 (See linked Metro bug). The ORB fix will have to wait for a subsequent release.
          Joe Di Pol made changes -
          Tags 3_1-next 3_1_1-scrubbed 3_1_2-review 3_1-next 3_1_1-scrubbed 3_1_2-exclude
          Scott Oaks made changes -
          Tags 3_1-next 3_1_1-scrubbed 3_1_2-exclude 3_1-next 3_1_1-scrubbed 3_1_2-exclude PSRBUG

            People

            • Assignee:
              Scott Oaks
              Reporter:
              Scott Oaks
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: