[GLASSFISH-14454] [Perf] High thread contention in felix/hk2 module while running performance benchmarks Created: 06/Nov/10  Updated: 03/Dec/12  Resolved: 09/Nov/10

Status: Closed
Project: glassfish
Component/s: other
Affects Version/s: 3.1
Fix Version/s: 3.1_ms07

Type: Bug Priority: Critical
Reporter: amitagarwal Assignee: Jagadish
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issue Links:
Dependency
blocks GLASSFISH-13573 [PERF] Huge regression with trade2 be... Closed
blocks GLASSFISH-13944 [PERF] specj enterprise 2010 benchmar... Closed
Issuezilla Id: 14,454
Tags: PSRBUG

 Description   

Please assign this to correct sub component.
While running Trade2 benchmark against 5th Nov nightly we have noticed pretty
high thread contention under following code path and this effects throughput
numbers a lot. Here is the stack trace,

1)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabeb43e08> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at org.jvnet.hk2.config.ConfigModel.getProxyType(ConfigModel.java:152)
    at org.jvnet.hk2.config.Dom.getProxyType(Dom.java:870)
    at org.jvnet.hk2.config.Dom.createProxy(Dom.java:861)
    at org.jvnet.hk2.config.ConfigModel$SingleNode.get(ConfigModel.java:471)
    at org.jvnet.hk2.config.Dom.getter(Dom.java:955)
    at org.jvnet.hk2.config.ConfigBean._getter(ConfigBean.java:179)
    at org.jvnet.hk2.config.ConfigBean.getter(ConfigBean.java:187)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:905)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy21.getResources(Unknown Source)
    at
    com.sun.enterprise.connectors.ConnectorRuntime.getResources(ConnectorRuntime.java:1399)
    at
    com.sun.enterprise.connectors.ConnectorRuntime.getResources(ConnectorRuntime.java:1395)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResources(ResourcesUtil.java:106)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResource(ResourcesUtil.java:1133)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.validateResourceAndPool(ConnectionManagerImpl.java:393)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:169)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:163)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:110)

2)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabeb43e08> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at org.jvnet.hk2.config.ConfigModel.getProxyType(ConfigModel.java:152)
    at org.jvnet.hk2.config.Dom.getProxyType(Dom.java:870)
    at org.jvnet.hk2.config.Dom.createProxy(Dom.java:861)
    at org.jvnet.hk2.config.ConfigModel$CollectionNode$1.get(ConfigModel.java:397)
    at org.jvnet.hk2.config.ConfigBean$2.get(ConfigBean.java:200)
    at java.util.AbstractList$Itr.next(AbstractList.java:345)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResources(Resources.java:105)
    at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jvnet.hk2.config.Dom.invokeDuckMethod(Dom.java:943)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:896)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResources(Unknown Source)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResourceByName(Resources.java:157)
    at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jvnet.hk2.config.Dom.invokeDuckMethod(Dom.java:943)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:896)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResourceByName(Unknown Source)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResource(ResourcesUtil.java:1136)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.validateResourceAndPool(ConnectionManagerImpl.java:393)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:169)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:163)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:110)

3)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabe8da1a0> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at com.sun.hk2.component.LazyInhabitant.loadClass(LazyInhabitant.java:1221)
    at com.sun.hk2.component.LazyInhabitant.type(LazyInhabitant.java:96)
    at org.jvnet.hk2.config.Dom.domNodeByTypeElements(Dom.java:760)
    at org.jvnet.hk2.config.ConfigModel$CollectionNode.get(ConfigModel.java:388)
    at org.jvnet.hk2.config.Dom.getter(Dom.java:955)
    at org.jvnet.hk2.config.ConfigBean._getter(ConfigBean.java:179)
    at org.jvnet.hk2.config.ConfigBean.getter(ConfigBean.java:187)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:905)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResources(Unknown Source)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResources(Resources.java:105)


 Comments   
Comment by Richard S. Hall [ 08/Nov/10 ]

Not exactly sure what can be done here. This is a sync block around a check to
findLoadedClass(), which as far as I can see must be guarded by a lock on the
class loader.

Comment by Scott Oaks [ 08/Nov/10 ]

Somewhere along the line (in the connectors code or the HK2 module), something
needs to get cached. We shouldn't have to do class lookups every time we
allocate a connection.

Comment by dochez [ 08/Nov/10 ]

I can certainly look into caching the class reference in hk2 but another optimization would be to stop
looking at configuration object (observe the ConfigBean/Dom access in the stack traces) and cache enough
of the configuration data to be able to create connection without coming back to the configuration tree.

Comment by dochez [ 08/Nov/10 ]

fixed hk2 to start caching the class object. I am still going to reassign to jagadish as configuration lookup
should not be done during user's request processing. Instead, the container should cache the necessary
configuration and listen to changes from the config events.

Comment by Jagadish [ 09/Nov/10 ]

Hi Jerome,

I can fix part (1) of the stack trace ie., to cache "Resources" configuration.

I am not sure it is possible to do anything from connector container /
Resources' duckType utility method w.r.t part (2) and (3) of stack trace as they
are searching a particular resource (by name or by type).
Both of them internally call Resources.getResources()

Comment by Jagadish [ 09/Nov/10 ]

https://glassfish-svn.dev.java.net/servlets/ReadMsg?list=commits&msgNo=25184
Provided the above fix to cache "Resources" configuration.
Fix will be available from 10th Nov 2010 nightly onwards.

Generated at Thu Sep 03 17:48:34 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.