Issue Details (XML | Word | Printable)

Key: GLASSFISH-14454
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Jagadish
Reporter: amitagarwal
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
glassfish

[Perf] High thread contention in felix/hk2 module while running performance benchmarks

Created: 06/Nov/10 07:29 PM   Updated: 03/Dec/12 11:06 PM   Resolved: 09/Nov/10 07:33 AM
Component/s: other
Affects Version/s: 3.1
Fix Version/s: 3.1_ms07

Time Tracking:
Not Specified

Environment:

Operating System: All
Platform: All

Issue Links:
Dependency
 

Issuezilla Id: 14,454
Tags: PSRBUG
Participants: amitagarwal, dochez, Jagadish, Richard S. Hall and Scott Oaks


 Description  « Hide

Please assign this to correct sub component.
While running Trade2 benchmark against 5th Nov nightly we have noticed pretty
high thread contention under following code path and this effects throughput
numbers a lot. Here is the stack trace,

1)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabeb43e08> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at org.jvnet.hk2.config.ConfigModel.getProxyType(ConfigModel.java:152)
    at org.jvnet.hk2.config.Dom.getProxyType(Dom.java:870)
    at org.jvnet.hk2.config.Dom.createProxy(Dom.java:861)
    at org.jvnet.hk2.config.ConfigModel$SingleNode.get(ConfigModel.java:471)
    at org.jvnet.hk2.config.Dom.getter(Dom.java:955)
    at org.jvnet.hk2.config.ConfigBean._getter(ConfigBean.java:179)
    at org.jvnet.hk2.config.ConfigBean.getter(ConfigBean.java:187)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:905)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy21.getResources(Unknown Source)
    at
    com.sun.enterprise.connectors.ConnectorRuntime.getResources(ConnectorRuntime.java:1399)
    at
    com.sun.enterprise.connectors.ConnectorRuntime.getResources(ConnectorRuntime.java:1395)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResources(ResourcesUtil.java:106)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResource(ResourcesUtil.java:1133)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.validateResourceAndPool(ConnectionManagerImpl.java:393)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:169)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:163)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:110)

2)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabeb43e08> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at org.jvnet.hk2.config.ConfigModel.getProxyType(ConfigModel.java:152)
    at org.jvnet.hk2.config.Dom.getProxyType(Dom.java:870)
    at org.jvnet.hk2.config.Dom.createProxy(Dom.java:861)
    at org.jvnet.hk2.config.ConfigModel$CollectionNode$1.get(ConfigModel.java:397)
    at org.jvnet.hk2.config.ConfigBean$2.get(ConfigBean.java:200)
    at java.util.AbstractList$Itr.next(AbstractList.java:345)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResources(Resources.java:105)
    at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jvnet.hk2.config.Dom.invokeDuckMethod(Dom.java:943)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:896)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResources(Unknown Source)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResourceByName(Resources.java:157)
    at sun.reflect.GeneratedMethodAccessor63.invoke(Unknown Source)
    at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.jvnet.hk2.config.Dom.invokeDuckMethod(Dom.java:943)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:896)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResourceByName(Unknown Source)
    at
    com.sun.enterprise.connectors.util.ResourcesUtil.getResource(ResourcesUtil.java:1136)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.validateResourceAndPool(ConnectionManagerImpl.java:393)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:169)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:163)
    at
    com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:158)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:110)

3)
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1756)

  • waiting to lock <0x00002aaabe8da1a0> (a
    org.apache.felix.framework.ModuleImpl$ModuleClassLoaderJava5)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    at com.sun.hk2.component.LazyInhabitant.loadClass(LazyInhabitant.java:1221)
    at com.sun.hk2.component.LazyInhabitant.type(LazyInhabitant.java:96)
    at org.jvnet.hk2.config.Dom.domNodeByTypeElements(Dom.java:760)
    at org.jvnet.hk2.config.ConfigModel$CollectionNode.get(ConfigModel.java:388)
    at org.jvnet.hk2.config.Dom.getter(Dom.java:955)
    at org.jvnet.hk2.config.ConfigBean._getter(ConfigBean.java:179)
    at org.jvnet.hk2.config.ConfigBean.getter(ConfigBean.java:187)
    at org.jvnet.hk2.config.Dom.invoke(Dom.java:905)
    at
    org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:119)
    at $Proxy91.getResources(Unknown Source)
    at
    com.sun.enterprise.config.serverbeans.Resources$Duck.getResources(Resources.java:105)


Jagadish added a comment - 09/Nov/10 07:33 AM

https://glassfish-svn.dev.java.net/servlets/ReadMsg?list=commits&msgNo=25184
Provided the above fix to cache "Resources" configuration.
Fix will be available from 10th Nov 2010 nightly onwards.


Jagadish added a comment - 09/Nov/10 01:55 AM

Hi Jerome,

I can fix part (1) of the stack trace ie., to cache "Resources" configuration.

I am not sure it is possible to do anything from connector container /
Resources' duckType utility method w.r.t part (2) and (3) of stack trace as they
are searching a particular resource (by name or by type).
Both of them internally call Resources.getResources()


dochez added a comment - 08/Nov/10 02:05 PM

fixed hk2 to start caching the class object. I am still going to reassign to jagadish as configuration lookup
should not be done during user's request processing. Instead, the container should cache the necessary
configuration and listen to changes from the config events.


dochez added a comment - 08/Nov/10 10:08 AM

I can certainly look into caching the class reference in hk2 but another optimization would be to stop
looking at configuration object (observe the ConfigBean/Dom access in the stack traces) and cache enough
of the configuration data to be able to create connection without coming back to the configuration tree.


Scott Oaks added a comment - 08/Nov/10 08:05 AM

Somewhere along the line (in the connectors code or the HK2 module), something
needs to get cached. We shouldn't have to do class lookups every time we
allocate a connection.


Richard S. Hall added a comment - 08/Nov/10 08:02 AM

Not exactly sure what can be done here. This is a sync block around a check to
findLoadedClass(), which as far as I can see must be guarded by a lock on the
class loader.