Issue Details (XML | Word | Printable)

Key: GLASSFISH-3960
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Shalini
Reporter: Jagadish
Votes: 1
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
glassfish

Possible deadlock when using connection leak tracing and reclaim

Created: 04/Jan/08 02:31 AM   Updated: 18/Jul/13 07:29 PM   Resolved: 21/Sep/08 10:19 PM
Component/s: jca
Affects Version/s: 9.1.1
Fix Version/s: 9.1.1

Time Tracking:
Not Specified

Environment:

Operating System: All
Platform: All


Issuezilla Id: 3,960
Status Whiteboard:

911Approved

Tags:
Participants: harpreet, Jagadish, Shalini and smithbr4


 Description  « Hide

When using connection-leak-tracing and leak-reclaim, there is a possibility of
deadlock.

Two locks are used
1) Pool
2) connection-leak-lock

There is a code path in which order of acquisition of lock is inverse which may
result in dead-lock.

code-path :
Path - 1:
addResource(ResourceSpec spec, ResourceHandle h)[acquires pool-lock]
setResourceStateToBusy
startConnectionLeakTracing [acquires connection-leak-lock]

Path - 2:
potentialConnectionLeakFound [acquires connection-leak-lock]
freeResource [acquires pool-lock]




Jagadish added a comment - 09/Jan/08 03:14 AM

transferring to Shalini


Shalini added a comment - 11/Feb/08 08:35 PM

A fix is available for this.
In AbstractResourcePool.java, when the connection-leak lock is obtained, the
pool-lock should be released.


harpreet added a comment - 12/Feb/08 11:09 PM

approved for v2.1


Jagadish added a comment - 19/Jun/08 02:11 AM
      • Issue 5179 has been marked as a duplicate of this issue. ***

Jagadish added a comment - 19/Jun/08 02:11 AM

adding cc


Shalini added a comment - 21/Sep/08 10:18 PM

Fixed this issue by releasing the pool lock when connection leak lock is obtained.

Checking in AbstractResourcePool.java;
/cvs/glassfish/appserv-core/src/java/com/sun/enterprise/resource/AbstractResourcePool.java,v
<-- AbstractResourcePool.java
new revision: 1.31.2.4; previous revision: 1.31.2.3
done


smithbr4 added a comment - 18/Jul/13 07:29 PM

In our production environment today we seemed to have experienced this bug even though we are running a much newer version than when this was fixed. We are running Glassfish 3.1.2.2 build 5 on REHL 5.5

Here is the deadlock section from our thread dump.

Found one Java-level deadlock:
=============================
"p: thread-pool-1; w: 39":
waiting to lock monitor 0x00007fd7814cc878 (object 0x0000000741eac850, a java.lang.Object),
which is held by "connector-timer-proxy"
"connector-timer-proxy":
waiting to lock monitor 0x00007fd7c8578288 (object 0x000000076d6dd2a8, a com.mysql.jdbc.JDBC4ResultSet),
which is held by "p: thread-pool-1; w: 27"
"p: thread-pool-1; w: 27":
waiting to lock monitor 0x00007fd7cc0c1b90 (object 0x0000000741f47060, a com.mysql.jdbc.JDBC4Connection),
which is held by "connector-timer-proxy"

Java stack information for the threads listed above:
===================================================
"p: thread-pool-1; w: 39":
at com.sun.enterprise.resource.pool.ConnectionLeakDetector.startConnectionLeakTracing(ConnectionLeakDetector.java:113)

  • waiting to lock <0x0000000741eac850> (a java.lang.Object)
    at com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324)
    at com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:758)
    at com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632)
    at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526)
    at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
    at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
    at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
    at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
    at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
    at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:46)
    at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
    at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
    at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
    at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573)
    at org.hibernate.loader.Loader.doQuery(Loader.java:696)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
    at org.hibernate.loader.Loader.doList(Loader.java:2228)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
    at org.hibernate.loader.Loader.list(Loader.java:2120)
    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
    at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
    at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
    at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
    at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
    ... company code reference ...
    at sun.reflect.GeneratedMethodAccessor610.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
    at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
    at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
    at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
    at $Proxy531.getFranchise(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:241)
    at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
    at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
    ... company code reference ...
    at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
    at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
    at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
    at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
    at $Proxy726.clientHasExpiringRetainer(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor607.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144)
    at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174)
    at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
    at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990)
    at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
    "connector-timer-proxy":
    at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7477)
  • waiting to lock <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet)
    at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:907)
    at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2478)
  • locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement)
    at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3091)
  • locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement)
    at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584)
    at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364)
    at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556)
  • locked <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection)
    at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:409)
    at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.destroyResource(LocalTxConnectorAllocator.java:136)
    at com.sun.enterprise.resource.pool.ConnectionPool.deleteResource(ConnectionPool.java:959)
    at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.removeResource(RWLockDataStructure.java:153)
    at com.sun.enterprise.resource.pool.ConnectionPool.reclaimConnection(ConnectionPool.java:1643)
    at com.sun.enterprise.resource.pool.ConnectionLeakDetector.potentialConnectionLeakFound(ConnectionLeakDetector.java:161)
  • locked <0x0000000741eac850> (a java.lang.Object)
    at com.sun.enterprise.resource.pool.ConnectionLeakDetector.access$000(ConnectionLeakDetector.java:60)
    at com.sun.enterprise.resource.pool.ConnectionLeakDetector$ConnectionLeakTask.run(ConnectionLeakDetector.java:222)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
    "p: thread-pool-1; w: 27":
    at com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata(ConnectionImpl.java:2798)
  • waiting to lock <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection)
    at com.mysql.jdbc.Field.getStringFromBytes(Field.java:710)
    at com.mysql.jdbc.Field.getOriginalName(Field.java:652)
    at com.mysql.jdbc.ResultSetImpl.buildIndexMapping(ResultSetImpl.java:749)
    at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1108)
  • locked <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet)
    at com.mysql.jdbc.ResultSetImpl.getInt(ResultSetImpl.java:2815)
    at com.sun.gjc.spi.base.ResultSetWrapper.getInt(ResultSetWrapper.java:482)
    at org.hibernate.type.IntegerType.get(IntegerType.java:51)
    at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184)
    at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173)
    at org.hibernate.loader.Loader.getKeyFromResultSet(Loader.java:1121)
    at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:588)
    at org.hibernate.loader.Loader.doQuery(Loader.java:724)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
    at org.hibernate.loader.Loader.doList(Loader.java:2228)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
    at org.hibernate.loader.Loader.list(Loader.java:2120)
    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
    at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
    at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
    at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
    at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
    at com.pts.core.reports.ReportToolsSessionBean.getLastDisputeDateByBureau(ReportToolsSessionBean.java:775)
    at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
    at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
    at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
    at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
    at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
    at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
    at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
    at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
    at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
    at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
    at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
    at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
    ... company code reference ...
    at sun.reflect.GeneratedMethodAccessor1002.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144)
    at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174)
    at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
    at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990)
    at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
    at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
    at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)

Found 1 deadlock.