glassfish
  1. glassfish
  2. GLASSFISH-3960

Possible deadlock when using connection leak tracing and reclaim

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 9.1.1
    • Fix Version/s: 9.1.1
    • Component/s: jca
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • Issuezilla Id:
      3,960
    • Status Whiteboard:
      Hide

      911Approved

      Show
      911Approved

      Description

      When using connection-leak-tracing and leak-reclaim, there is a possibility of
      deadlock.

      Two locks are used
      1) Pool
      2) connection-leak-lock

      There is a code path in which order of acquisition of lock is inverse which may
      result in dead-lock.

      code-path :
      Path - 1:
      addResource(ResourceSpec spec, ResourceHandle h)[acquires pool-lock]
      setResourceStateToBusy
      startConnectionLeakTracing [acquires connection-leak-lock]

      Path - 2:
      potentialConnectionLeakFound [acquires connection-leak-lock]
      freeResource [acquires pool-lock]

        Activity

        Show
        Jagadish added a comment - File : https://glassfish.dev.java.net/source/browse/glassfish/appserv-core/src/java/com/sun/enterprise/resource/AbstractResourcePool.java
        Hide
        Jagadish added a comment -

        transferring to Shalini

        Show
        Jagadish added a comment - transferring to Shalini
        Hide
        Shalini added a comment -

        A fix is available for this.
        In AbstractResourcePool.java, when the connection-leak lock is obtained, the
        pool-lock should be released.

        Show
        Shalini added a comment - A fix is available for this. In AbstractResourcePool.java, when the connection-leak lock is obtained, the pool-lock should be released.
        Hide
        harpreet added a comment -

        approved for v2.1

        Show
        harpreet added a comment - approved for v2.1
        Hide
        Jagadish added a comment -
            • Issue 5179 has been marked as a duplicate of this issue. ***
        Show
        Jagadish added a comment - Issue 5179 has been marked as a duplicate of this issue. ***
        Hide
        Jagadish added a comment -

        adding cc

        Show
        Jagadish added a comment - adding cc
        Hide
        Shalini added a comment -

        Fixed this issue by releasing the pool lock when connection leak lock is obtained.

        Checking in AbstractResourcePool.java;
        /cvs/glassfish/appserv-core/src/java/com/sun/enterprise/resource/AbstractResourcePool.java,v
        <-- AbstractResourcePool.java
        new revision: 1.31.2.4; previous revision: 1.31.2.3
        done

        Show
        Shalini added a comment - Fixed this issue by releasing the pool lock when connection leak lock is obtained. Checking in AbstractResourcePool.java; /cvs/glassfish/appserv-core/src/java/com/sun/enterprise/resource/AbstractResourcePool.java,v <-- AbstractResourcePool.java new revision: 1.31.2.4; previous revision: 1.31.2.3 done
        Hide
        smithbr4 added a comment -

        In our production environment today we seemed to have experienced this bug even though we are running a much newer version than when this was fixed. We are running Glassfish 3.1.2.2 build 5 on REHL 5.5

        Here is the deadlock section from our thread dump.

        Found one Java-level deadlock:
        =============================
        "p: thread-pool-1; w: 39":
        waiting to lock monitor 0x00007fd7814cc878 (object 0x0000000741eac850, a java.lang.Object),
        which is held by "connector-timer-proxy"
        "connector-timer-proxy":
        waiting to lock monitor 0x00007fd7c8578288 (object 0x000000076d6dd2a8, a com.mysql.jdbc.JDBC4ResultSet),
        which is held by "p: thread-pool-1; w: 27"
        "p: thread-pool-1; w: 27":
        waiting to lock monitor 0x00007fd7cc0c1b90 (object 0x0000000741f47060, a com.mysql.jdbc.JDBC4Connection),
        which is held by "connector-timer-proxy"

        Java stack information for the threads listed above:
        ===================================================
        "p: thread-pool-1; w: 39":
        at com.sun.enterprise.resource.pool.ConnectionLeakDetector.startConnectionLeakTracing(ConnectionLeakDetector.java:113)

        • waiting to lock <0x0000000741eac850> (a java.lang.Object)
          at com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324)
          at com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:758)
          at com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632)
          at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526)
          at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
          at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
          at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
          at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
          at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
          at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
          at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
          at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
          at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
          at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:46)
          at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
          at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
          at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
          at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573)
          at org.hibernate.loader.Loader.doQuery(Loader.java:696)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
          at org.hibernate.loader.Loader.doList(Loader.java:2228)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
          at org.hibernate.loader.Loader.list(Loader.java:2120)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
          at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
          ... company code reference ...
          at sun.reflect.GeneratedMethodAccessor610.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
          at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
          at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
          at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
          at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
          at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
          at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
          at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
          at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
          at $Proxy531.getFranchise(Unknown Source)
          at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:241)
          at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152)
          at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227)
          ... company code reference ...
          at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
          at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
          at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
          at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
          at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
          at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
          at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
          at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
          at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
          at $Proxy726.clientHasExpiringRetainer(Unknown Source)
          at sun.reflect.GeneratedMethodAccessor607.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144)
          at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174)
          at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
          at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990)
          at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
          at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
          at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
          "connector-timer-proxy":
          at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7477)
        • waiting to lock <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet)
          at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:907)
          at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2478)
        • locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement)
          at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3091)
        • locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement)
          at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584)
          at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364)
          at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556)
        • locked <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection)
          at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:409)
          at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.destroyResource(LocalTxConnectorAllocator.java:136)
          at com.sun.enterprise.resource.pool.ConnectionPool.deleteResource(ConnectionPool.java:959)
          at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.removeResource(RWLockDataStructure.java:153)
          at com.sun.enterprise.resource.pool.ConnectionPool.reclaimConnection(ConnectionPool.java:1643)
          at com.sun.enterprise.resource.pool.ConnectionLeakDetector.potentialConnectionLeakFound(ConnectionLeakDetector.java:161)
        • locked <0x0000000741eac850> (a java.lang.Object)
          at com.sun.enterprise.resource.pool.ConnectionLeakDetector.access$000(ConnectionLeakDetector.java:60)
          at com.sun.enterprise.resource.pool.ConnectionLeakDetector$ConnectionLeakTask.run(ConnectionLeakDetector.java:222)
          at java.util.TimerThread.mainLoop(Timer.java:512)
          at java.util.TimerThread.run(Timer.java:462)
          "p: thread-pool-1; w: 27":
          at com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata(ConnectionImpl.java:2798)
        • waiting to lock <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection)
          at com.mysql.jdbc.Field.getStringFromBytes(Field.java:710)
          at com.mysql.jdbc.Field.getOriginalName(Field.java:652)
          at com.mysql.jdbc.ResultSetImpl.buildIndexMapping(ResultSetImpl.java:749)
          at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1108)
        • locked <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet)
          at com.mysql.jdbc.ResultSetImpl.getInt(ResultSetImpl.java:2815)
          at com.sun.gjc.spi.base.ResultSetWrapper.getInt(ResultSetWrapper.java:482)
          at org.hibernate.type.IntegerType.get(IntegerType.java:51)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184)
          at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173)
          at org.hibernate.loader.Loader.getKeyFromResultSet(Loader.java:1121)
          at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:588)
          at org.hibernate.loader.Loader.doQuery(Loader.java:724)
          at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
          at org.hibernate.loader.Loader.doList(Loader.java:2228)
          at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125)
          at org.hibernate.loader.Loader.list(Loader.java:2120)
          at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401)
          at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361)
          at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
          at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148)
          at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
          at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67)
          at com.pts.core.reports.ReportToolsSessionBean.getLastDisputeDateByBureau(ReportToolsSessionBean.java:775)
          at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
          at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
          at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388)
          at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
          at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
          at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
          at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
          at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
          at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360)
          at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348)
          at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206)
          at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79)
          ... company code reference ...
          at sun.reflect.GeneratedMethodAccessor1002.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144)
          at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174)
          at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
          at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990)
          at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
          at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
          at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
          at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)

        Found 1 deadlock.

        Show
        smithbr4 added a comment - In our production environment today we seemed to have experienced this bug even though we are running a much newer version than when this was fixed. We are running Glassfish 3.1.2.2 build 5 on REHL 5.5 Here is the deadlock section from our thread dump. Found one Java-level deadlock: ============================= "p: thread-pool-1; w: 39": waiting to lock monitor 0x00007fd7814cc878 (object 0x0000000741eac850, a java.lang.Object), which is held by "connector-timer-proxy" "connector-timer-proxy": waiting to lock monitor 0x00007fd7c8578288 (object 0x000000076d6dd2a8, a com.mysql.jdbc.JDBC4ResultSet), which is held by "p: thread-pool-1; w: 27" "p: thread-pool-1; w: 27": waiting to lock monitor 0x00007fd7cc0c1b90 (object 0x0000000741f47060, a com.mysql.jdbc.JDBC4Connection), which is held by "connector-timer-proxy" Java stack information for the threads listed above: =================================================== "p: thread-pool-1; w: 39": at com.sun.enterprise.resource.pool.ConnectionLeakDetector.startConnectionLeakTracing(ConnectionLeakDetector.java:113) waiting to lock <0x0000000741eac850> (a java.lang.Object) at com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324) at com.sun.enterprise.resource.pool.ConnectionPool.getResourceFromPool(ConnectionPool.java:758) at com.sun.enterprise.resource.pool.ConnectionPool.getUnenlistedResource(ConnectionPool.java:632) at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:526) at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381) at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245) at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170) at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338) at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165) at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160) at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113) at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:46) at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446) at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167) at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1573) at org.hibernate.loader.Loader.doQuery(Loader.java:696) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259) at org.hibernate.loader.Loader.doList(Loader.java:2228) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125) at org.hibernate.loader.Loader.list(Loader.java:2120) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67) ... company code reference ... at sun.reflect.GeneratedMethodAccessor610.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348) at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206) at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79) at $Proxy531.getFranchise(Unknown Source) at sun.reflect.GeneratedMethodAccessor609.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:241) at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:152) at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:227) ... company code reference ... at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348) at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206) at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79) at $Proxy726.clientHasExpiringRetainer(Unknown Source) at sun.reflect.GeneratedMethodAccessor607.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144) at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174) at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528) at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990) at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540) "connector-timer-proxy": at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:7477) waiting to lock <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet) at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:907) at com.mysql.jdbc.StatementImpl.realClose(StatementImpl.java:2478) locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3091) locked <0x000000076bceb840> (a com.mysql.jdbc.JDBC4PreparedStatement) at com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java:1584) at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364) at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) locked <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection) at com.sun.gjc.spi.ManagedConnection.destroy(ManagedConnection.java:409) at com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator.destroyResource(LocalTxConnectorAllocator.java:136) at com.sun.enterprise.resource.pool.ConnectionPool.deleteResource(ConnectionPool.java:959) at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.removeResource(RWLockDataStructure.java:153) at com.sun.enterprise.resource.pool.ConnectionPool.reclaimConnection(ConnectionPool.java:1643) at com.sun.enterprise.resource.pool.ConnectionLeakDetector.potentialConnectionLeakFound(ConnectionLeakDetector.java:161) locked <0x0000000741eac850> (a java.lang.Object) at com.sun.enterprise.resource.pool.ConnectionLeakDetector.access$000(ConnectionLeakDetector.java:60) at com.sun.enterprise.resource.pool.ConnectionLeakDetector$ConnectionLeakTask.run(ConnectionLeakDetector.java:222) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) "p: thread-pool-1; w: 27": at com.mysql.jdbc.ConnectionImpl.getCharacterSetMetadata(ConnectionImpl.java:2798) waiting to lock <0x0000000741f47060> (a com.mysql.jdbc.JDBC4Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:710) at com.mysql.jdbc.Field.getOriginalName(Field.java:652) at com.mysql.jdbc.ResultSetImpl.buildIndexMapping(ResultSetImpl.java:749) at com.mysql.jdbc.ResultSetImpl.findColumn(ResultSetImpl.java:1108) locked <0x000000076d6dd2a8> (a com.mysql.jdbc.JDBC4ResultSet) at com.mysql.jdbc.ResultSetImpl.getInt(ResultSetImpl.java:2815) at com.sun.gjc.spi.base.ResultSetWrapper.getInt(ResultSetWrapper.java:482) at org.hibernate.type.IntegerType.get(IntegerType.java:51) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:184) at org.hibernate.type.NullableType.nullSafeGet(NullableType.java:173) at org.hibernate.loader.Loader.getKeyFromResultSet(Loader.java:1121) at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:588) at org.hibernate.loader.Loader.doQuery(Loader.java:724) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259) at org.hibernate.loader.Loader.doList(Loader.java:2228) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2125) at org.hibernate.loader.Loader.list(Loader.java:2120) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:401) at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:361) at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196) at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1148) at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102) at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:67) at com.pts.core.reports.ReportToolsSessionBean.getLastDisputeDateByBureau(ReportToolsSessionBean.java:775) at sun.reflect.GeneratedMethodAccessor1003.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052) at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124) at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5388) at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162) at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144) at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861) at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800) at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370) at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5360) at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5348) at com.sun.ejb.containers.EJBObjectInvocationHandler.invoke(EJBObjectInvocationHandler.java:206) at com.sun.ejb.containers.EJBObjectInvocationHandlerDelegate.invoke(EJBObjectInvocationHandlerDelegate.java:79) ... company code reference ... at sun.reflect.GeneratedMethodAccessor1002.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie.dispatchToMethod(ReflectiveTie.java:144) at com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:174) at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528) at com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990) at com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539) at com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540) Found 1 deadlock.

          People

          • Assignee:
            Shalini
            Reporter:
            Jagadish
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: