[GLASSFISH-19190] Concurrent modification of HashMap in connection pool causes race condition Created: 19/Oct/12  Updated: 20/Dec/16  Resolved: 12/Jun/14

Status: Resolved
Project: glassfish
Component/s: jca
Affects Version/s: 3.1.2_dev
Fix Version/s: 4.1_dev

Type: Bug Priority: Critical
Reporter: jaapcoomans Assignee: dhaneesh.t.b
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OS: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
CPU: 4 x Intel(R) Xeon(R) CPU X5670 @ 2.93GHz
RAM: 8174464 kB
JRE: 1.6.0_31-b04


Tags: 4_0_1-approved, connectionpool, connectors

 Description   

My company is running an application on GlassFish that uses a JackRabbit JCR Repository via a JCA connector.

Every once in a while a thread falls in some kind of livelock while obtaining a JCR connection from the connection pool.
It starts out with one thread, but spreads out to more threads within a hour until all CPU's are fully occupied and the system becomes too slow to operate.

Obtained the following stacktrace via jstack:

"p: thread-pool-1; w: 251" daemon prio=10 tid=0x00002aaaba11e000 nid=0x940 runnable [0x0000000043e3a000]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
        at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
        at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
        at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
        at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
        at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:242)
        at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:167)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
        at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
        at org.apache.jackrabbit.jca.JCARepositoryHandle.login(JCARepositoryHandle.java:75)
        at org.apache.jackrabbit.commons.AbstractRepository.login(AbstractRepository.java:123)
-- Snip: the rest of the stacktrace due to company secret information

It seems that the resourceAppAssociationMap in ConnectionPoolEmitterImpl should actually be a ConcurrentHashMap.



 Comments   
Comment by shreedhar_ganapathy [ 13/Dec/12 ]

-> Siva

Comment by Sivakumar Thyagarajan [ 17/Dec/12 ]

Requesting Jagadish to look into this.

Comment by pdudits [ 10/May/13 ]

I've also hit this issue, since it happened during JPA query, it also managed to block entire persistence unit.

I also see no changes to this class in trunk, so likely 4.0 is also affected.

"p: thread-pool-1; w: 10" - Thread t@191
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.get(HashMap.java:320)
	at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
	at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
	at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
	at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
	at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
	at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:332)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
	at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
	at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
	at sun.reflect.GeneratedMethodAccessor147.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.sun.enterprise.resource.naming.DynamicResourceReconfigurator.invoke(DynamicResourceReconfigurator.java:111)
	at $Proxy171.getConnection(Unknown Source)
	at org.eclipse.persistence.sessions.JNDIConnector.connect(JNDIConnector.java:123)
	at org.eclipse.persistence.sessions.DatasourceLogin.connectToDatasource(DatasourceLogin.java:162)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.connectInternal(DatasourceAccessor.java:330)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.connectInternal(DatabaseAccessor.java:293)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.reconnect(DatasourceAccessor.java:565)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.reconnect(DatabaseAccessor.java:1508)
	at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:305)
	- locked <68ccfd1d> (a org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:579)
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1717)
	at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectOneRow(DatasourceCallQueryMechanism.java:666)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRowFromTable(ExpressionQueryMechanism.java:2656)
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectOneRow(ExpressionQueryMechanism.java:2627)
	at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:450)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)
	at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:418)
	at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2831)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)
	at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:323)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2098)
	at org.eclipse.persistence.mappings.OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1695)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:1987)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1353)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:445)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:798)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:603)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:560)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:717)
	at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:459)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)
	at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:418)
	at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2831)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)
	at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:323)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2098)
	at org.eclipse.persistence.mappings.OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1695)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:1987)
	at org.eclipse.persistence.mappings.ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1353)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:445)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:798)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:1684)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:664)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:601)
	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:560)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:717)
	at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:769)
	at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:433)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:392)
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1128)
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2871)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1463)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:485)
	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getResultList(EJBQueryImpl.java:742)
Comment by sbarlabanov [ 08/Jul/13 ]

We face the same issue on Glassfish 3.1.2.2. When jdbc pool monitoring is on, Glassfish begin to eat 100% CPU. Thread dumps show the same stack trace like above:

java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:320)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionAcquired(ConnectionPoolEmitterImpl.java:132)
at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionAcquired(PoolLifeCycleListenerRegistry.java:115)
at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:390)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:332)
at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)
at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)
at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82)
at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.continueInvocation(ConnectionProxyHandler.java:138)
at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at $Proxy945.prepareStatement(Unknown Source)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:147)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:166)
at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:145)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1720)
at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1697)
at org.hibernate.loader.Loader.doQuery(Loader.java:832)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:293)
at org.hibernate.loader.Loader.doList(Loader.java:2382)
at org.hibernate.loader.Loader.doList(Loader.java:2368)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2198)
at org.hibernate.loader.Loader.list(Loader.java:2193)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:470)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1244)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:257)

Comment by sbarlabanov [ 08/Jul/13 ]

And the hash map causing the problem seems to be introduced in the fix to GLASSFISH-14101.

Comment by dhaneesh.t.b [ 12/Jun/14 ]

------------------------------------------------------------------------
r63349 | dhaneesh.t.b | 2014-06-11 17:07:21 +0530 (Wed, 11 Jun 2014) | 1 line
Changed paths:
M /trunk/main/appserver/connectors/connectors-runtime/src/main/java/com/sun/enterprise/resource/pool/monitor/ConnectionPoolEmitterImpl.java

Fix for JIRA : https://java.net/jira/browse/GLASSFISH-19190 ,Executed RQ Targets: quicklook_all,apsdevtests_jdbc,apsdevtests_jms,apsdevtests_connector,cts_smoke,findbugs,copyright, Executed Local Hudson Test JOBS : resource-admin-cli,JDBC4.1,CTS 1.7 (classic),CTS 1.7 (web)
------------------------------------------------------------------------

Comment by sah_java [ 02/Jul/14 ]

Excellent. Any plans to backport to 3.1.2.2 per GLASSFISH-21051, looks like it's the same problem.

Generated at Wed Apr 26 19:54:48 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.