glassfish
  1. glassfish
  2. GLASSFISH-21051

ConnectionPoolEmitterImpl is not thread safe. Misuse of HashMap (resourceAppAssociationMap)

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2.2
    • Fix Version/s: None
    • Component/s: jdbc
    • Labels:
      None
    • Environment:

      REDHAT Linux 6

      Description

      We configured max thread pool size to be 150 in glassfish 3.1.2.2. We noticed most thread were blocked after a while, and there was an exception like this: 'java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096'. We did thread dump analysis and heapdump analysis. There were a couple of threads in runnable state with stacktrace like this:

      (first thread)
      "connector-timer-proxy" daemon prio=5 tid=69 RUNNABLE
      at java.util.HashMap.removeEntryForKey(HashMap.java:569)
      Local Variable: java.util.HashMap$Entry#3068822
      Local Variable: java.util.HashMap$Entry#2436928
      at java.util.HashMap.remove(HashMap.java:538)
      Local Variable: java.util.HashMap#166939
      Local Variable: java.lang.Long#267315
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionDestroyed(ConnectionPoolEmitterImpl.java:184)
      Local Variable: com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl#1
      at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionDestroyed(PoolLifeCycleListenerRegistry.java:151)
      Local Variable: java.util.AbstractList$Itr#106505
      at com.sun.enterprise.resource.pool.ConnectionPool.deleteResource(ConnectionPool.java:974)
      at com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure.removeResource(RWLockDataStructure.java:153)
      Local Variable: com.sun.enterprise.resource.pool.datastructure.RWLockDataStructure#1
      at com.sun.enterprise.resource.pool.ConnectionPool.reclaimConnection(ConnectionPool.java:1643)
      Local Variable: java.lang.String#1077104
      at com.sun.enterprise.resource.pool.ConnectionLeakDetector.potentialConnectionLeakFound(ConnectionLeakDetector.java:161)
      Local Variable: java.lang.StackTraceElement[]#2777
      Local Variable: com.sun.enterprise.resource.pool.ConnectionPool#1
      at com.sun.enterprise.resource.pool.ConnectionLeakDetector.access$000(ConnectionLeakDetector.java:60)
      Local Variable: com.sun.enterprise.resource.ResourceHandle#88
      Local Variable: com.sun.enterprise.resource.pool.ConnectionLeakDetector#1
      at com.sun.enterprise.resource.pool.ConnectionLeakDetector$ConnectionLeakTask.run(ConnectionLeakDetector.java:222)
      Local Variable: com.sun.enterprise.resource.pool.ConnectionLeakDetector$ConnectionLeakTask#1
      at java.util.TimerThread.mainLoop(Timer.java:512)
      at java.util.TimerThread.run(Timer.java:462)

      (second thread)
      "http-thread-pool-8009(56)" daemon prio=5 tid=296 RUNNABLE
      at java.util.HashMap.transfer(HashMap.java:484)
      Local Variable: java.util.HashMap$Entry#3402865
      at java.util.HashMap.resize(HashMap.java:463)
      Local Variable: java.util.HashMap$Entry[]#800390
      at java.util.HashMap.addEntry(HashMap.java:755)
      at java.util.HashMap.put(HashMap.java:385)
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:320)
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
      at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
      Local Variable: java.util.AbstractList$Itr#106556
      at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
      Local Variable: com.sun.enterprise.resource.ResourceHandle#377
      at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
      Local Variable: com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator#166
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
      Local Variable: com.sun.enterprise.resource.ResourceSpec#504
      at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
      at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)

      (third thread)
      "http-thread-pool-8009(138)" daemon prio=5 tid=378 RUNNABLE
      at java.util.HashMap.get(HashMap.java:303)
      Local Variable: java.lang.Long#267316
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:313)
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
      at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
      Local Variable: java.util.AbstractList$Itr#106506
      at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
      at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
      Local Variable: com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator#79
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
      Local Variable: com.sun.enterprise.resource.ResourceSpec#96
      at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
      at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)

      (fourth thread)
      "http-thread-pool-8009(11)" daemon prio=5 tid=251 RUNNABLE
      at java.util.HashMap.transfer(HashMap.java:484)
      Local Variable: java.util.HashMap$Entry[]#943178
      Local Variable: java.util.HashMap$Entry#3075959
      at java.util.HashMap.resize(HashMap.java:463)
      Local Variable: java.util.HashMap$Entry[]#1092899
      at java.util.HashMap.addEntry(HashMap.java:755)
      at java.util.HashMap.put(HashMap.java:385)
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.getAppName(ConnectionPoolEmitterImpl.java:320)
      Local Variable: java.lang.String#1256767
      at com.sun.enterprise.resource.pool.monitor.ConnectionPoolEmitterImpl.connectionUsed(ConnectionPoolEmitterImpl.java:234)
      at com.sun.enterprise.resource.pool.PoolLifeCycleListenerRegistry.connectionUsed(PoolLifeCycleListenerRegistry.java:145)
      Local Variable: java.util.AbstractList$Itr#106557
      at com.sun.enterprise.resource.pool.ConnectionPool.internalGetResource(ConnectionPool.java:532)
      Local Variable: com.sun.enterprise.resource.ResourceHandle#351
      at com.sun.enterprise.resource.pool.ConnectionPool.getResource(ConnectionPool.java:381)
      Local Variable: com.sun.enterprise.resource.allocator.LocalTxConnectorAllocator#165
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResourceFromPool(PoolManagerImpl.java:245)
      at com.sun.enterprise.resource.pool.PoolManagerImpl.getResource(PoolManagerImpl.java:170)
      Local Variable: com.sun.enterprise.resource.ResourceSpec#503
      at com.sun.enterprise.connectors.ConnectionManagerImpl.getResource(ConnectionManagerImpl.java:338)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.internalGetConnection(ConnectionManagerImpl.java:301)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:190)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:165)
      at com.sun.enterprise.connectors.ConnectionManagerImpl.allocateConnection(ConnectionManagerImpl.java:160)
      at com.sun.gjc.spi.base.DataSource.getConnection(DataSource.java:113)

      The CPU usage was way higher than normal. And all blocked threads were blocked on the same java lock and the same line of code. The following is an example.

      "http-thread-pool-8009(96)" daemon prio=5 tid=336 BLOCKED
      at com.sun.enterprise.resource.pool.ConnectionLeakDetector.startConnectionLeakTracing(ConnectionLeakDetector.java:113)
      at com.sun.enterprise.resource.pool.ConnectionPool.setResourceStateToBusy(ConnectionPool.java:324)

      Basically all 3 threads were running in infinite loop. The cause was the misuse of a HashMap object (resourceAppAssociationMap) in a concurrent environment in ConnectionPoolEmitterImpl. This is a related article.
      http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html

      The fix actually should be quite easy. Just use ConcurrentHashMap instead of HashMap for resourceAppAssociationMap.

      Can we get a patch soon on this?

        Activity

        Hide
        reza_rahman added a comment -

        Note that this is now under review for 4.0.1

        Show
        reza_rahman added a comment - Note that this is now under review for 4.0.1
        Hide
        syangjava added a comment -

        So there is no plan to have a release after 3.1.2.2?

        Is there a way we can get a patch?

        Show
        syangjava added a comment - So there is no plan to have a release after 3.1.2.2? Is there a way we can get a patch?
        Hide
        reza_rahman added a comment - - edited

        Not sure if it will be back-ported. The only way to get an individual patch is through GlassFish 3 commercial support if you have purchased it.

        Show
        reza_rahman added a comment - - edited Not sure if it will be back-ported. The only way to get an individual patch is through GlassFish 3 commercial support if you have purchased it.
        Hide
        Jagadish added a comment -
        Show
        Jagadish added a comment - Same as issue : https://java.net/jira/browse/GLASSFISH-19190
        Hide
        syangjava added a comment -

        So this problem was known in the '3.1.2_b14' release and never fixed? Is there any plan to make another glassfish 3 release soon, for example, 3.1.2.3 with this fix?

        Show
        syangjava added a comment - So this problem was known in the '3.1.2_b14' release and never fixed? Is there any plan to make another glassfish 3 release soon, for example, 3.1.2.3 with this fix?
        Hide
        reza_rahman added a comment -

        As you may be aware, we are currently very focused on GlassFish 4.0.1. Although a further GlassFish 3 release may happen, it's not currently in planning that I know of. If this is a critical production issue that needs to be addressed quickly your best bet is really talking to your Oracle sales rep or perhaps patching it yourself?

        Show
        reza_rahman added a comment - As you may be aware, we are currently very focused on GlassFish 4.0.1. Although a further GlassFish 3 release may happen, it's not currently in planning that I know of. If this is a critical production issue that needs to be addressed quickly your best bet is really talking to your Oracle sales rep or perhaps patching it yourself?
        Hide
        syangjava added a comment -

        We think it is possible we can patch this ourselves. Is there any legal restrictions on this?

        Show
        syangjava added a comment - We think it is possible we can patch this ourselves. Is there any legal restrictions on this?
        Hide
        reza_rahman added a comment - - edited

        I would just review the CDDL/licensing terms that should be very lightweight and well understood in your case? BTW, the safest bet is probably just contributing the patch back.

        Show
        reza_rahman added a comment - - edited I would just review the CDDL/licensing terms that should be very lightweight and well understood in your case? BTW, the safest bet is probably just contributing the patch back.

          People

          • Assignee:
            Jagadish
            Reporter:
            syangjava
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 3 hours
              3h
              Remaining:
              Remaining Estimate - 3 hours
              3h
              Logged:
              Time Spent - Not Specified
              Not Specified