glassfish
  1. glassfish
  2. GLASSFISH-4689

deadlock around ConcurrencyManager/IdentityMap

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 9.0peur1
    • Fix Version/s: 9.1.1
    • Component/s: entity-persistence
    • Labels:
      None
    • Environment:

      Operating System: Solaris
      Platform: All

    • Issuezilla Id:
      4,689
    • Status Whiteboard:
      Hide

      911Approved

      Show
      911Approved

      Description

      Experiencing a deadlock in production.

      Relevant parts of stack dump will are:

      Blocked threads:

      "httpSSLWorkerThread-8080-4" daemon prio=3 tid=0x013b0800 nid=0x4d9 waiting for
      monitor entry [0xabcfd000..0xabcffa70]
      java.lang.Thread.State: BLOCKED (on object monitor)
      at
      oracle.toplink.essentials.internal.identitymaps.IdentityMap.getCacheKey(IdentityMap.java:291)

      • waiting to lock <0xbe3f37a0> (a
        oracle.toplink.essentials.internal.identitymaps.SoftCacheWeakIdentityMap)
        at
        oracle.toplink.essentials.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:531)
        at
        oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:328)
        at
        oracle.toplink.essentials.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:705)
        at
        oracle.toplink.essentials.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:224)
        at
        oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:436)
        at
        oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:575)
        at
        oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:692)
        at
        oracle.toplink.essentials.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:1834)
        at
        oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
        at
        oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:101)
        at
        oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:91)
        at
        oracle.toplink.essentials.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:105)
      • locked <0xe683a4e8> (a
        oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder)
        at
        oracle.toplink.essentials.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:175)
        at
        oracle.toplink.essentials.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:247)
        at
        oracle.toplink.essentials.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:105)
      • locked <0xe683a8d0> (a
        oracle.toplink.essentials.internal.indirection.UnitOfWorkQueryValueHolder)
        at
        org.webonweb.runtime.impl.repository.db.jpa.DbWidgetResource._toplink_getcontent(DbWidgetResource.java)

      Blocking thread:

      "httpSSLWorkerThread-8080-1" daemon prio=3 tid=0x01263400 nid=0x62 in
      Object.wait() [0xa95fd000..0xa95ffbf0]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0xe76cf180> (a
        oracle.toplink.essentials.internal.helper.ConcurrencyManager)
        at java.lang.Object.wait(Object.java:485)
        at
        oracle.toplink.essentials.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:217)
      • locked <0xe76cf180> (a
        oracle.toplink.essentials.internal.helper.ConcurrencyManager)
        at
        oracle.toplink.essentials.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:205)
        at
        oracle.toplink.essentials.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:158)
        at
        oracle.toplink.essentials.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:66)
        at
        oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.invalidateClass(IdentityMapAccessor.java:630)
      • locked <0xbe3f37a0> (a
        oracle.toplink.essentials.internal.identitymaps.SoftCacheWeakIdentityMap)
        at
        oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.invalidateClass(IdentityMapAccessor.java:614)
        at
        oracle.toplink.essentials.queryframework.ModifyAllQuery.invalidateCache(ModifyAllQuery.java:256)
        at
        oracle.toplink.essentials.queryframework.ModifyAllQuery.mergeChangesIntoSharedCache(ModifyAllQuery.java:271)
        at
        oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.mergeChangesIntoParent(UnitOfWorkImpl.java:2542)
        at
        oracle.toplink.essentials.internal.ejb.cmp3.base.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:118)
        at
        oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:856)
        at
        oracle.toplink.essentials.internal.ejb.cmp3.transaction.base.EntityTransactionImpl.commit(EntityTransactionImpl.java:102)
        at
        oracle.toplink.essentials.internal.ejb.cmp3.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:60)
      1. GF-bug-4689.patch
        2 kB
        gyorke
      2. stack.txt
        68 kB
        jbelis

        Activity

        jbelis created issue -
        Hide
        gyorke added a comment -

        Created an attachment (id=1432)
        Proposed Patch

        Show
        gyorke added a comment - Created an attachment (id=1432) Proposed Patch
        Hide
        gyorke added a comment -

        I did add a proposed patch but this was added prematurely. There is not enough
        threads here to track down the deadlock.

        Can you attach the entire thread dump?

        Show
        gyorke added a comment - I did add a proposed patch but this was added prematurely. There is not enough threads here to track down the deadlock. Can you attach the entire thread dump?
        Hide
        jbelis added a comment -

        Created an attachment (id=1433)
        entire stack trace

        Show
        jbelis added a comment - Created an attachment (id=1433) entire stack trace
        Hide
        jbelis added a comment -

        adding cc

        Show
        jbelis added a comment - adding cc
        Hide
        jbelis added a comment -

        adding cc

        Show
        jbelis added a comment - adding cc
        Hide
        jbelis added a comment -

        more descriptive title

        Show
        jbelis added a comment - more descriptive title
        Hide
        gyorke added a comment -

        The two threads in conflict in stack.txt are httpSSLWorkerThread-8080-1 and
        httpSSLWorkerThread-8080-2 . The conflict occurs when a thread commits a
        transaction containing a bulk update that TopLink can not resolve in memory so
        TopLink invalidates all cached instances of the updated class. In this case
        another thread is currently committing changes to similar objects and has
        locked all of the objects. The invalidating thread has the "cache" locked and
        the other thread has the cached objects locked.

        Patch:
        The attached patch should resolve the deadlock.

        Show
        gyorke added a comment - The two threads in conflict in stack.txt are httpSSLWorkerThread-8080-1 and httpSSLWorkerThread-8080-2 . The conflict occurs when a thread commits a transaction containing a bulk update that TopLink can not resolve in memory so TopLink invalidates all cached instances of the updated class. In this case another thread is currently committing changes to similar objects and has locked all of the objects. The invalidating thread has the "cache" locked and the other thread has the cached objects locked. Patch: The attached patch should resolve the deadlock.
        Hide
        gyorke added a comment -

        Workaround:
        Potential workaround is to change the cache isolation level through a
        session customizer to CONCURRENT_READ_WRITE.

        Persistence.xml entry:

        <property name="toplink.session.customizer"
        value="mypackage.IsolationCustomizer"/>

        Customizer Class :

        package mypackage;
        import oracle.toplink.essentials.sessions.Session;

        /**

        • PUBLIC:
        • This interface is to allow extra customization on a TopLink Session
          */
          public class IsolationCustomizer extends SessionCustomizer {
          public void customize(Session session) throws Exception { session.getLogin().setCacheTransactionIsolation (DatasourceLogin.CONCURRENT_READ_WRITE); }

          }

        Show
        gyorke added a comment - Workaround: Potential workaround is to change the cache isolation level through a session customizer to CONCURRENT_READ_WRITE. Persistence.xml entry: <property name="toplink.session.customizer" value="mypackage.IsolationCustomizer"/> Customizer Class : package mypackage; import oracle.toplink.essentials.sessions.Session; /** PUBLIC: This interface is to allow extra customization on a TopLink Session */ public class IsolationCustomizer extends SessionCustomizer { public void customize(Session session) throws Exception { session.getLogin().setCacheTransactionIsolation (DatasourceLogin.CONCURRENT_READ_WRITE); } }
        Hide
        harpreet added a comment -

        approved for v2.1

        Show
        harpreet added a comment - approved for v2.1
        Hide
        harpreet added a comment -

        adding approved tag in status whiteboard, missed it last time.

        Show
        harpreet added a comment - adding approved tag in status whiteboard, missed it last time.
        Hide
        gyorke added a comment -

        reassigning

        Show
        gyorke added a comment - reassigning
        Hide
        gyorke added a comment -

        patch submitted to 2.1 stream

        Show
        gyorke added a comment - patch submitted to 2.1 stream
        kenaiadmin made changes -
        Field Original Value New Value
        issue.field.bugzillaimportkey 4689 36293

          People

          • Assignee:
            gyorke
            Reporter:
            jbelis
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: