Issue Details (XML | Word | Printable)

Key: GLASSFISH-4689
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Blocker Blocker
Assignee: gyorke
Reporter: jbelis
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
glassfish

deadlock around ConcurrencyManager/IdentityMap

Created: 08/Apr/08 06:01 PM   Updated: 09/Apr/08 12:51 PM   Resolved: 09/Apr/08 12:51 PM
Component/s: entity-persistence
Affects Version/s: 9.0peur1
Fix Version/s: 9.1.1

Time Tracking:
Not Specified

File Attachments: 1. Text File GF-bug-4689.patch (2 kB) 09/Apr/08 08:09 AM - gyorke
2. Text File stack.txt (68 kB) 09/Apr/08 09:11 AM - jbelis

Environment:

Operating System: Solaris
Platform: All


Issuezilla Id: 4,689
Status Whiteboard:

911Approved

Tags:
Participants: gyorke, harpreet and jbelis


 Description  « Hide

Experiencing a deadlock in production.

Relevant parts of stack dump will are:

Blocked threads:

"httpSSLWorkerThread-8080-4" daemon prio=3 tid=0x013b0800 nid=0x4d9 waiting for
monitor entry [0xabcfd000..0xabcffa70]
java.lang.Thread.State: BLOCKED (on object monitor)
at
oracle.toplink.essentials.internal.identitymaps.IdentityMap.getCacheKey(IdentityMap.java:291)

  • waiting to lock <0xbe3f37a0> (a
    oracle.toplink.essentials.internal.identitymaps.SoftCacheWeakIdentityMap)
    at
    oracle.toplink.essentials.internal.identitymaps.IdentityMapManager.getFromIdentityMap(IdentityMapManager.java:531)
    at
    oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:328)
    at
    oracle.toplink.essentials.internal.queryframework.ExpressionQueryMechanism.checkCacheForObject(ExpressionQueryMechanism.java:705)
    at
    oracle.toplink.essentials.queryframework.ReadObjectQuery.checkEarlyReturnImpl(ReadObjectQuery.java:224)
    at
    oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.checkEarlyReturn(ObjectLevelReadQuery.java:436)
    at
    oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:575)
    at
    oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:692)
    at
    oracle.toplink.essentials.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:1834)
    at
    oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
    at
    oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:101)
    at
    oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:91)
    at
    oracle.toplink.essentials.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:105)
  • locked <0xe683a4e8> (a
    oracle.toplink.essentials.internal.indirection.QueryBasedValueHolder)
    at
    oracle.toplink.essentials.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:175)
    at
    oracle.toplink.essentials.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:247)
    at
    oracle.toplink.essentials.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:105)
  • locked <0xe683a8d0> (a
    oracle.toplink.essentials.internal.indirection.UnitOfWorkQueryValueHolder)
    at
    org.webonweb.runtime.impl.repository.db.jpa.DbWidgetResource._toplink_getcontent(DbWidgetResource.java)

Blocking thread:

"httpSSLWorkerThread-8080-1" daemon prio=3 tid=0x01263400 nid=0x62 in
Object.wait() [0xa95fd000..0xa95ffbf0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

  • waiting on <0xe76cf180> (a
    oracle.toplink.essentials.internal.helper.ConcurrencyManager)
    at java.lang.Object.wait(Object.java:485)
    at
    oracle.toplink.essentials.internal.helper.ConcurrencyManager.acquireReadLock(ConcurrencyManager.java:217)
  • locked <0xe76cf180> (a
    oracle.toplink.essentials.internal.helper.ConcurrencyManager)
    at
    oracle.toplink.essentials.internal.helper.ConcurrencyManager.checkReadLock(ConcurrencyManager.java:205)
    at
    oracle.toplink.essentials.internal.identitymaps.CacheKey.checkReadLock(CacheKey.java:158)
    at
    oracle.toplink.essentials.internal.identitymaps.IdentityMapKeyEnumeration.nextElement(IdentityMapKeyEnumeration.java:66)
    at
    oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.invalidateClass(IdentityMapAccessor.java:630)
  • locked <0xbe3f37a0> (a
    oracle.toplink.essentials.internal.identitymaps.SoftCacheWeakIdentityMap)
    at
    oracle.toplink.essentials.internal.sessions.IdentityMapAccessor.invalidateClass(IdentityMapAccessor.java:614)
    at
    oracle.toplink.essentials.queryframework.ModifyAllQuery.invalidateCache(ModifyAllQuery.java:256)
    at
    oracle.toplink.essentials.queryframework.ModifyAllQuery.mergeChangesIntoSharedCache(ModifyAllQuery.java:271)
    at
    oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.mergeChangesIntoParent(UnitOfWorkImpl.java:2542)
    at
    oracle.toplink.essentials.internal.ejb.cmp3.base.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:118)
    at
    oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:856)
    at
    oracle.toplink.essentials.internal.ejb.cmp3.transaction.base.EntityTransactionImpl.commit(EntityTransactionImpl.java:102)
    at
    oracle.toplink.essentials.internal.ejb.cmp3.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:60)


gyorke added a comment - 09/Apr/08 08:09 AM

Created an attachment (id=1432)
Proposed Patch


gyorke added a comment - 09/Apr/08 08:19 AM

I did add a proposed patch but this was added prematurely. There is not enough
threads here to track down the deadlock.

Can you attach the entire thread dump?


jbelis added a comment - 09/Apr/08 09:11 AM

Created an attachment (id=1433)
entire stack trace


jbelis added a comment - 09/Apr/08 09:21 AM

adding cc


jbelis added a comment - 09/Apr/08 09:22 AM

adding cc


jbelis added a comment - 09/Apr/08 09:27 AM

more descriptive title


gyorke added a comment - 09/Apr/08 10:20 AM

The two threads in conflict in stack.txt are httpSSLWorkerThread-8080-1 and
httpSSLWorkerThread-8080-2 . The conflict occurs when a thread commits a
transaction containing a bulk update that TopLink can not resolve in memory so
TopLink invalidates all cached instances of the updated class. In this case
another thread is currently committing changes to similar objects and has
locked all of the objects. The invalidating thread has the "cache" locked and
the other thread has the cached objects locked.

Patch:
The attached patch should resolve the deadlock.


gyorke added a comment - 09/Apr/08 10:37 AM

Workaround:
Potential workaround is to change the cache isolation level through a
session customizer to CONCURRENT_READ_WRITE.

Persistence.xml entry:

<property name="toplink.session.customizer"
value="mypackage.IsolationCustomizer"/>

Customizer Class :

package mypackage;
import oracle.toplink.essentials.sessions.Session;

/**

  • PUBLIC:
  • This interface is to allow extra customization on a TopLink Session
    */
    public class IsolationCustomizer extends SessionCustomizer {
    public void customize(Session session) throws Exception { session.getLogin().setCacheTransactionIsolation (DatasourceLogin.CONCURRENT_READ_WRITE); }
    }

harpreet added a comment - 09/Apr/08 12:11 PM

approved for v2.1


harpreet added a comment - 09/Apr/08 12:22 PM

adding approved tag in status whiteboard, missed it last time.


gyorke added a comment - 09/Apr/08 12:49 PM

reassigning


gyorke added a comment - 09/Apr/08 12:51 PM

patch submitted to 2.1 stream