Issue Details (XML | Word | Printable)

Key: GLASSFISH-3747
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: tware
Reporter: batzee
Votes: 1
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
glassfish

Server Deadlock

Created: 07/Oct/07 07:58 AM   Updated: 01/Dec/10 09:08 AM   Resolved: 21/Dec/07 10:37 AM
Component/s: entity-persistence
Affects Version/s: 9.1pe
Fix Version/s: 9.1.1_dev

Time Tracking:
Not Specified

File Attachments: 1. Text File stacktrace-20071004-2.txt (935 kB) 07/Oct/07 08:00 AM - batzee
2. Text File stacktrace-20071004.zip (24 kB) 07/Oct/07 08:03 AM - batzee
3. Text File stacktrace-20071006.zip (62 kB) 07/Oct/07 08:04 AM - batzee
4. Text File stacktrace-20071007-1410.txt (299 kB) 07/Oct/07 08:01 AM - batzee
5. Java Archive File toplink-essentials-3747patch-updated.jar (21 kB) 22/Oct/07 12:14 PM - tware
6. Java Archive File toplink-essentials-3747patch.jar (21 kB) 19/Oct/07 10:48 AM - tware

Environment:

Operating System: All
Platform: All


Issuezilla Id: 3,747
Status Whiteboard:

as91ur1-na,911Approved

Tags:
Participants: batzee, gfbugbridge, harpreet and tware


 Description  « Hide

I have experienced a server deadlock several times on our production system. The
stacktraces which are all attached to this issue have one thing in common: There
is always one (and only one) thread stuck in the AfterCompletion listener,
hanging in

oracle.toplink.essentials.internal.identitymaps.CacheKey.checkReadLock

Then there are a varying number of other threads trying to execute a query,
these are hanging in

oracle.toplink.essentials.internal.helper.ConcurrencyManager.releaseDeferredLock

And also a varying number of threads executing a query and hanging in

oracle.toplink.essentials.internal.helper.ConcurrencyManager.acquireReadLock

Using glassfish2 build 58g.
ASQuickStartup is off.

If I can provide any more info please let me know.



batzee added a comment - 07/Oct/07 08:00 AM

Created an attachment (id=1187)
server stacktrace made with "asadmin generate-jvm-report"


batzee added a comment - 07/Oct/07 08:01 AM

Created an attachment (id=1188)
server stacktrace made with "asadmin generate-jvm-report"


batzee added a comment - 07/Oct/07 08:03 AM

Created an attachment (id=1189)
server stacktrace made with "asadmin generate-jvm-report"


batzee added a comment - 07/Oct/07 08:04 AM

Created an attachment (id=1190)
server stacktrace made with "asadmin generate-jvm-report"


gfbugbridge added a comment - 07/Oct/07 05:01 PM

<BT6613947>


batzee added a comment - 18/Oct/07 08:55 AM

Can anyone have a look at this please?


tware added a comment - 18/Oct/07 11:07 AM

Looking at this issue.

batzee, I may need to get you to test my fix against your application when it is
done.


tware added a comment - 19/Oct/07 10:45 AM

I am attaching a jar file that contains a set of patched files for the TopLink
Essentials jar that solve a problem with almost identical stack dumps. Since I
do not have a recreation of this particular issue, I will need help to
determine whether this fix solves this bug.

batzee, is it possible to give these changes a try by either replacing the
classes in your toplink-essentials jar with these classes or by putting this
jar ahead of toplink-essentials.jar on your classpath? If so, please let me
know if your issue is solved.


tware added a comment - 19/Oct/07 10:48 AM

Created an attachment (id=1215)
potential patch for the bug


tware added a comment - 22/Oct/07 12:14 PM

Created an attachment (id=1216)
updated patch for bug


tware added a comment - 22/Oct/07 12:15 PM

Added a new version of the patch that should be more portable between VMs.


tware added a comment - 05/Nov/07 12:31 PM

Fix will not make ur1 cutoff. Changing target milestone.

According to the following forum post, the attached patch provides a partial
fix. We are investigating the full fix.

http://forums.java.net/jive/thread.jspa?messageID=241912


harpreet added a comment - 19/Dec/07 04:21 PM

Approving changes for 9.1.1. Solves customer production issues. Fix was supposed to make into 9.1UR1
but was not able to get in.


tware added a comment - 21/Dec/07 10:37 AM

Issue number: 3747
Obtained from: TopLink
Submitted by: Tom Ware
Reviewed by: Obtained from Oracle TopLink - reviewed when included there

Closes a window where locks on IndirectCollections can cause deadlocks.

Tested with Oracle-internal tests and entity-persistence-tests

This fix has solved the issue for several customers. There is still an issue
for one customer in the forum post above, it it is a separate issue.