Issue Details (XML | Word | Printable)

Key: GLASSFISH-19130
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: marina vatkina
Reporter: hapi
Votes: 1
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
glassfish

Timer Creating "Active" Transactions in Transaction Service Monitor - Memory Leak

Created: 05/Oct/12 06:14 PM   Updated: 30/Oct/12 03:17 AM   Resolved: 23/Oct/12 06:36 PM
Component/s: jts
Affects Version/s: 3.1.2
Fix Version/s: 4.0_b60

Time Tracking:
Not Specified

File Attachments: 1. Java Archive File jta.jar (64 kB) 23/Oct/12 01:30 AM - marina vatkina
2. Java Archive File jts.jar (365 kB) 23/Oct/12 01:30 AM - marina vatkina


Tags: transaction service memory-leak
Participants: hapi, marina vatkina and pdudits


 Description  « Hide

After much isolation trying to pin down a memory leak I've gotten to what may be the root cause. We have the monitoring level set to HIGH for the Transaction Service and are using 3.1.2.2 build 5.

If I create a simple stateless bean that contains a method with a @Schedule annotation:

@Schedule(second = "", minute = "", hour = "", dayOfWeek = "", dayOfMonth = "", month = "", year = "*", info = "MasterScheduler")

This will cause the Transaction Service monitor to note an "active transaction" every time the timer is invoked.

Over the course of a day there's a noticeable memory leak and it will eventually run out of memory.

If we turn the monitoring level for the Transaction Service to OFF it goes away.



marina vatkina added a comment - 09/Oct/12 12:16 AM

Timeout method (the one that is marked with @Schedule in your case) by default runs in a transaction. So the fact that transaction is active for each timeout, is a correct behavior. Do you see the monitoring table grow? Or do you see 1 transaction all the time?


hapi added a comment - 09/Oct/12 01:28 PM

Yes with the Transaction Service monitoring turned on I see an active transaction added for each time the Timeout method is invoked and it never leaves the monitoring table. The table grows until it runs out of memory.

I would expect to see active transactions go to 1 and then go back to 0 after the method has completed. Am I looking at that the wrong way?


marina vatkina added a comment - 09/Oct/12 05:01 PM

No, this shouldn't happen. I'll look into it.


hapi added a comment - 11/Oct/12 09:23 PM

Curious - were you able to experience the same behavior? Should be quite simple to duplicate. Thanks.


marina vatkina added a comment - 11/Oct/12 09:32 PM

Not yet - was busy with other stuff, but will check it as soon as I can


pdudits added a comment - 22/Oct/12 08:59 AM - edited

I am diagnosing similar situation right now, so I can put in some details from the heap dump. Our timer datasource is in a Oracle database, the pool is transactional (XATransaction), monitoring level for transaction service is set to LOW. The datasource is also used by the application itself.

We had the server die with OutOfMemoryError with J2EETransactionManagerSimplified.activeTransactions referring to 162319 transactions and allocating 1GB of heap. The com.sun.jts.CosTransactionTransactionState had following distribution by transaction state:

Transaction state Number of objects
7 STATE_COMMITED 135.327
11 STATE_COMMITTED_ONE_PHASE_OK 26.882
1 STATE_ACTIVE 76
9 STATE_ROLLED_BACK 18
10 STATE_COMMITTING_ONE_PHASE 2

One workaround was to annotate the timeout method with TransactionAttribute.NOT_SUPPORTED, but that resulted in last fire time of the timer not being updated in EJB_TIMER_TBL. This behaviour could not be reproduced with default datasource. The other is to disable monitoring of transaction service.

Another notable discovery from the heap dump is that all instances of com.sun.enterprise.transaction.JavaEETransactionImpl has value of isTimerTask == true.


marina vatkina added a comment - 22/Oct/12 11:01 PM

yes, I do see it growing with some transaction removed and some not. Investigating...


marina vatkina added a comment - 22/Oct/12 11:51 PM

Actually, concurrency might not be an issue, but the way transactions are committed - if they become XA transactions during commit, the monitoring table isn't updated, and the XA transaction can't be removed.


marina vatkina added a comment - 23/Oct/12 01:30 AM

Can you try attached jars (you need both) with your 3.1.2 install? If you are using the trunk build, the fix will be available after the checkin.


marina vatkina added a comment - 23/Oct/12 06:36 PM

Fixed with
Sending transaction/jta/src/main/java/com/sun/enterprise/transaction/JavaEETransactionImpl.java
Sending transaction/jta/src/main/java/com/sun/enterprise/transaction/JavaEETransactionManagerSimplified.java
Sending transaction/jts/src/main/java/com/sun/enterprise/transaction/jts/JavaEETransactionManagerJTSDelegate.java
Transmitting file data ...
Committed revision 56702.

Note that the changes include some cleanup in addition to the actual fix, which was to move monitoring table update to JavaEETransactionImpl (before it clears jtsTx ref)


hapi added a comment - 30/Oct/12 02:05 AM

I can confirm (somewhat belatedly) that the attached jars did seem to solve the problem.

Will this be a part of some future sub-release? 3.1.2.3?


marina vatkina added a comment - 30/Oct/12 03:17 AM

If there is a public release for 3.1.2.x, then yes. Otherwise you can get a patch from Oracle sustaining if you have a support contract.