Issue Details (XML | Word | Printable)

Key: MIMEPULL-5
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Martin Grebac
Reporter: hahmed
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
mimepull

MIMEpull need to request GC under heavy load

Created: 12/Oct/10 12:49 PM   Updated: 29/Jan/13 09:27 AM   Resolved: 30/Nov/12 11:55 AM
Component/s: www
Affects Version/s: current
Fix Version/s: 1.9

Time Tracking:
Not Specified

File Attachments: 1. Java Source File WeakDataFile.java (5 kB) 12/Oct/10 12:51 PM - hahmed

Environment:

Operating System: All
Platform: All


Issuezilla Id: 5
Tags:
Participants: guoyong, hahmed, Martin Grebac and zhous2


 Description  « Hide

Using JAX-WS RI, and under heavy load of SOAP requests with binary attachments
we noticed that the system runs out of file handles. That is due to the large
numbers of tmp files opened by MIMEpull and are substantially delayed in garbage
collection. Also fixing the MAX_ITERATIONS to 2 or 100 might lead to the problem
if for example new 200 file handles are open during the loop to close the 100
in WeakDataFile's drainRefQueueBounded().



hahmed added a comment - 12/Oct/10 12:51 PM

Created an attachment (id=6)
Attaching a patch for this issue


guoyong added a comment - 20/Oct/11 01:29 PM

It was fixed in mimepull-1.8.

3 Changes were made in the patch to improve MIMEPULL implicit clean up mechanism:
1) MemoryData
When the tmp file is created, added tempFile.deleteOnExit() to request the file be deleted when the virtual machine terminates. This will remove the tmp file uopn normal server shutdown even the cleanup mechanism doesn't kick in for it yet.

2) WeakDataFile. drainRefQueueBounded()
Instead of MAX_ITERATIONS of 2, close all the WeakDataFile in the queue to prevent potential disastrous result when under heavy load.

3) CleanUpExecutorFactory
Added org.jvnet.mimepull.CleanUpExecutorFactory, which uses service factory pattern. MIMEPULL when initialized, will search for CleanUpExecutorFactory first in system property and then in META-INF/services. If located, it will uses the executor from the implementation class to monitor and clean up the queue, instead of using WeakDataFile constructor to call drainRefQueueBounded(). This will overcome the drawback that MIMEPULL will always leave uncleaned WeakDataFile in the system since the clean up doesn't start until the next WeakDataFile construction.
MIMEPULL itself contains only the hook, without any CleanUpExecutorFactory implementation. So normal usage will experience the exact same behavior.


zhous2 added a comment - 29/Jan/13 09:27 AM - edited

Hi Guoyong,

For item #3 above, I didn't see entry org.jvnet.mimepull.CleanUpExecutorFactory under folder META-INF\services, in versions 1.8, 1.9 and 1.9.1. Is there anything missing? What should I put in this file if this is not an out-of-box configuration? I suppose I need to implement the below abstract method in CleanUpExecutorFactory and place the inherited class in the file:

  • public abstract Executor getExecutor();

We have been suffering from the temp file issue caused by mimepull.jar for a long time. In the past, we updated MAX_ITERATIONS to 100 to remove temp files. But this doesn't work recently. Not sure whether this is caused by Java update (Full GC seems have little effect on the release of temp file handlers).