mimepull
  1. mimepull
  2. MIMEPULL-5

MIMEpull need to request GC under heavy load

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: current
    • Fix Version/s: 1.9
    • Component/s: www
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • Issuezilla Id:
      5

      Description

      Using JAX-WS RI, and under heavy load of SOAP requests with binary attachments
      we noticed that the system runs out of file handles. That is due to the large
      numbers of tmp files opened by MIMEpull and are substantially delayed in garbage
      collection. Also fixing the MAX_ITERATIONS to 2 or 100 might lead to the problem
      if for example new 200 file handles are open during the loop to close the 100
      in WeakDataFile's drainRefQueueBounded().

        Activity

        Hide
        hahmed added a comment -

        Created an attachment (id=6)
        Attaching a patch for this issue

        Show
        hahmed added a comment - Created an attachment (id=6) Attaching a patch for this issue
        Hide
        guoyong added a comment -

        It was fixed in mimepull-1.8.

        3 Changes were made in the patch to improve MIMEPULL implicit clean up mechanism:
        1) MemoryData
        When the tmp file is created, added tempFile.deleteOnExit() to request the file be deleted when the virtual machine terminates. This will remove the tmp file uopn normal server shutdown even the cleanup mechanism doesn't kick in for it yet.

        2) WeakDataFile. drainRefQueueBounded()
        Instead of MAX_ITERATIONS of 2, close all the WeakDataFile in the queue to prevent potential disastrous result when under heavy load.

        3) CleanUpExecutorFactory
        Added org.jvnet.mimepull.CleanUpExecutorFactory, which uses service factory pattern. MIMEPULL when initialized, will search for CleanUpExecutorFactory first in system property and then in META-INF/services. If located, it will uses the executor from the implementation class to monitor and clean up the queue, instead of using WeakDataFile constructor to call drainRefQueueBounded(). This will overcome the drawback that MIMEPULL will always leave uncleaned WeakDataFile in the system since the clean up doesn't start until the next WeakDataFile construction.
        MIMEPULL itself contains only the hook, without any CleanUpExecutorFactory implementation. So normal usage will experience the exact same behavior.

        Show
        guoyong added a comment - It was fixed in mimepull-1.8. 3 Changes were made in the patch to improve MIMEPULL implicit clean up mechanism: 1) MemoryData When the tmp file is created, added tempFile.deleteOnExit() to request the file be deleted when the virtual machine terminates. This will remove the tmp file uopn normal server shutdown even the cleanup mechanism doesn't kick in for it yet. 2) WeakDataFile. drainRefQueueBounded() Instead of MAX_ITERATIONS of 2, close all the WeakDataFile in the queue to prevent potential disastrous result when under heavy load. 3) CleanUpExecutorFactory Added org.jvnet.mimepull.CleanUpExecutorFactory, which uses service factory pattern. MIMEPULL when initialized, will search for CleanUpExecutorFactory first in system property and then in META-INF/services. If located, it will uses the executor from the implementation class to monitor and clean up the queue, instead of using WeakDataFile constructor to call drainRefQueueBounded(). This will overcome the drawback that MIMEPULL will always leave uncleaned WeakDataFile in the system since the clean up doesn't start until the next WeakDataFile construction. MIMEPULL itself contains only the hook, without any CleanUpExecutorFactory implementation. So normal usage will experience the exact same behavior.
        Hide
        zhous2 added a comment - - edited

        Hi Guoyong,

        For item #3 above, I didn't see entry org.jvnet.mimepull.CleanUpExecutorFactory under folder META-INF\services, in versions 1.8, 1.9 and 1.9.1. Is there anything missing? What should I put in this file if this is not an out-of-box configuration? I suppose I need to implement the below abstract method in CleanUpExecutorFactory and place the inherited class in the file:

        • public abstract Executor getExecutor();

        We have been suffering from the temp file issue caused by mimepull.jar for a long time. In the past, we updated MAX_ITERATIONS to 100 to remove temp files. But this doesn't work recently. Not sure whether this is caused by Java update (Full GC seems have little effect on the release of temp file handlers).

        Show
        zhous2 added a comment - - edited Hi Guoyong, For item #3 above, I didn't see entry org.jvnet.mimepull.CleanUpExecutorFactory under folder META-INF\services, in versions 1.8, 1.9 and 1.9.1. Is there anything missing? What should I put in this file if this is not an out-of-box configuration? I suppose I need to implement the below abstract method in CleanUpExecutorFactory and place the inherited class in the file: public abstract Executor getExecutor(); We have been suffering from the temp file issue caused by mimepull.jar for a long time. In the past, we updated MAX_ITERATIONS to 100 to remove temp files. But this doesn't work recently. Not sure whether this is caused by Java update (Full GC seems have little effect on the release of temp file handlers).

          People

          • Assignee:
            Martin Grebac
            Reporter:
            hahmed
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: