glassfish
  1. glassfish
  2. GLASSFISH-11088

[embedded] Stopping GFv3 leaves several daemon threads running

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: V3
    • Fix Version/s: not determined
    • Component/s: embedded
    • Labels:
      None
    • Environment:

      Operating System: Windows XP
      Platform: PC

      Description

      After stopping the server there is a bunch of daemon threads:
      transaction-manager, AutoDeployer, DynamicReloader, pool-X-thread-Y.

      Most of these threads (all except of some instances of pool-X-thread-Y) are
      never finished until VM exit.
      In the use case, when embedded GF is restarted many times during main
      application session this can cause "Out-Of-Memory" problem and other side effects.

      1. GlassFish11088.java
        3 kB
        Bhavanishankar
      2. jstack
        7 kB
        Bhavanishankar

        Issue Links

          Activity

          bavik78 created issue -
          Hide
          bavik78 added a comment -

          2 CC added

          Show
          bavik78 added a comment - 2 CC added
          Hide
          dochez added a comment -

          not a showstopper, few deamon threads hanging around after server.close()

          Show
          dochez added a comment - not a showstopper, few deamon threads hanging around after server.close()
          Hide
          kumara added a comment -

          Setting target release for unresolved issues submitted on v3 release to the next release. Not changing
          issues submitted on v2.x release because they might not apply to v3.next release.

          Show
          kumara added a comment - Setting target release for unresolved issues submitted on v3 release to the next release. Not changing issues submitted on v2.x release because they might not apply to v3.next release.
          Hide
          sirajg added a comment -

          transfer

          Show
          sirajg added a comment - transfer
          Hide
          sirajg added a comment -

          After a start, deploy, stop, I see these threads (Thread.getAllStackTraces) :
          Keep-Alive-Timer
          Reference Handler
          pool-8-thread-1
          pool-1-thread-1
          pool-6-thread-1
          pool-7-thread-1
          pool-4-thread-1
          DynamicReloader
          pool-2-thread-1
          transaction-manager
          AutoDeployer
          Signal Dispatcher
          pool-10-thread-1
          pool-9-thread-1
          pool-5-thread-1
          pool-3-thread-1

          Show
          sirajg added a comment - After a start, deploy, stop, I see these threads (Thread.getAllStackTraces) : Keep-Alive-Timer Reference Handler pool-8-thread-1 pool-1-thread-1 pool-6-thread-1 pool-7-thread-1 pool-4-thread-1 DynamicReloader pool-2-thread-1 transaction-manager AutoDeployer Signal Dispatcher pool-10-thread-1 pool-9-thread-1 pool-5-thread-1 pool-3-thread-1
          Hide
          Bhavanishankar added a comment -

          This issue is reproducible in v3.1 as well.

          This is a core issue in glassfish, not particularly an embedded issue. But the
          embedded users will have the impact. Bunch of threads belonging to different
          modules remain even after GlassFishRuntime.shutdown(), so we need to go through
          each of them and see why they remain.

          May not be able to fix this for v3.1, hence updating the keyword.

          Show
          Bhavanishankar added a comment - This issue is reproducible in v3.1 as well. This is a core issue in glassfish, not particularly an embedded issue. But the embedded users will have the impact. Bunch of threads belonging to different modules remain even after GlassFishRuntime.shutdown(), so we need to go through each of them and see why they remain. May not be able to fix this for v3.1, hence updating the keyword.
          kenaiadmin made changes -
          Field Original Value New Value
          issue.field.bugzillaimportkey 11088 42692
          Bhavanishankar made changes -
          Tags 3_1-exclude 3_1-exclude 3_1_1-scrubbed
          Hide
          Bhavanishankar added a comment -

          Simple testcase.

          Instructions to run the test case:

          1. Compile it:

          javac -g -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar GlassFish11088.java

          2. Run it

          java -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar:. GlassFish11088

          3. When it prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack.

          In the jstack you will see number of glassfish threads still running. (I have attached the jstack output that I got with this testcase)

          Once this issue is fixed, there should not be any running glassfish threads in the jstack.

          Show
          Bhavanishankar added a comment - Simple testcase. Instructions to run the test case: 1. Compile it: javac -g -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar GlassFish11088.java 2. Run it java -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar:. GlassFish11088 3. When it prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack. In the jstack you will see number of glassfish threads still running. (I have attached the jstack output that I got with this testcase) Once this issue is fixed, there should not be any running glassfish threads in the jstack.
          Bhavanishankar made changes -
          Attachment GlassFish11088.java [ 46068 ]
          Attachment jstack [ 46069 ]
          Satish Kumar made changes -
          Assignee Bhavanishankar [ bhavanishankar ] Satish Kumar [ sats ]
          Hide
          brdjcx added a comment -

          Bug two years old and "No work has yet been logged on this issue."?

          May be tolerable on Unix/Mac but a stopper on Windows. Those stray processes hold files open that IDEs/maven want to delete (mvn clean). The workaround on Windows goes like this: stop GF, stop Netbeans, kill the strays with TaskManager, and start everything back up and wait, wait, wait. Dozens of times per hour. NOT good.

          Show
          brdjcx added a comment - Bug two years old and "No work has yet been logged on this issue."? May be tolerable on Unix/Mac but a stopper on Windows. Those stray processes hold files open that IDEs/maven want to delete (mvn clean). The workaround on Windows goes like this: stop GF, stop Netbeans, kill the strays with TaskManager, and start everything back up and wait, wait, wait. Dozens of times per hour. NOT good.
          Hide
          Tim Quinn added a comment - - edited

          partial fix checked into the 3.1.1 branch and the trunk. The rev for the 3.1.1 branch is

          Project: glassfish
          Repository: svn
          Revision: 47365
          Author: tjquinn
          Date: 2011-06-08 16:02:35 UTC
          Link:

          Here is the full check-in log for the trunk check-in. The files changes and the diffs are the same for both check-ins.

          Project: glassfish
          Repository: svn
          Revision: 47366
          Author: tjquinn
          Date: 2011-06-08 16:06:58 UTC
          Link:

          Log Message:
          ------------
          Partial fix for 11088

          The dynamic reloader and auto-deployer daemon threads were not stopped as part of shutdown. This caused problems during embedded use.

          These changes correct where the "cancel requested" indicator is cleared.

          Tests: autodeploy and "touch .reload" manual tests

          Revisions:
          ----------
          47366

          Modified Paths:
          ---------------
          trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployService.java
          trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloadService.java
          trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloader.java
          trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployer.java

          Show
          Tim Quinn added a comment - - edited partial fix checked into the 3.1.1 branch and the trunk. The rev for the 3.1.1 branch is Project: glassfish Repository: svn Revision: 47365 Author: tjquinn Date: 2011-06-08 16:02:35 UTC Link: Here is the full check-in log for the trunk check-in. The files changes and the diffs are the same for both check-ins. Project: glassfish Repository: svn Revision: 47366 Author: tjquinn Date: 2011-06-08 16:06:58 UTC Link: Log Message: ------------ Partial fix for 11088 The dynamic reloader and auto-deployer daemon threads were not stopped as part of shutdown. This caused problems during embedded use. These changes correct where the "cancel requested" indicator is cleared. Tests: autodeploy and "touch .reload" manual tests Revisions: ---------- 47366 Modified Paths: --------------- trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployService.java trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloadService.java trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloader.java trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployer.java
          Hide
          Bhavanishankar added a comment -

          Apart from the simple start/stop of embedded GlassFish, we will also have to check which threads keep running in the start/deploy/undeploy/stop scenario.

          To do that, the existing devtest can be used like this:

          1. cd v3/tests/embedded/cdi_ejb_jpa (in your 3.1.1 workspace).

          2. Modify the test a bit so that it gives a chance to take the jstack/jmap before it exits, like this:

          
          --- src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java    (revision 47440)
          +++ src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java    (working copy)
          @@ -105,6 +105,11 @@
           
                   glassfish.dispose();
           
          +               System.out.println("GlassFish has been stopped and disposed. " +
          +                                   "After taking jstack, press any key to quit this program");
          +
          +                       new BufferedReader(new InputStreamReader(System.in)).readLine();
          +
               }   
           
               private void get(String urlStr, String result) throws Exception {
          
          

          3. Set S1AS_HOME to your 3.1.1 glassfish image (eg., /space/image/glassfish3/glassfish)

          4. Execute the test : mvn -P run-with-static-shell clean verify

          5. Once the test finishes and prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack/jmap of the "SurefireBooter" process.

          Show
          Bhavanishankar added a comment - Apart from the simple start/stop of embedded GlassFish, we will also have to check which threads keep running in the start/deploy/undeploy/stop scenario. To do that, the existing devtest can be used like this: 1. cd v3/tests/embedded/cdi_ejb_jpa (in your 3.1.1 workspace). 2. Modify the test a bit so that it gives a chance to take the jstack/jmap before it exits, like this: --- src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java (revision 47440) +++ src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java (working copy) @@ -105,6 +105,11 @@ glassfish.dispose(); + System .out.println( "GlassFish has been stopped and disposed. " + + "After taking jstack, press any key to quit this program" ); + + new BufferedReader( new InputStreamReader( System .in)).readLine(); + } private void get( String urlStr, String result) throws Exception { 3. Set S1AS_HOME to your 3.1.1 glassfish image (eg., /space/image/glassfish3/glassfish) 4. Execute the test : mvn -P run-with-static-shell clean verify 5. Once the test finishes and prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack/jmap of the "SurefireBooter" process.
          Hide
          Satish Kumar added a comment -

          Out of the threads mentioned in the jstack, the following have been resolved-

          DynamicReloader
          AutoDeployer

          The following threads are started by the event notification framework (EventsImpl) and return eventually after notifying the event listeners. This is taking a bit of time 30-60 secs before they return. But these are not long running threads.

          pool-2-thread-1
          pool-3-thread-1
          pool-4-thread-1
          pool-5-thread-1

          transaction-manager:
          The jta module starts a timer service from JavaEETransactionManagerSimplified which is not stopped during appserver shutdown. I have created a separate issue to track this - 16873.

          Show
          Satish Kumar added a comment - Out of the threads mentioned in the jstack, the following have been resolved- DynamicReloader AutoDeployer The following threads are started by the event notification framework (EventsImpl) and return eventually after notifying the event listeners. This is taking a bit of time 30-60 secs before they return. But these are not long running threads. pool-2-thread-1 pool-3-thread-1 pool-4-thread-1 pool-5-thread-1 transaction-manager: The jta module starts a timer service from JavaEETransactionManagerSimplified which is not stopped during appserver shutdown. I have created a separate issue to track this - 16873.
          Satish Kumar made changes -
          Tags 3_1-exclude 3_1_1-scrubbed 3_1-exclude 3_1_1-review 3_1_1-scrubbed
          Satish Kumar made changes -
          Link This issue depends on GLASSFISH-16873 [ GLASSFISH-16873 ]
          Hide
          Satish Kumar added a comment -

          Linking this issue to GF 16873...

          Show
          Satish Kumar added a comment - Linking this issue to GF 16873...
          scatari made changes -
          Tags 3_1-exclude 3_1_1-review 3_1_1-scrubbed 3_1-exclude 3_1_1-approved 3_1_1-review 3_1_1-scrubbed
          scatari made changes -
          Tags 3_1-exclude 3_1_1-approved 3_1_1-review 3_1_1-scrubbed 3_1-exclude 3_1_1-approved 3_1_1-scrubbed
          Hide
          scatari added a comment -

          Pre-approving for 3.1.1 as this is needed for embedded.

          Show
          scatari added a comment - Pre-approving for 3.1.1 as this is needed for embedded.
          Satish Kumar made changes -
          Tags 3_1-exclude 3_1_1-approved 3_1_1-scrubbed 3_1-exclude 3_1_1-approved 3_1_1-next 3_1_1-scrubbed
          Satish Kumar made changes -
          Tags 3_1-exclude 3_1_1-approved 3_1_1-next 3_1_1-scrubbed 3_1-exclude 3_1_1-next 3_1_1-scrubbed
          Hide
          Satish Kumar added a comment -

          marking this for the next release since this issue is still waiting for 16873 which is unlikely to get fixed for 3.1.1

          Show
          Satish Kumar added a comment - marking this for the next release since this issue is still waiting for 16873 which is unlikely to get fixed for 3.1.1
          Rebecca Parks made changes -
          Tags 3_1-exclude 3_1_1-next 3_1_1-scrubbed 3_1-exclude 3_1-next_release-notes 3_1_1-next 3_1_1-scrubbed
          Rebecca Parks made changes -
          Tags 3_1-exclude 3_1-next_release-notes 3_1_1-next 3_1_1-scrubbed 3_1-exclude 3_1-next_release-note-added 3_1-next_release-notes 3_1_1-next 3_1_1-scrubbed
          Hide
          Bhavanishankar added a comment -

          .

          Show
          Bhavanishankar added a comment - .
          Bhavanishankar made changes -
          Assignee Satish Kumar [ sats ] sakshi.jain [ sakshi.jain ]
          Hide
          Tom Mueller added a comment -

          Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

          Show
          Tom Mueller added a comment - Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.
          Tom Mueller made changes -
          Fix Version/s not determined [ 11149 ]
          Fix Version/s 3.1 [ 10968 ]

            People

            • Assignee:
              sakshi.jain
              Reporter:
              bavik78
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: