[GLASSFISH-11088] [embedded] Stopping GFv3 leaves several daemon threads running Created: 18/Nov/09  Updated: 06/Mar/12

Status: Open
Project: glassfish
Component/s: embedded
Affects Version/s: V3
Fix Version/s: not determined

Type: Bug Priority: Major
Reporter: bavik78 Assignee: sakshi.jain
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Windows XP
Platform: PC


Attachments: Java Source File GlassFish11088.java     HTML File jstack    
Issue Links:
Dependency
depends on GLASSFISH-16873 The transaction-manager timer thread ... Resolved
Issuezilla Id: 11,088
Status Whiteboard:

v3_exclude

Tags: 3_1-exclude, 3_1-next_release-note-added, 3_1-next_release-notes, 3_1_1-next, 3_1_1-scrubbed

 Description   

After stopping the server there is a bunch of daemon threads:
transaction-manager, AutoDeployer, DynamicReloader, pool-X-thread-Y.

Most of these threads (all except of some instances of pool-X-thread-Y) are
never finished until VM exit.
In the use case, when embedded GF is restarted many times during main
application session this can cause "Out-Of-Memory" problem and other side effects.



 Comments   
Comment by bavik78 [ 18/Nov/09 ]

2 CC added

Comment by dochez [ 18/Nov/09 ]

not a showstopper, few deamon threads hanging around after server.close()

Comment by kumara [ 07/Dec/09 ]

Setting target release for unresolved issues submitted on v3 release to the next release. Not changing
issues submitted on v2.x release because they might not apply to v3.next release.

Comment by sirajg [ 08/Jul/10 ]

transfer

Comment by sirajg [ 09/Jul/10 ]

After a start, deploy, stop, I see these threads (Thread.getAllStackTraces) :
Keep-Alive-Timer
Reference Handler
pool-8-thread-1
pool-1-thread-1
pool-6-thread-1
pool-7-thread-1
pool-4-thread-1
DynamicReloader
pool-2-thread-1
transaction-manager
AutoDeployer
Signal Dispatcher
pool-10-thread-1
pool-9-thread-1
pool-5-thread-1
pool-3-thread-1

Comment by Bhavanishankar [ 10/Oct/10 ]

This issue is reproducible in v3.1 as well.

This is a core issue in glassfish, not particularly an embedded issue. But the
embedded users will have the impact. Bunch of threads belonging to different
modules remain even after GlassFishRuntime.shutdown(), so we need to go through
each of them and see why they remain.

May not be able to fix this for v3.1, hence updating the keyword.

Comment by Bhavanishankar [ 01/Jun/11 ]

Simple testcase.

Instructions to run the test case:

1. Compile it:

javac -g -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar GlassFish11088.java

2. Run it

java -cp $S1AS_HOME/lib/embedded/glassfish-embedded-static-shell.jar:. GlassFish11088

3. When it prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack.

In the jstack you will see number of glassfish threads still running. (I have attached the jstack output that I got with this testcase)

Once this issue is fixed, there should not be any running glassfish threads in the jstack.

Comment by brdjcx [ 07/Jun/11 ]

Bug two years old and "No work has yet been logged on this issue."?

May be tolerable on Unix/Mac but a stopper on Windows. Those stray processes hold files open that IDEs/maven want to delete (mvn clean). The workaround on Windows goes like this: stop GF, stop Netbeans, kill the strays with TaskManager, and start everything back up and wait, wait, wait. Dozens of times per hour. NOT good.

Comment by Tim Quinn [ 08/Jun/11 ]

partial fix checked into the 3.1.1 branch and the trunk. The rev for the 3.1.1 branch is

Project: glassfish
Repository: svn
Revision: 47365
Author: tjquinn
Date: 2011-06-08 16:02:35 UTC
Link:

Here is the full check-in log for the trunk check-in. The files changes and the diffs are the same for both check-ins.

Project: glassfish
Repository: svn
Revision: 47366
Author: tjquinn
Date: 2011-06-08 16:06:58 UTC
Link:

Log Message:
------------
Partial fix for 11088

The dynamic reloader and auto-deployer daemon threads were not stopped as part of shutdown. This caused problems during embedded use.

These changes correct where the "cancel requested" indicator is cleared.

Tests: autodeploy and "touch .reload" manual tests

Revisions:
----------
47366

Modified Paths:
---------------
trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployService.java
trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloadService.java
trunk/v3/core/kernel/src/main/java/com/sun/enterprise/v3/server/DynamicReloader.java
trunk/v3/deployment/autodeploy/src/main/java/org/glassfish/deployment/autodeploy/AutoDeployer.java

Comment by Bhavanishankar [ 14/Jun/11 ]

Apart from the simple start/stop of embedded GlassFish, we will also have to check which threads keep running in the start/deploy/undeploy/stop scenario.

To do that, the existing devtest can be used like this:

1. cd v3/tests/embedded/cdi_ejb_jpa (in your 3.1.1 workspace).

2. Modify the test a bit so that it gives a chance to take the jstack/jmap before it exits, like this:


--- src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java    (revision 47440)
+++ src/test/java/org/glassfish/tests/embedded/cdi_ejb_jpa/BasicCDITest.java    (working copy)
@@ -105,6 +105,11 @@
 
         glassfish.dispose();
 
+               System.out.println("GlassFish has been stopped and disposed. " +
+                                   "After taking jstack, press any key to quit this program");
+
+                       new BufferedReader(new InputStreamReader(System.in)).readLine();
+
     }   
 
     private void get(String urlStr, String result) throws Exception {

3. Set S1AS_HOME to your 3.1.1 glassfish image (eg., /space/image/glassfish3/glassfish)

4. Execute the test : mvn -P run-with-static-shell clean verify

5. Once the test finishes and prompts "GlassFish has been stopped and disposed. After taking jstack, press any key to quit this program", please take the jstack/jmap of the "SurefireBooter" process.

Comment by Satish Kumar [ 20/Jun/11 ]

Out of the threads mentioned in the jstack, the following have been resolved-

DynamicReloader
AutoDeployer

The following threads are started by the event notification framework (EventsImpl) and return eventually after notifying the event listeners. This is taking a bit of time 30-60 secs before they return. But these are not long running threads.

pool-2-thread-1
pool-3-thread-1
pool-4-thread-1
pool-5-thread-1

transaction-manager:
The jta module starts a timer service from JavaEETransactionManagerSimplified which is not stopped during appserver shutdown. I have created a separate issue to track this - 16873.

Comment by Satish Kumar [ 21/Jun/11 ]

Linking this issue to GF 16873...

Comment by scatari [ 25/Jun/11 ]

Pre-approving for 3.1.1 as this is needed for embedded.

Comment by Satish Kumar [ 06/Jul/11 ]

marking this for the next release since this issue is still waiting for 16873 which is unlikely to get fixed for 3.1.1

Comment by Bhavanishankar [ 01/Dec/11 ]

.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

Generated at Tue Sep 01 13:33:20 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.