Issue Details (XML | Word | Printable)

Key: GLASSFISH-15691
Type: Bug Bug
Status: Resolved Resolved
Resolution: Cannot Reproduce
Priority: Major Major
Assignee: Hong Zhang
Reporter: easarina
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
glassfish

Regression. Win. deploy with a version, undeploy stateless-simple, then webapps-caching - got a deployment failure.

Created: 25/Jan/11 08:10 PM   Updated: 26/Oct/12 04:10 PM   Resolved: 26/Oct/12 04:10 PM
Component/s: deployment
Affects Version/s: None
Fix Version/s: 4.0

Time Tracking:
Not Specified

File Attachments: 1. File stateless-simple.ear (10 kB) 25/Jan/11 08:10 PM - easarina
2. File webapps-caching.war (14 kB) 25/Jan/11 08:10 PM - easarina

Environment:

Windows


Tags: 3_1-exclude 3_1-regression 3_1_1-scrubbed 3_1_2-exclude
Participants: Alex Pineda, easarina, Hong Zhang, Jeremy_Lv, sb110099, scatari, Shing Wai Chan and Tim Quinn


 Description  « Hide

Build 39. Windows machines.

Installed a glassfish on one machine, created a cluster with two instances. Executed such commands:
============================================================
1) asadmin deploy --name temp:1 --contextroot helloworld --target c1 C:/hudson/workspace/deployment/appserver-sqe/pe/deployment_v3/version/stateless-simple.ear

Application deployed with name temp:1.
Command deploy executed successfully.

2) asadmin undeploy --target c1 temp:*
Command undeploy executed successfully.

3) asadmin deploy --name temp:1 --contextroot helloworld --target c1 C:/hudson/workspace/deployment/appserver-sqe/pe/deployment_v3/version/webapps-caching.war

remote failure: Error occurred during deployment: Exception while deploying the app [temp:1] : Application [temp:1] contains no valid components. Please see server.log for more details.
Command deploy failed.
remote failure: Application temp has no version registered
===========================================================

If to deploy/undeploy webapps-caching first and stateless-simple after that, everything will be OK.

This is a regression bug. I did not see this issue before. I saw this issue only on Windows machines.

To reproduce the issue can be executed a hudson job:

http://sqe-hudson.us.oracle.com:8080/hudson/view/Elena_Jobs/job/deployment/



Hong Zhang added a comment - 26/Jan/11 05:49 AM

This seems similar to issue 13774, assign to Tim to take a look


Tim Quinn added a comment - 26/Jan/11 09:19 AM

This is a duplicate of Issue 13774.

In this case, something in the web container is opening but not closing a JAR in the first application's WEB-INF/LIB directory. On Windows, this file cannot be deleted so when the second (completely different) app is deployed with the same name, the same directories are used. GlassFish sees the left-over file which confuses its decision-making about what type of module is being deployed.

The stack trace below shows where the JAR was opened.

show
Current list of opened but unclosed jar files matching the filter:
Path "C:\gf\glassfish3\glassfish\domains\domain1\applications\temp~1\stateless-simple_war\WEB-INF\lib\stateless-simpleEjb.jar"
..Opened by hashCode object 1157 from:
java.util.jar.JarFile.<init>(java\util\jar\JarFile.java:137)
java.util.jar.JarFile.<init>(java\util\jar\JarFile.java:99)
com.sun.enterprise.glassfish.web.WarHandler.configureLoaderProperties(com\sun\enterprise\glassfish\web\WarHandler.java:250)
com.sun.enterprise.glassfish.web.WarHandler.getClassLoader(com\sun\enterprise\glassfish\web\WarHandler.java:151)
org.glassfish.javaee.full.deployment.EarHandler.getClassLoader(org\glassfish\javaee\full\deployment\EarHandler.java:400)
org.glassfish.deployment.common.DeploymentContextImpl.createClassLoader(org\glassfish\deployment\common\DeploymentContextImpl.java:214)
org.glassfish.deployment.common.DeploymentContextImpl.createApplicationClassLoader(org\glassfish\deployment\common\DeploymentContextImpl.java:204)
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(com\sun\enterprise\v3\server\ApplicationLifecycle.java:399)
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(com\sun\enterprise\v3\server\ApplicationLifecycle.java:240)
org.glassfish.deployment.admin.DeployCommand.execute(org\glassfish\deployment\admin\DeployCommand.java:370)
com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:354)
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:369)
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1061)
com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:95)
com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1260)
com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1248)
com.sun.enterprise.v3.admin.AdminAdapter.doCommand(com\sun\enterprise\v3\admin\AdminAdapter.java:453)
com.sun.enterprise.v3.admin.AdminAdapter.service(com\sun\enterprise\v3\admin\AdminAdapter.java:220)
com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(com\sun\grizzly\tcp\http11\GrizzlyAdapter.java:168)
com.sun.enterprise.v3.server.HK2Dispatcher.dispath(com\sun\enterprise\v3\server\HK2Dispatcher.java:117)
com.sun.enterprise.v3.services.impl.ContainerMapper.service(com\sun\enterprise\v3\services\impl\ContainerMapper.java:234)
com.sun.grizzly.http.ProcessorTask.invokeAdapter(com\sun\grizzly\http\ProcessorTask.java:822)
com.sun.grizzly.http.ProcessorTask.doProcess(com\sun\grizzly\http\ProcessorTask.java:719)
com.sun.grizzly.http.ProcessorTask.process(com\sun\grizzly\http\ProcessorTask.java:1013)
com.sun.grizzly.http.DefaultProtocolFilter.execute(com\sun\grizzly\http\DefaultProtocolFilter.java:225)
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(com\sun\grizzly\DefaultProtocolChain.java:137)
com.sun.grizzly.DefaultProtocolChain.execute(com\sun\grizzly\DefaultProtocolChain.java:104)
com.sun.grizzly.DefaultProtocolChain.execute(com\sun\grizzly\DefaultProtocolChain.java:90)
com.sun.grizzly.http.HttpProtocolChain.execute(com\sun\grizzly\http\HttpProtocolChain.java:79)
com.sun.grizzly.ProtocolChainContextTask.doCall(com\sun\grizzly\ProtocolChainContextTask.java:54)
com.sun.grizzly.SelectionKeyContextTask.call(com\sun\grizzly\SelectionKeyContextTask.java:59)
com.sun.grizzly.ContextTask.run(com\sun\grizzly\ContextTask.java:71)
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(com\sun\grizzly\util\AbstractThreadPool.java:532)
com.sun.grizzly.util.AbstractThreadPool$Worker.run(com\sun\grizzly\util\AbstractThreadPool.java:513)
java.lang.Thread.run(java\lang\Thread.java:662)

>


Tim Quinn added a comment - 26/Jan/11 09:22 AM

It is probably better to leave this issue open - but assigned to the web container team for attention in a release later than 3.1.


Tim Quinn added a comment - 26/Jan/11 09:25 AM

We have changes in the generic deployment logic that will work around problems like this which we plan to check in for 3.2.

In addition we should find and fix places where JARs are opened but not closed.

The stack trace shows one place where this happens.

show
Current list of opened but unclosed jar files matching the filter:
Path "C:\gf\glassfish3\glassfish\domains\domain1\applications\temp~1\stateless-simple_war\WEB-INF\lib\stateless-simpleEjb.jar"
..Opened by hashCode object 1157 from:
java.util.jar.JarFile.<init>(java\util\jar\JarFile.java:137)
java.util.jar.JarFile.<init>(java\util\jar\JarFile.java:99)
com.sun.enterprise.glassfish.web.WarHandler.configureLoaderProperties(com\sun\enterprise\glassfish\web\WarHandler.java:250)
com.sun.enterprise.glassfish.web.WarHandler.getClassLoader(com\sun\enterprise\glassfish\web\WarHandler.java:151)
org.glassfish.javaee.full.deployment.EarHandler.getClassLoader(org\glassfish\javaee\full\deployment\EarHandler.java:400)
org.glassfish.deployment.common.DeploymentContextImpl.createClassLoader(org\glassfish\deployment\common\DeploymentContextImpl.java:214)
org.glassfish.deployment.common.DeploymentContextImpl.createApplicationClassLoader(org\glassfish\deployment\common\DeploymentContextImpl.java:204)
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(com\sun\enterprise\v3\server\ApplicationLifecycle.java:399)
com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(com\sun\enterprise\v3\server\ApplicationLifecycle.java:240)
org.glassfish.deployment.admin.DeployCommand.execute(org\glassfish\deployment\admin\DeployCommand.java:370)
com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:354)
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:369)
com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1061)
com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:95)
com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1260)
com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(com\sun\enterprise\v3\admin\CommandRunnerImpl.java:1248)
com.sun.enterprise.v3.admin.AdminAdapter.doCommand(com\sun\enterprise\v3\admin\AdminAdapter.java:453)
com.sun.enterprise.v3.admin.AdminAdapter.service(com\sun\enterprise\v3\admin\AdminAdapter.java:220)
com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(com\sun\grizzly\tcp\http11\GrizzlyAdapter.java:168)
com.sun.enterprise.v3.server.HK2Dispatcher.dispath(com\sun\enterprise\v3\server\HK2Dispatcher.java:117)
com.sun.enterprise.v3.services.impl.ContainerMapper.service(com\sun\enterprise\v3\services\impl\ContainerMapper.java:234)
com.sun.grizzly.http.ProcessorTask.invokeAdapter(com\sun\grizzly\http\ProcessorTask.java:822)
com.sun.grizzly.http.ProcessorTask.doProcess(com\sun\grizzly\http\ProcessorTask.java:719)
com.sun.grizzly.http.ProcessorTask.process(com\sun\grizzly\http\ProcessorTask.java:1013)
com.sun.grizzly.http.DefaultProtocolFilter.execute(com\sun\grizzly\http\DefaultProtocolFilter.java:225)
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(com\sun\grizzly\DefaultProtocolChain.java:137)
com.sun.grizzly.DefaultProtocolChain.execute(com\sun\grizzly\DefaultProtocolChain.java:104)
com.sun.grizzly.DefaultProtocolChain.execute(com\sun\grizzly\DefaultProtocolChain.java:90)
com.sun.grizzly.http.HttpProtocolChain.execute(com\sun\grizzly\http\HttpProtocolChain.java:79)
com.sun.grizzly.ProtocolChainContextTask.doCall(com\sun\grizzly\ProtocolChainContextTask.java:54)
com.sun.grizzly.SelectionKeyContextTask.call(com\sun\grizzly\SelectionKeyContextTask.java:59)
com.sun.grizzly.ContextTask.run(com\sun\grizzly\ContextTask.java:71)
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(com\sun\grizzly\util\AbstractThreadPool.java:532)
com.sun.grizzly.util.AbstractThreadPool$Worker.run(com\sun\grizzly\util\AbstractThreadPool.java:513)
java.lang.Thread.run(java\lang\Thread.java:662)

>


Shing Wai Chan added a comment - 26/Jan/11 05:05 PM

There are three relevant calling stacks:
A. com.sun.enterprise.glassfish.web.WarHandler.configureLoaderProperties(WarHandler.java:250)
at com.sun.enterprise.glassfish.web.WarHandler.getClassLoader(WarHandler.java:151)
at org.glassfish.javaee.full.deployment.EarHandler.getClassLoader(EarHandler.java:400)
at org.glassfish.deployment.common.DeploymentContextImpl.createClassLoader(DeploymentContextImpl.java:214)
at org.glassfish.deployment.common.DeploymentContextImpl.createDeploymentClassLoader(DeploymentContextImpl.java:197)

B. com.sun.enterprise.glassfish.web.WarHandler.configureLoaderProperties(WarHandler.java:250)
at com.sun.enterprise.glassfish.web.WarHandler.getClassLoader(WarHandler.java:151)
at org.glassfish.javaee.full.deployment.EarHandler.getClassLoader(EarHandler.java:400)
at org.glassfish.deployment.common.DeploymentContextImpl.createClassLoader(DeploymentContextImpl.java:214)
at org.glassfish.deployment.common.DeploymentContextImpl.createApplicationClassLoader(DeploymentContextImpl.java:204)

C. org.glassfish.web.loader.WebappClassLoader.closeJARs(WebappClassLoader.java:1750)
at org.apache.catalina.loader.WebappLoader.closeJARs(WebappLoader.java:516)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5358)
at com.sun.enterprise.web.WebModule.start(WebModule.java:500)

We have the following:
In DAS, (A) and (B).
In inst1 and inst2, (A), (B) and (C) calling for the classloader in (B).

So, the WebappClassLoader are created twice.
And only one of them is closed in instance.
This explains the file locking issue.


Hong Zhang added a comment - 26/Jan/11 06:35 PM

(A) is the temporary classloader used for sniffer retrieval, DOL loading etc. (B) is the actual application classloader used to load the application. (A) should be destroyed when we start to use (B), see DeploymentContextImpl.getClassLoader, we call preDestroy on the sharebleTemp (A) before return (B). Probably WebappClassLoader.preDestroy method should try to close all the open jars as well?


Shing Wai Chan added a comment - 26/Jan/11 07:01 PM - edited

The WebappClassLoader.preDestroy call WebappClassLoader.stop which has already called the jarFiles[i].close().

Further debug shows that inst1 and inst2 with preDestroy being called. But for DAS, only the classloader from (A) has preDestroy being called as follows:

at org.glassfish.web.loader.WebappClassLoader.stop(WebappClassLoader.java:1691)
at org.glassfish.web.loader.WebappClassLoader.preDestroy(WebappClassLoader.java:1636)
at org.glassfish.javaee.full.deployment.EarClassLoader.preDestroy(EarClassLoader.java:91)
at org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:232)
at org.glassfish.deployment.common.DeploymentContextImpl.getClassLoader(DeploymentContextImpl.java:184)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:450)


Hong Zhang added a comment - 27/Jan/11 08:11 AM

Thanks for the investigation! Seems we need to call preDestroy for (B) on DAS in this scenario to make sure all the resources are cleaned up also. I will look into this.


Hong Zhang added a comment - 23/Feb/11 09:47 AM

Cleaned up the application classloader on DAS for deploy/undeploy with cluster target scenario.


sb110099 added a comment - 23/Feb/11 06:30 PM

Regression , only on windows platform


Hong Zhang added a comment - 14/Mar/11 07:09 PM

Remove 3_1-release-notes tag as the same use case was already release noted in issue 13774.


easarina added a comment - 08/Jul/11 01:37 AM

3.1.1 b10. Again saw this issue. Reproduced this issue on two Win 2008 machines.


Hong Zhang added a comment - 08/Jul/11 01:44 AM

When was the last build these tests passed?


easarina added a comment - 08/Jul/11 01:52 AM

I don't know.


easarina added a comment - 08/Jul/11 02:05 AM

Also in the instance server.log I saw such correspondent messages:

[#|2011-07-07T18:35:27.412-0700|SEVERE|glassfish3.1.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=26;_ThreadName=Thread-2;|Exception while deploying the app [temp:1]|#]

[#|2011-07-07T18:35:27.415-0700|SEVERE|glassfish3.1.1|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=26;_ThreadName=Thread-2;|Application [temp:1] contains no valid components
java.lang.IllegalArgumentException: Application [temp:1] contains no valid components
at com.sun.enterprise.deployment.util.ApplicationValidator.accept(ApplicationValidator.java:77)
at com.sun.enterprise.deployment.Application.visit(Application.java:1761)
at com.sun.enterprise.deployment.archivist.ApplicationFactory.openArchive(ApplicationFactory.java:195)
at org.glassfish.javaee.core.deployment.DolProvider.load(DolProvider.java:181)
at org.glassfish.javaee.core.deployment.DolProvider.load(DolProvider.java:93)
at com.sun.enterprise.v3.server.ApplicationLifecycle.loadDeployer(ApplicationLifecycle.java:828)
at com.sun.enterprise.v3.server.ApplicationLifecycle.setupContainerInfos(ApplicationLifecycle.java:770)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:368)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:186)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:355)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:370)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1045)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:96)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1244)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1232)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:459)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:209)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)


Alex Pineda added a comment - 08/Jul/11 04:53 PM

This bug was marked as a duplicate of 13774 and it was release noted in GF 3.1, thus, I assume the test would not have passed in GF 3.1.1. I'm looking for results from old run to verify. Regardless, it looks like we want to Release Note this issue again in GF 3.1.1. I hope everyone agrees.


easarina added a comment - 08/Jul/11 04:59 PM

For 3.1.1 b10 bug 13774 is fixed. But this bug still exists.


scatari added a comment - 04/Nov/11 09:57 PM

Please evaluate this for possible inclusion into 3.1.2.


easarina added a comment - 14/Dec/11 02:16 AM

GF 3.1.2 b14, Win 2008. Still saw this issue.


Jeremy_Lv added a comment - 09/Aug/12 09:40 AM

I'll try to investigate this issue, I think the issue may be similar to:
http://java.net/jira/browse/GLASSFISH-18789


Jeremy_Lv added a comment - 22/Aug/12 06:36 AM

I can't reappear this phenomenon about GF3.1.2 b23 and the lastest version of GF 4.0_b49 on the environment of windows xp and win 2008.
Did this issue revised by anyone else?


Hong Zhang added a comment - 28/Aug/12 01:27 PM

Jeremy: This might be an intermittent issue or needs to be reproduced through a special set of the steps.

Elena: Can you check if the issue still exists in latest GlassFish builds? If yes, can you document the exact set of the steps to reproduce the issue? Thanks.


Hong Zhang added a comment - 26/Oct/12 04:10 PM

Close the issue as we haven't heard back from the bug filer. Please re-open the issue if the problem still exists, and please attach the test cases/steps which could be used to reproduce the problem.