Issue Details (XML | Word | Printable)

Key: GLASSFISH-17094
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Tim Quinn
Reporter: ldaroczi
Votes: 1
Watchers: 6
Operations

If you were logged in you would be able to see more operations.
glassfish

deployment of war files packaged in ear is wery slow (often causes timeout: No response from Domain Admin Server after 600 seconds.)

Created: 23/Jul/11 03:53 PM   Updated: 11/Jun/12 11:58 PM   Resolved: 19/Aug/11 10:12 PM
Component/s: deployment
Affects Version/s: 3.1.1_b12
Fix Version/s: 3.1.2_b02, 4.0

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive issue.zip (3.78 MB) 23/Jul/11 03:53 PM - ldaroczi

Environment:

Windows 7 Ultimate x64;
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)

Issue Links:
Duplicate
 
Related
 

Tags:
Participants: ldaroczi, mkarg, Rebecca Parks, Tim Quinn and ymajoros


 Description  « Hide

We have several enterprise applications which contain web modules developed with Google Web toolkit.
The applications deploy and run correctly with GlassFish 3.1.
I have tried to deploy the same enterprise applications to GlassFish 3.1.1 b12 but the deployment fails with the following message:

asadmin> deploy app.ear
No response from Domain Admin Server after 600 seconds.
The command is either taking too long to complete or the server has failed.
Please see the server log files for command status.
Command deploy failed.

It seams that the extraction of the war file included in the ear is very slow, it takes more than 10 minutes to complete the extraction of files.
If I deploy the war file standalone (not packaged in ear) then the deploy is fast and successful like in GlassFish 3.1.

To reproduce the issue i have attached an ear file with a war file wich contains only the static files (images, java script, HTML, CSS) from a real application.
The deployment of this ear takes more than 2,5 minutes (154 739 milliseconds) with GlassFish 3.1.1 b12. The same ear deploys in 3 456 milliseconds with GlassFish 3.1. This is a performance decrease of 4 477%!!!
The deployment of the war file standalone (not packaged in ear) takes 3 085 milliseconds with GlassFish 3.1.1 b12.
I have also included the server.log files from the tests.



ymajoros added a comment - 13/Dec/11 03:16 PM

Nice:

EPCFull was successfully deployed in 23,155 milliseconds.

Which is about the same as my best try.


Tim Quinn added a comment - 12/Dec/11 04:35 PM

ymajoros,

What difference in deployment speed did you see, if any, for your app using 3.1.2 b02 or later, without your own changes?


ymajoros added a comment - 09/Dec/11 03:06 PM - edited

I thought I was impacted by this, but I had another problem. I could drastically improve deploy time (from 350s to 30s, same application):

http://ymajoros.blogspot.com/2011/12/improve-glassfish-deployment-time-by.html


Tim Quinn added a comment - 28/Nov/11 10:31 PM

No doc impact.


Rebecca Parks added a comment - 28/Nov/11 10:07 PM

Does this apply to 3.1.2? If there's a doc impact, please file a related doc bug.


Tim Quinn added a comment - 19/Aug/11 10:12 PM

Fix checked in.

Project: glassfish
Repository: svn
Revision: 48935
Author: tjquinn
Date: 2011-08-19 22:10:05 UTC
Link:

Log Message:
------------
Fix for 17094

An earlier change addressed problems with "stale" files left in the applications/appName directory which confused later deployments using the same app name. There were actually two attempts to fix this. The first attempt involved reading submodules from the original archive, not the expanded directory. The second attempt does not require that but this logic from the original fix was left as-is. Reading large submodules from the archive is much slower than reading them from the expanded directory. We no longer need to do and this check-in changes that.

Tests: deployment devtests, QL, manual tests of the bug scenario that gave rise to 13774 (the original "stale" file problem)

Revisions:
----------
48935

Modified Paths:
---------------
branches/3.1.2/deployment/javaee-full/src/main/java/org/glassfish/javaee/full/deployment/EarHandler.java


Tim Quinn added a comment - 17/Aug/11 03:51 PM

I am marking this as fixed in 3.2 and leaving it open for (we hope) inclusion in 3.1.2.

Fix checked into the trunk:

Project: glassfish
Repository: svn
Revision: 48852
Author: tjquinn
Date: 2011-08-17 15:36:18 UTC
Link:

Log Message:
------------
Fix for 17094

An earlier change addressed problems with "stale" files left in the applications/appName directory which confused later deployments using the same app name. There were actually two attempts to fix this. The first attempt involved reading submodules from the original archive, not the expanded directory. The second attempt does not require that but this logic from the original fix was left as-is. Reading large submodules from the archive is much slower than reading them from the expanded directory. We no longer need to do and this check-in changes that.

Tests: deployment devtests, QL, manual tests of the bug scenario that gave rise to 13774 (the original "stale" file problem)

Revisions:
----------
48852

Modified Paths:
---------------
trunk/main/appserver/deployment/javaee-full/src/main/java/org/glassfish/javaee/full/deployment/EarHandler.java


Tim Quinn added a comment - 14/Aug/11 02:05 PM

Yes. Most but not quite all of the testing I mentioned earlier is done. Then some other work came up I had to take care of first. I expect that this week I'll be checking in the fix for this if the remaining testing is successful.


ldaroczi added a comment - 13/Aug/11 07:30 PM

Any progress solving this issue?


Tim Quinn added a comment - 27/Jul/11 12:09 PM

Markus,

Such a change would break other features of GlassFish. You might be aware of the .reload feature, in which a user can create or update the file .reload in the application directory which GlassFish creates for the application. When GlassFish detects that the .reload file has appeared or has changed, it essentially redeploys the application using the bits there in the directory.

Although this is usually most useful for users who perform directory deployments, it also works - and is documented and supported - with the GlassFish-managed expansion directory for the archive.

A fix is currently undergoing testing.


mkarg added a comment - 27/Jul/11 06:42 AM

If the problem really is caused solely by the fact that you cannot delete a JAR that was in use by the JVM before, the solution is as simple as creating a new directory for every EAR deployment, and give the directory a unique name and do not reuse existing directories. Possible solutions could be using an always-increasing counter as the name, or a GUID, or the current timestamp, or use the Windows Tempoprary File API via File.createTempFile.


Tim Quinn added a comment - 26/Jul/11 05:55 PM

Associating this issue with the original one that Elena found which ultimately identified the "stale file problem." The fix for that problem has caused the slowdown reported in this issue as an unintended side effect.


Tim Quinn added a comment - 26/Jul/11 04:46 PM

The slow-down is happening because of a side-effect of an earlier fix to a Windows-only problem.

On Windows an open file cannot be deleted (during undeployment or redeployment for example). So if you revise an app so it no longer includes a given file, but that file could not be removed from the GlassFish application directory, then when you redeploy that app GlassFish used to still see that "stale" file and could get confused by it. This would happen particularly if the file was a directory for a subarchive within an EAR.

As part of the fix for that problem, we changed the code which expands an app into the GlassFish application directory so that when it processes submodules (a WAR within an EAR, for example) it reads from the WAR as an entry inside the original EAR, rather than from the as-yet partially filled expansion directory. That's where the problem comes in, because accessing an archive within an archive is very slow. When it has to be done many times - as in expanding the files from inside a large WAR that's inside an EAR - then the cost is very visible.

I'm investigating how best to take care of this.


Tim Quinn added a comment - 26/Jul/11 02:47 PM

I had thought this might be a Windows-specific issue, but the deployment time is about the same on a Mac.

I'll run some profiling to see where the time is being spent.


Tim Quinn added a comment - 25/Jul/11 07:22 PM

I will take a look. This looks similar to 17031.