[GLASSFISH-16958] Administration Console is gone after a while Created: 05/Jul/11  Updated: 21/Jul/11  Resolved: 21/Jul/11

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1
Fix Version/s: 3.1.1_b11

Type: Bug Priority: Major
Reporter: mkarg Assignee: Anissa Lam
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Win7 Pro SP1 64 Bit de_DE


Attachments: File HowToCrashAdminGUI.wmv    
Tags: 3_1-next, 3_1_1-scrubbed

 Description   

Sporadically it happens that administration console is just "gone":

It happened two times in the past day that on http://localhost:4848 the following text is shown:

oracle.com
GlassFish Server 3.1

Your server is now running
To replace this page, overwrite the file index.html in the document root folder of this server. The document root folder for this server is the docroot subdirectory of this server's domain directory.

To manage a server on the local host with the default administration port, go to the Administration Console.

(etc.)

The funny thing is that the the word "Administration Console" actually is a link to http://localhost:4848, so clicking on it shows exactly the same page again.

The server.log does not tell anything about this strange circumstances: Only the usual message are contained that the console was started.



 Comments   
Comment by Anissa Lam [ 05/Jul/11 ]

Please provide more info on how to reproduce this. Sounds like console got undeployed.
What were you doing at that time ? Have you leave it idle for sometime so timeout clickes in ?
so, you restart the server to get back to the console ?
Have you tried clear the browser cache, restart the browser to see how it goes ?

We fixed some issue relating to timeout in 3.1.1.
If you can try the latest promoted build http://dlc.sun.com.edgesuite.net/glassfish/3.1.1/promoted/ and see if you are still seeing that, that will help.

Comment by Anissa Lam [ 05/Jul/11 ]

Since we have passed HCF and there is no easy way to reproduce this, (I have tried and also never seen this before), I am targeting this for 3.2.

Comment by mkarg [ 06/Jul/11 ]

Meanwhile we are two people over here having had that problem, so it seems it is not dependend on my local installation. Both are using Win7 Pro SP1 64 Bit de_DE, installed using the GlassFish Installer EXE for Windows (EN, not ML).

It seems there is no clear way to reproduce it, as it happens sporadically, but we both have a strong feeling that it is a combination of doing asadmin undeploy on our own EAR after leaving the GUI open for a while in the browser.

Just a minute ago I managed to reproduce the problem with the following sequence, but I am not sure whether this will ALWAYS reproduce it:

asadmin start-domain
Open http://localhost:4848 in browser
Click on "Applications" node to see installed EARs (there are none)
Do NOT close browser or click somewhere.
asadmin deploy MyApp.ear
In the already opened browser, once more click on "Applications" (now "MyApp" is shown).
Do NOT close browser or click somewhere.
asadmin undeploy MyApp
In the already opened "Applications" view, click on "MyApp" (which is still be seen as the browser did not do an automatic refresh).
The screen now gets corrupted: While most of the screen is intact, some boxes show 404.
Click on the "Applications" node again (now the complete screen turns into a 404).
Type "http://localhost:4848" in the adress line of the browser (now you have produced the problem: Browser shows "Your server is running").

I was able to reproduce this (at least) two times in a row, without notable waiting times (so it cannot be a timeout).

It is neither a browser nor cache problem. After detecting that IE9 presented that "Your server is running" screen instead of the admin GUI, I opened Firefox and got the same screen. So the server seems to actively sent it to both browsers (I never used Firefox with GFv3 before, so it cannot be taken out of Firefox's cache).

An explicit cache clear (F5 key) does not solve the problem.

After doing asadmin stop-domain followed by asadmin start-domain, still the "wrong" screen is shown, but THEN a cache clear (F5 key) DOES solve the problem and restarts the GUI.

About the target release of this bug report I need to say that for a production grade system this is an unacceptable showstopper (as one has to stop the server to solve it) and such should prevent publication of 3.1.1 without fixing it.

If you still cannot reproduce even with above steps, please let me know. I then will download and try with latest 3.1.1 promo.

Comment by Anissa Lam [ 07/Jul/11 ]

Thanks for the detail steps.
I am trying with my build of 3.1.1, which has the latest code, and cannot reproduce the problem on my Mac.
I have a very strong feeling that this is related to GLASSFISH-16223 and GLASSFISH-16112.
Undeploying an application (JSF in that case) may bring down the console.
Since 3.1 is released, the application deployment code in GUI in 3.1.1 has been rewritten to also use REST and those problem is resolved.
Since it sounds like you are seeing this quite often, please see if you can reproduce this with latest 3.1.1 promoted build , available at http://dlc.sun.com.edgesuite.net/glassfish/3.1.1/promoted/

Comment by mkarg [ 11/Jul/11 ]

I will try RC1 as soon as it got published.

Comment by mkarg [ 13/Jul/11 ]

Tried again using GFv3.1.1_b11 and the problem still exists.

In fact it looks like it even got worse, as virtually after every fifth simple undeploy-deploy cycle the GUI is gone now.

Comment by Anissa Lam [ 13/Jul/11 ]

I still cannot reproduce the problem on my Mac.
Can you please attach the application you are using to reproduce the problem ?

Comment by mkarg [ 14/Jul/11 ]

Can you please request SuperSimple.ear from Tim Quinn (tjquinn)? He has permission to share it with all Oracle employees for cases liket his one (but please erase file after case is solved, as this EAR is confidential closed source).

Comment by Anissa Lam [ 14/Jul/11 ]

sure, I will work with Tim on this.

Comment by Anissa Lam [ 14/Jul/11 ]

One question. Can you reproduce the problem if you just use repeatedly using asadmin (CLI) to deploy/undeploy your application ?

Comment by Anissa Lam [ 14/Jul/11 ]

Obtained SuperSimple.ear from Tim, however, even after creating the required JDBC resource, I get errors relating to persistence and deployment failed.
I need instructions on what needs to be performed first before deploying the app.

Comment by mkarg [ 15/Jul/11 ]

It shouldn't be necessary to make the application really "work" (i. e. find the needed resources) for this issue – I did not create most of that resources (and get the same warnings) but this issues can be reproduced, though. But if you like you can create the needed resources. This template should tell you the seetings you need to mannually do (the EAR does not check java:app, so the file does not solve by itself):

<resources>
<connector-connection-pool name="QUIPSY_KERNEL" resource-adapter-name="SuperSimple#KernelConnector" connection-definition-name="de.quipsy.connector.kernel.api.outbound.KernelConnectionFactory" ping="true" />
<connector-resource jndi-name="QUIPSY_KERNEL" pool-name="QUIPSY_KERNEL" />
<jdbc-connection-pool name="QUIPSY4" datasource-classname="ianywhere.ml.jdbcodbc.jdbc3.ASADataSource" ping="true">
<property name="URL" value="jdbc:ianywhere:DSN=QUIPSY 4 DBFILE" />
</jdbc-connection-pool>
<jdbc-resource jndi-name="jdbc/QUIPSY/5" pool-name="QUIPSY4" />
<mail-resource jndi-name="QUIPSY_MAIL" host="your-server" user="anissa" from="anissa" />
<admin-object-resource res-adapter="jmsra" res-type="javax.jms.ConnectionFactory" jndi-name="QUIPSY_CONNECTION_FACTORY">
<property name="MaxPoolSize" value="1000" />
</admin-object-resource>
<admin-object-resource res-adapter="jmsra" res-type="javax.jms.Topic" jndi-name="QUIPSY_TOPIC">
<property name="PhysicalDestinationName" value="QUIPSY_TOPIC" />
</admin-object-resource>
</resources>

Comment by mkarg [ 15/Jul/11 ]

Anissa: "One question. Can you reproduce the problem if you just use repeatedly using asadmin (CLI) to deploy/undeploy your application ?"

I always only use asadmin deploy and undeploy at CLI, not using any other means of deployment. So the answer is: YES.

Comment by Anissa Lam [ 18/Jul/11 ]

I thought you have been using GUI to deploy and undeploy and after couple times, console is no longer available.
So, this bug is saying, using asadmin deploy/undeploy superSimple.ear several times, you cannot access the admin console. No wonder you said GUI fix for GLASSFISH-16223 and GLASSFISH-16112 has no effect for you, since those changes the GUI code in application deployment, but you are not using GUI to do the deployment at all.

Comment by Anissa Lam [ 18/Jul/11 ]

I can't get SuperSimple.ear to deploy, even after i created the resource one by one that deploy complains.
Here is what i did:

~/Sun/v3/3.1.1/pb11/glassfish/bin 27)  ./asadmin deploy ~/DeployApp/bug16958/SuperSimple.ear

remote failure: Error occurred during deployment: Exception while preparing the app : Invalid resource : jdbc/QUIPSY/5__pm. Please see server.log for more details.
Exception while invoking class org.glassfish.persistence.jpa.JPADeployer prepare method : java.lang.RuntimeException: Invalid resource : jdbc/QUIPSY/5__pm
Invalid resource : jdbc/QUIPSY/5__pm
Command deploy failed.
~/Sun/v3/3.1.1/pb11/glassfish/bin 28)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 28)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 28)  ./asdmin create-jdbc-resource --connectionpoolid __TimerPool jdbc/QUIPSY/5__pm
-bash: ./asdmin: No such file or directory
~/Sun/v3/3.1.1/pb11/glassfish/bin 29)  ./asadmin create-jdbc-resource --connectionpoolid __TimerPool jdbc/QUIPSY/5__pm
JDBC resource jdbc/QUIPSY/5__pm created successfully.
Command create-jdbc-resource executed successfully.
~/Sun/v3/3.1.1/pb11/glassfish/bin 30)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 30)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 30)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 30)  ./asadmin deploy ~/DeployApp/bug16958/SuperSimple.ear

remote failure: Error occurred during deployment: Exception while preparing the app : Invalid resource : jdbc/QUIPSY/5__nontx. Please see server.log for more details.
Exception while invoking class org.glassfish.persistence.jpa.JPADeployer prepare method : java.lang.RuntimeException: Invalid resource : jdbc/QUIPSY/5__nontx
Invalid resource : jdbc/QUIPSY/5__nontx
Command deploy failed.
~/Sun/v3/3.1.1/pb11/glassfish/bin 31)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 31)  ./asadmin list-components
Nothing to list.
Command list-components executed successfully.
~/Sun/v3/3.1.1/pb11/glassfish/bin 32)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 32)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 32)  ./asadmin create-jdbc-resource --connectionpoolid __TimerPool jdbc/QUIPSY/5__nontx
JDBC resource jdbc/QUIPSY/5__nontx created successfully.
Command create-jdbc-resource executed successfully.
~/Sun/v3/3.1.1/pb11/glassfish/bin 33)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 33)  
~/Sun/v3/3.1.1/pb11/glassfish/bin 33)  ./asadmin deploy ~/DeployApp/bug16958/SuperSimple.ear
remote failure: Unknown plain text format.  A properly formatted response from a PlainTextActionReporter 
always starts with one of these 2 strings: PlainTextActionReporterSUCCESS or PlainTextActionReporterFAILURE.  The response we received from the server was not understood: Signature-Version: 1.0
message: Error occurred during deployment: Exception while preparing t
 he app : Exception [EclipseLink-28018] (Eclipse Persistence Services 
 - 2.3.0.v20110604-r9504): org.eclipse.persistence.exceptions.EntityMa
 nagerSetupException
Exception Description: Predeployment of P
 ersistenceUnit [QUIPSY] failed.
Internal Exception: Exception
  [EclipseLink-7333] (Eclipse Persistence Services - 2.3.0.v20110604-r
 9504): org.eclipse.persistence.exceptions.ValidationException%%%EOL%%
 %Exception Description: The reference column name [INTENTIONALLY_IMPL
 EMENTED_BUG_TO_PREVENT_DEPLOYMENT] mapped on the element [field custo
 mer] does not correspond to a valid field on the mapping reference.. 
 Please see server.log for more details.
Exception while invok
 ing class org.glassfish.persistence.jpa.JPADeployer prepare method : 
 javax.persistence.PersistenceException: Exception [EclipseLink-28018]
  (Eclipse Persistence Services - 2.3.0.v20110604-r9504): org.eclipse.
 persistence.exceptions.EntityManagerSetupException
Exception 
 Description: Predeployment of PersistenceUnit [QUIPSY] failed.%%%EOL%
 %%Internal Exception: Exception [EclipseLink-7333] (Eclipse Persisten
 ce Services - 2.3.0.v20110604-r9504): org.eclipse.persistence.excepti
 ons.ValidationException
Exception Description: The reference 
 column name [INTENTIONALLY_IMPLEMENTED_BUG_TO_PREVENT_DEPLOYMENT] map
 ped on the element [field customer] does not correspond to a valid fi
 eld on the mapping reference.
Exception [EclipseLink-28018] (
 Eclipse Persistence Services - 2.3.0.v20110604-r9504): org.eclipse.pe
 rsistence.exceptions.EntityManagerSetupException
Exception Descriptio
 n: Predeployment of PersistenceUnit [QUIPSY] failed.
Internal Excepti
 on: Exception [EclipseLink-7333] (Eclipse Persistence Services - 2.3.
 0.v20110604-r9504): org.eclipse.persistence.exceptions.ValidationExce
 ption
Exception Description: The reference column name [INTENTIONALLY
 _IMPLEMENTED_BUG_TO_PREVENT_DEPLOYMENT] mapped on the element [field 
 customer] does not correspond to a valid field on the mapping referen
 ce.
name_value: SuperSimple
name_name: name
keys: name
use-main-children-attribute: false
exit-code: FAILURE


Command deploy failed.
~/Sun/v3/3.1.1/pb11/glassfish/bin 34)  ./asadmin list-applications
Nothing to list.
Command list-applications executed successfully.
~/Sun/v3/3.1.1/pb11/glassfish/bin 35)

Comment by mkarg [ 19/Jul/11 ]

I see. The problem is the EAR that Tim gave you – it was specifically damaged to provide exactly that deployment error you are experiencing now. I hoped that multiply trying to deploy will produce the problem, which seems not to be the case actually. Sad but true, I do not have any other EAR. After I was fixing "Tim's" problem I have had some other issues (the deployment worked, but we needed to change some more stuff like broken ejb-refs etc.). THAT other EAR was the one I that happened to produce "Your" problem relatively often. "Unfortunately" we fixed all our stuff meanwhile and since then we can deploy/undeploy without producing "Your" problem anymore. So I am sorry but as it seems, "Your" problem only happens in case of an EAR that can be deployed, but has severe problems (like broken ejb-ref etc.) and gets undeployed therefore. Since I do not have such a broken EAR anymore, it seems there is nothing more I can do for you. Sad.

Comment by mkarg [ 19/Jul/11 ]

Anissa, good news: A team member is able to reproduce always using his EAR and GFv3.1! Can you please provide your email address so I can tell you how to download that closed source EAR from our FTP server (I may not upload it to a public tracker, and it is larger than 12 MB anyways

Comment by mkarg [ 20/Jul/11 ]

Found another way to reproduce a similar (but not exactly the same) problem, independent of the EAR:

asadmin start-domain
open browser to view localhost:4848
click somewhere, e. g. server-config
asadmin restart-domain
click somewhere else WHILE THE RESTART IS IN PROGRESS STILL
browser will wait until the restart is done and then show a nice glassfish-provided page telling that this JSF is not found

Comment by Anissa Lam [ 20/Jul/11 ]

I followed the EXACT steps above and cannot reproduce the problem.
Lets focus on reproducing the case as originally filed.

So far, all deployment of application failed, let alone deploy/undeploy/ cycle as mentioned in this bug report.
I have gotten a copy of SuperSimple.ear that was intended to force deployment failure. So, that deployment failed.

Then, I got another copy of SuperSimple.ear from Tim that 'suppose' can be deployed successfully. That failed also.

~/DeployApp/bug16958 9)  v3admin deploy SuperSimple.ear
remote failure: Error occurred during deployment: Exception while preparing the app : Invalid resource : jdbc/QUIPSY/5__pm. Please see server.log for more details.
Exception while invoking class org.glassfish.persistence.jpa.JPADeployer prepare method : java.lang.RuntimeException: Invalid resource : jdbc/QUIPSY/5__pm
Invalid resource : jdbc/QUIPSY/5__pm
Command deploy failed.
~/DeployApp/bug16958 10)  v3admin  create-jdbc-resource --connectionpoolid __TimerPool jdbc/QUIPSY/5__pm
JDBC resource jdbc/QUIPSY/5__pm created successfully.
Command create-jdbc-resource executed successfully.
~/DeployApp/bug16958 11)  v3admin  create-jdbc-resource --connectionpoolid __TimerPool jdbc/QUIPSY/5
JDBC resource jdbc/QUIPSY/5 created successfully.
Command create-jdbc-resource executed successfully.
~/DeployApp/bug16958 12)  v3admin deploy SuperSimple.ear
remote failure: Error occurred during deployment: Exception while loading the app : EJB Container initialization error. Please see server.log for more details.
Command deploy failed.
~/DeployApp/bug16958 13)  v3admin list-applications
Nothing to list.
Command list-applications executed successfully.
~/DeployApp/bug16958 14)

Here is what i need if you want me to continue looking into this issue:

  • Send an application that is REALLY deployable. You can send that to anilam@java.net and i will keep that private and won't share with other.
  • Provide the instructions (CLI command) on what resources I need to create first before deploying the application.
  • I will follow exactly all the CLI commands that you provide to reproduce the case.

I am going to lower this to P3 until i get a reproducible case.
thanks.

Comment by mkarg [ 21/Jul/11 ]

About the prio: I need to object since the prio should not be dependend on your local success to reproduce it but solely upong the actualy impact on the user, which is independend of your personal situation. We have two machines here that produce the behaviour and are about to ship GF 3.1.1 to 200+ enterprises world wide and expect that many of those will call our hotline to complain about this problem, which induces costs that should (and could) be prevented.

About the scenarios: No more need to send you the EAR. At one machine where it happens with our EAR we updated from 3.1 to 3.1.1_b11 and THAT scenario is gone then. So it makes no sense to further insist on getting any SuperSimple.ear that produces this problem as THAT scenario only happened back in 3.1. But, the second scenario (the one that you also cannot reproduce) happens on 3.1.1_b11 and to make you believe I will attach a video so you see what we do exactly to crash it. That machine is a HP Z400 QuadXeon with Win7 Pro SP1 64 Bit de_DE running a Firefox 3.6.18. Maybe you should use such an environment as it possible does not happen on a Mac?

Comment by mkarg [ 21/Jul/11 ]

"How To Crash Admin GUI" video, showing that a freshly installed GF can be crashed easily by just clicking in the GUI while the domain is restarting.

Comment by Anissa Lam [ 21/Jul/11 ]

>> About the scenarios: No more need to send you the EAR. At one machine where it happens with our EAR we updated from 3.1 to 3.1.1_b11 and THAT scenario is gone then. So it makes no sense to further insist on getting any SuperSimple.ear that produces this problem as THAT scenario only happened back in 3.1.
Just want to confirm this. Are you saying that you no longer experience the problem where "deploy/undeploy several times using CLI results in not getting the admin console" is no longer reproducible using 3.1.1 b11 ?
If that is the case, do u mind that I mark this "Cannot reproduce" or "Fixed" since thats what this issue is about ?

Thanks for the video, not that i don't believe you, i just can't reproduce it on my Mac.
I will open another issue for this 2nd scenario – "Cannot access admin console when server is restarting" , will that be fine with you ?
I think it makes it clearer that way.

Comment by mkarg [ 21/Jul/11 ]

While I do not see a need to close this and open another issue, I do not object that and hereby confirm that in GFv3.1.1_b11 the deploy/undeploy case is not reproducible. The only open issue now is the one shown in the video.

Comment by Anissa Lam [ 21/Jul/11 ]

Closing this as "Cannot Reproduce" in 3.1.1_b11.
I have opened GLASSFISH-17086 for the "Admin Console is not available during server re-start". mkarg, you may want to add yourself to watch that issue. That bug is set to "Critical" at your request
If you can attach server.log to GLASSFISH-17086, that will help too.
thanks.

Generated at Wed Apr 01 04:59:30 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.