[GLASSFISH-16162] Cannot start standalone instance Created: 04/Mar/11 Updated: 09/Jan/13 Resolved: 10/Apr/11
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
|Attachments:||8691 8743 server.log.started-notrunning|
I'm trying to start a standalone instance but the action hangs from both Admin Console and CLI. After trying with --verbose, with Joe's advice, here is what I see:
tuppy(j2eetest):/export/home/j2eetest/v3.1/glassfish3/glassfish# a start-local-instance --verbose in1
The exception seems to come from logging, hence I'm logging this bug under this module.
A little background: I've been using this solaris system for verifying 3.1 bugs for the past week or so. I encountered this issue yesterday. The last thing I did was undeploy a hello.war application from this instance while it was stopped (didn't actually mean to do this). When I went to the standalone instance's page and clicked on Start button. The "processing, please wait" message was displayed and never went away. I checked CLI and it reported this instance as stopped but with odd status:
tuppy(j2eetest):/export/home/j2eetest/v3.1/glassfish3/glassfish/domains/domain1/logs# a list-instances
After reloading Admin Console, the instance was reported as stopped there too.
I checked jps -v and it looked like there were two in1 processes running:
tuppy(j2eetest):/export/home/j2eetest/v3.1/glassfish3/glassfish/domains/domain1/logs# jps -v
At this point I asked Joe for help and he confirmed that indeed there were two in1 processes on the system. I'm attaching thread dump for both of them. I also checked instance's server.log and it looks like the shutdown from that day was not clean. Here is the output of the shutdown from a couple days earlier on the same instance that completed:
Feb 25, 2011 12:13:58 PM com.sun.enterprise.admin.launcher.GFLauncherLogger info
While the latest shutdown is missing the "Shutdown procedure finished" line:
Mar 3, 2011 3:51:25 PM com.sun.enterprise.admin.launcher.GFLauncherLogger info
At this point, I killed both processes and tried to start the instance again - it would still hang and the output of the verbose option is listed at the top (hence we came the full circle). I'm attaching server.log from the start of testing till the initial startup hang and thread dumps of both processes that belonged to the instance. The machine is still available for debugging.
|Comment by naman_mehta [ 07/Mar/11 ]|
Don't able to reproduce the same. Please provide me the exact steps to reproduce.
Looks like it's machine specific issue.
|Comment by naman_mehta [ 21/Mar/11 ]|
Wouldn't able to reproduce the same. Assigning this issue in your name. If still reproducible by you then give me exact steps and machine details.
|Comment by lidiam [ 22/Mar/11 ]|
It is not an issue that can be easily reproduced. I do not know what caused it in the first place, however, since the outcome was quite severe, I was asked to log an issue. You can close it if you want. If anybody encounters it in the future, we can reopen.
|Comment by lidiam [ 22/Mar/11 ]|
One more note, Joe Di Pol was helping debug this issue and this is how far we got. I kept the machine intact for a couple of days for debugging, but Glassfish has been reinstalled since. Glassfish worked flawlessly on this machine for months, so it's not necessarily a machine specific issue. After reinstall everything is working fine again. Series of many steps got Glassfish in this situation and unfortunately can't be reproduced.
|Comment by naman_mehta [ 22/Mar/11 ]|
As per the comments by lidia this issue is not reproducible.
But to avoid NPE in the code made some changes to catch the same. Committed revision 45678.
|Comment by Matthias Wimmer [ 14/Apr/11 ]|
Same problem existed here today. I found the reason for this to be some configuration properties missing in the file logging.properties in the config directory within the domain.
Probably the broken logging.properties file was produced by editing a log level setting using the web interface by one of my coleagues last night.
The broken logging.properties file produced by the web interface we had, solely contains the following content:
#GlassFish logging.properties list
|Comment by naman_mehta [ 14/Apr/11 ]|
We added code for checking that properties is present or not so NPE is avoided now.
Tried to made changes under Logger Settings page from admin GUI but couldn't reproduce broken logging.properties file. So question is how to reproduce broken logging.properties file from web interface. Let me know if it's reproducible.
|Comment by juergenschmied [ 22/Feb/12 ]|
Same problem. Rebuilding logging.properties helped.
|Comment by roinou [ 06/Apr/12 ]|
Same problem here.
It happened in our PRODUCTION environment this morning. After a routine restart, it was no longer possible to restart the server. Thanks to this issue, we reconstructed the "logging.properties" from our ACCEPTANCE and it fixed the problem.
The question is, HOW did this file get corrupted? last modification date stated 2012/03/29, today is 2012/04/06. This is a serious issue, we lost 3 hours of working time this morning.
|Comment by cobre420 [ 09/Jan/13 ]|
Same trouble today (9.1.2013) with glassfish 22.214.171.124. end of stacktrace
Replacing the logging.properties helped.. Any advice or help, how to prevent from this behavior?