[JAVASERVERFACES-2599] The deployment of several apps with --name to in2 failed: com.sun.faces.config.ConfigurationException Created: 13/Nov/12  Updated: 12/Mar/13  Resolved: 12/Mar/13

Status: Closed
Project: javaserverfaces
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Trivial
Reporter: Manfred Riem Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to GLASSFISH-18005 The deployment of several apps with -... Closed
Tags: 312_qa, 312_regression, 3_1_2-exclude

 Description   

GF 3.1.2 b14, Win 2008 machine. Was created a cluster with two instances on one machine.

Was executed the automated deployment test. During that test many apps were deployed/disabled/enabled/redeployed/undeployed/deployed with context and so on.

So when was executed the follow command:
asadmin deploy --target <cluster> --name qwerty1 --contextroot temp <app_name>

The deployment of several apps to instance2 failed. It happened, for example for bookstore and scrumtoys. The first appp, from the point of of the order of the execution, for with the problem happened was scrumptoys. Immediately before scrumptoys was deployed petstore. The petstore deployment did not fail, but all petstore deployment actions created the follow errors (only for instance2):
=======================================================================================================================================[#|2011-12-14T11:43:08.334-0800|WARNING|glassfish3.1.2|javax.enterprise.system.core.org.glassfish.kernel.event|_ThreadID=30;_ThreadName=Thread-2;|Exception while dispatching an event
java.lang.IllegalStateException: Attempting to execute an operation on a closed EntityManagerFactory.
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.verifyOpen(EntityManagerFactoryDelegate.java:305)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.close(EntityManagerFactoryDelegate.java:240)
at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.close(EntityManagerFactoryImpl.java:257)
at org.glassfish.persistence.jpa.JPADeployer.closeEMFs(JPADeployer.java:405)
at org.glassfish.persistence.jpa.JPADeployer.event(JPADeployer.java:396)
at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:128)
at com.sun.enterprise.v3.server.ApplicationLifecycle.unload(ApplicationLifecycle.java:1001)
at com.sun.enterprise.v3.server.ApplicationLifecycle.disable(ApplicationLifecycle.java:1971)
at org.glassfish.deployment.admin.DisableCommand.execute(DisableCommand.java:287)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:43:38.793-0800|INFO|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.config|_ThreadID=21;_ThreadName=Thread-2;|Initializing Mojarra 2.1.6 (SNAPSHOT 20111206) for context '/temp'|#]

[#|2011-12-14T11:43:39.442-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=21;_ThreadName=Thread-2;|WEB0671: Loading application [qwerty1] at [/temp]|#]

[#|2011-12-14T11:43:39.481-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=21;_ThreadName=Thread-2;|qwerty1 was successfully deployed in 16,817 milliseconds.|#]
================================================================================

Then for scrumptoys and several other apps, the deployment with contextroot and name failed on instance2, but passed on instance1. All other deploy actions, except redeployment (redeploy --target <cluster> --name temp <app>), passed for scrumptoys and other affected apps. For deployment with contectroot and name were seen such error messages in instance2 server.log:

======================================================================================================
[#|2011-12-14T11:46:30.126-0800|INFO|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.config|_ThreadID=23;_ThreadName=Thread-2;|Initializing Mojarra 2.1.6 (SNAPSHOT 20111206) for context '/temp'|#]

[#|2011-12-14T11:46:30.623-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.config|_ThreadID=23;_ThreadName=Thread-2;|Critical error during deployment:
com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:279)
at com.sun.faces.config.processor.RenderKitConfigProcessor.addRenderers(RenderKitConfigProcessor.java:313)
at com.sun.faces.config.processor.RenderKitConfigProcessor.process(RenderKitConfigProcessor.java:179)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ManagedBeanConfigProcessor.process(ManagedBeanConfigProcessor.java:270)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ValidatorConfigProcessor.process(ValidatorConfigProcessor.java:120)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ConverterConfigProcessor.process(ConverterConfigProcessor.java:126)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ComponentConfigProcessor.process(ComponentConfigProcessor.java:117)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ApplicationConfigProcessor.process(ApplicationConfigProcessor.java:340)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.LifecycleConfigProcessor.process(LifecycleConfigProcessor.java:116)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.FactoryConfigProcessor.process(FactoryConfigProcessor.java:222)
at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:360)
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:225)
at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:4750)
at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:550)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5366)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.javaee.blueprints.components.ui.popup.PopupRenderer.<clinit>(PopupRenderer.java:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:268)
... 59 more
Caused by: java.lang.ClassNotFoundException: org.apache.shale.remoting.XhtmlHelper
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1509)
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1359)
... 67 more

#]

[#|2011-12-14T11:46:30.627-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.StandardContext|_ThreadID=23;_ThreadName=Thread-2;|PWC1306: Startup of context /temp failed due to previous errors|#]

[#|2011-12-14T11:46:30.635-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.StandardContext|_ThreadID=23;_ThreadName=Thread-2;|PWC1305: Exception during cleanup after start failed
org.apache.catalina.LifecycleException: PWC2769: Manager has not yet been started
at org.apache.catalina.session.StandardManager.stop(StandardManager.java:873)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:5571)
at com.sun.enterprise.web.WebModule.stop(WebModule.java:527)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5384)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:30.636-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.ContainerBase|_ThreadID=23;_ThreadName=Thread-2;|ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5389)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:292)
at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:4750)
at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:550)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5366)
... 38 more
Caused by: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:279)
at com.sun.faces.config.processor.RenderKitConfigProcessor.addRenderers(RenderKitConfigProcessor.java:313)
at com.sun.faces.config.processor.RenderKitConfigProcessor.process(RenderKitConfigProcessor.java:179)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ManagedBeanConfigProcessor.process(ManagedBeanConfigProcessor.java:270)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ValidatorConfigProcessor.process(ValidatorConfigProcessor.java:120)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ConverterConfigProcessor.process(ConverterConfigProcessor.java:126)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ComponentConfigProcessor.process(ComponentConfigProcessor.java:117)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ApplicationConfigProcessor.process(ApplicationConfigProcessor.java:340)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.LifecycleConfigProcessor.process(LifecycleConfigProcessor.java:116)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.FactoryConfigProcessor.process(FactoryConfigProcessor.java:222)
at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:360)
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:225)
... 41 more
Caused by: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.javaee.blueprints.components.ui.popup.PopupRenderer.<clinit>(PopupRenderer.java:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:268)
... 59 more
Caused by: java.lang.ClassNotFoundException: org.apache.shale.remoting.XhtmlHelper
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1509)
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1359)
... 67 more

#]

[#|2011-12-14T11:46:30.641-0800|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=23;_ThreadName=Thread-2;|java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:921)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:30.657-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=23;_ThreadName=Thread-2;|Exception while invoking class com.sun.enterprise.web.WebApplication start method
java.lang.Exception: java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:138)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:30.660-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=23;_ThreadName=Thread-2;|Exception while loading the app|#]

[#|2011-12-14T11:46:35.918-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=23;_ThreadName=Thread-2;|Exception while loading the app : java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper|#]

[#|2011-12-14T11:46:53.944-0800|INFO|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.config|_ThreadID=21;_ThreadName=Thread-2;|Initializing Mojarra 2.1.6 (SNAPSHOT 20111206) for context '/scrumtoys'|#]

[#|2011-12-14T11:46:54.252-0800|SEVERE|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.config|_ThreadID=21;_ThreadName=Thread-2;|Critical error during deployment:
com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:279)
at com.sun.faces.config.processor.RenderKitConfigProcessor.addRenderers(RenderKitConfigProcessor.java:313)
at com.sun.faces.config.processor.RenderKitConfigProcessor.process(RenderKitConfigProcessor.java:179)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ManagedBeanConfigProcessor.process(ManagedBeanConfigProcessor.java:270)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ValidatorConfigProcessor.process(ValidatorConfigProcessor.java:120)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ConverterConfigProcessor.process(ConverterConfigProcessor.java:126)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ComponentConfigProcessor.process(ComponentConfigProcessor.java:117)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ApplicationConfigProcessor.process(ApplicationConfigProcessor.java:340)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.LifecycleConfigProcessor.process(LifecycleConfigProcessor.java:116)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.FactoryConfigProcessor.process(FactoryConfigProcessor.java:222)
at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:360)
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:225)
at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:4750)
at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:550)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5366)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.javaee.blueprints.components.ui.popup.PopupRenderer.<clinit>(PopupRenderer.java:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:268)
... 59 more
Caused by: java.lang.ClassNotFoundException: org.apache.shale.remoting.XhtmlHelper
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1509)
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1359)
... 67 more

#]

[#|2011-12-14T11:46:54.254-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.StandardContext|_ThreadID=21;_ThreadName=Thread-2;|PWC1306: Startup of context /scrumtoys failed due to previous errors|#]

[#|2011-12-14T11:46:54.257-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.StandardContext|_ThreadID=21;_ThreadName=Thread-2;|PWC1305: Exception during cleanup after start failed
org.apache.catalina.LifecycleException: PWC2769: Manager has not yet been started
at org.apache.catalina.session.StandardManager.stop(StandardManager.java:873)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:5571)
at com.sun.enterprise.web.WebModule.stop(WebModule.java:527)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5384)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:54.258-0800|SEVERE|glassfish3.1.2|org.apache.catalina.core.ContainerBase|_ThreadID=21;_ThreadName=Thread-2;|ContainerBase.addChild: start:
org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5389)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:292)
at org.apache.catalina.core.StandardContext.contextListenerStart(StandardContext.java:4750)
at com.sun.enterprise.web.WebModule.contextListenerStart(WebModule.java:550)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5366)
... 38 more
Caused by: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:279)
at com.sun.faces.config.processor.RenderKitConfigProcessor.addRenderers(RenderKitConfigProcessor.java:313)
at com.sun.faces.config.processor.RenderKitConfigProcessor.process(RenderKitConfigProcessor.java:179)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ManagedBeanConfigProcessor.process(ManagedBeanConfigProcessor.java:270)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ValidatorConfigProcessor.process(ValidatorConfigProcessor.java:120)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ConverterConfigProcessor.process(ConverterConfigProcessor.java:126)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ComponentConfigProcessor.process(ComponentConfigProcessor.java:117)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.ApplicationConfigProcessor.process(ApplicationConfigProcessor.java:340)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.LifecycleConfigProcessor.process(LifecycleConfigProcessor.java:116)
at com.sun.faces.config.processor.AbstractConfigProcessor.invokeNext(AbstractConfigProcessor.java:114)
at com.sun.faces.config.processor.FactoryConfigProcessor.process(FactoryConfigProcessor.java:222)
at com.sun.faces.config.ConfigManager.initialize(ConfigManager.java:360)
at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:225)
... 41 more
Caused by: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.javaee.blueprints.components.ui.popup.PopupRenderer.<clinit>(PopupRenderer.java:61)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at com.sun.faces.config.processor.AbstractConfigProcessor.createInstance(AbstractConfigProcessor.java:268)
... 59 more
Caused by: java.lang.ClassNotFoundException: org.apache.shale.remoting.XhtmlHelper
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1509)
at org.glassfish.web.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1359)
... 67 more

#]

[#|2011-12-14T11:46:54.260-0800|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=21;_ThreadName=Thread-2;|java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:921)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:733)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:54.261-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=21;_ThreadName=Thread-2;|Exception while invoking class com.sun.enterprise.web.WebApplication start method
java.lang.Exception: java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:138)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:462)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-12-14T11:46:54.262-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=21;_ThreadName=Thread-2;|Exception while loading the app|#]

[#|2011-12-14T11:46:59.887-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=21;_ThreadName=Thread-2;|Exception while loading the app : java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: java.lang.RuntimeException: com.sun.faces.config.ConfigurationException:
Source Document: jar:file:/C:/hudson/workspace/deployment-w/glassfish3/glassfish/nodes/localhost-domain1/my-in2/applications/qwerty1/WEB-INF/lib/bp-ui-5.jar!/META-INF/faces-config.xml
Cause: Class 'com.sun.javaee.blueprints.components.ui.popup.PopupRenderer' is missing a runtime dependency: java.lang.NoClassDefFoundError: org/apache/shale/remoting/XhtmlHelper|#]
========================================================================================================

I did not see this issue for Solaris and Linux executions. When I've executed the failed command alone, it did not fail.



 Comments   
Comment by Manfred Riem [ 11/Dec/12 ]

Asked for an update at the original issue to make sure we don't lose track

Comment by Manfred Riem [ 11/Jan/13 ]

Lowering priority because of no response (see original issue)

Comment by Manfred Riem [ 12/Feb/13 ]

Lowering priority because of no response (also see original issue)

Comment by Manfred Riem [ 12/Mar/13 ]

Closing because of inactivity





[GLASSFISH-18343] Win XP, the deployment of mdb_simple to second instance - failed: Auto-creation of destination MyQueue is not allowed because service jms is currently in restricted service mode Created: 09/Feb/12  Updated: 16/Feb/13  Resolved: 16/Feb/13

Status: Closed
Project: glassfish
Component/s: jms
Affects Version/s: 3.1.2_b21
Fix Version/s: 3.1.2

Type: Bug Priority: Major
Reporter: easarina Assignee: David Zhao
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File logs.tar     File mdb-simple.ear    
Tags: 312_qa, 312_regression, 3_1_2-exclude

 Description   

GF 3.1.2 build 31, JDK 1.6.0_31 (b 05). Created on Win XP machine a cluster with two instances. Tried to deploy mdb-simple app, the deployment of this up to the second instance - failed. I saw in server.log of this instance such error messages:

====================================================================================================================

[#|2012-02-08T13:23:48.428-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=22;_ThreadName=Thread-2;|[I500]: Caught JVM Exception: com.sun.messaging.jms.JMSException: [CREATE_DESTINATION_REPLY(35)] [C4036]: A broker error occurred. :[503] [B4286]: [Thread-jms[0]]Auto-creation of destination MyQueue is not allowed because service jms is currently in restricted service mode: Persistent store has not been synchronized with master broker [broker2(mq://10.133.169.190:18686/)] user=guest, broker=localhost:38686(3748)|#]

[#|2012-02-08T13:23:48.459-0800|WARNING|glassfish3.1.2|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|_ThreadID=22;_ThreadName=Thread-2;|RAR8501: Exception during endpoint activation for ra [ jmsra ], activationSpecClass [ com.sun.messaging.jms.ra.ActivationSpec ] : javax.resource.NotSupportedException: MQRA:EC:Error creating Remote Message Consumer:
[CREATE_DESTINATION_REPLY(35)] [C4036]: A broker error occurred. :[503] [B4286]: [Thread-jms[0]]Auto-creation of destination MyQueue is not allowed because service jms is currently in restricted service mode: Persistent store has not been synchronized with master broker [broker2(mq://10.133.169.190:18686/)] user=guest, broker=localhost:38686(3748)|#]

[#|2012-02-08T13:23:48.459-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.container.ejb.mdb.com.sun.ejb.containers|_ThreadID=22;_ThreadName=Thread-2;|MDB00017: [SimpleMessageEJB]: Exception in creating message-driven bean container: [java.lang.Exception]|#]

[#|2012-02-08T13:23:48.459-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.container.ejb.mdb.com.sun.ejb.containers|_ThreadID=22;_ThreadName=Thread-2;|java.lang.Exception
java.lang.Exception
at com.sun.enterprise.connectors.inbound.ConnectorMessageBeanClient.setup(ConnectorMessageBeanClient.java:233)
at com.sun.ejb.containers.MessageBeanContainer.<init>(MessageBeanContainer.java:205)
at com.sun.ejb.containers.ContainerFactoryImpl.createContainer(ContainerFactoryImpl.java:121)
at org.glassfish.ejb.startup.EjbApplication.loadContainers(EjbApplication.java:230)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:299)
at org.glassfish.ejb.startup.EjbDeployer.load(EjbDeployer.java:105)
at org.glassfish.internal.data.ModuleInfo.load(ModuleInfo.java:186)
at org.glassfish.internal.data.ApplicationInfo.load(ApplicationInfo.java:264)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:460)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1259)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

=============================================================================

I've attached all server.log file and mq log files.

I did not see this error, when I've executed the same test against b17 on Win 2008 machine with JDK 1.7.0_02. Also I did not see this error when executed this test on RHL with JDK 1.7.0_03 against b20.

I've attached the sample. This sample required such resources:

$S1AS_HOME/bin/asadmin --user admin --passwordfile $OUT_DIR/password312.txt create-jms-resource --target $CLUSTER --restype
javax.jms.Queue --property Name=MyQueue jms/MyQueue
$S1AS_HOME/bin/asadmin --user admin --passwordfile $OUT_DIR/password312.txt create-jms-resource --target $CLUSTER --restype
javax.jms.QueueConnectionFactory jms/MyMDBQcf



 Comments   
Comment by David Zhao [ 09/Feb/12 ]

easarina:

The MQ JMX service can not be initialized, which is the root cause - apparently a SOCKS error related to the network configuration.

java.io.IOException: Cannot bind to URL [rmi://jws-v60x-4.us.oracle.com:18786/jws-v60x-4.us.oracle.com/18686/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: jws-v60x-4.us.oracle.com; nested exception is:
java.net.SocketException: Reply from SOCKS server has bad version]

So can you check if you set system property of socksProxyHost in either cluster-config's JVM settings or programmaticly? Also did you setup SSH on the win boxes? If so, please retry it with disabling SSH on the servers for the two cluster instances are on the same box in your case and SSH is not necessary at the scenario.

Comment by easarina [ 09/Feb/12 ]

As I've mentioned at the description: "Created on Win XP machine a cluster with two instances". I.e. was used one machine, both instances and DAS were on one machine. So SSH was not used at all. You can login to jws-v60x-4 machine, using remote desktop connection: id=hudson, password=hudson.

Comment by easarina [ 09/Feb/12 ]

Forgot to mention, that was used a default domain1 and I did not setup any JVM options. So were used default JVM setting.

Comment by David Zhao [ 10/Feb/12 ]

Amy,

In the imq log file, now I see the following exception on jws-v60x-4 machine:

[09/Feb/2012:23:44:22 PST] WARNING Exception caught while starting JMX Connector Server jmxrmi:
java.io.IOException: Cannot bind to URL [rmi://jws-v60x-4.us.oracle.com:18786/jws-v60x-4.us.oracle.com/18686/jmxrmi]: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: jws-v60x-4.us.oracle.com; nested exception is:
java.net.SocketException: Software caused connection abort: socket write error]

Could you take a look from the MQ perspective?

Comment by amyk [ 10/Feb/12 ]

same comment as my previous one, and

"java.net.SocketException: Software caused connection abort: socket write error"

This is same SocketException(s) shown in server.log.

Comment by easarina [ 10/Feb/12 ]

I've re-run the test on another Win XP machine (was used the same build 21 and the same jdk 1.6.0_31) and this test passed. I don't know what is the difference between these two machines. Before, on jws-v60x-4, I've executed the test three times, and the deployment of mdb-simple always failed with that error message.

Comment by Joe Di Pol [ 14/Feb/12 ]

3.1.2 is pretty much down. Deferring this bug to next release while investigation continures.

Comment by David Zhao [ 14/Feb/12 ]

easarina,

Can you try uninstalling firewall software McAfee from jws-v60x-4?

Comment by easarina [ 15/Feb/12 ]

I've changed firewall config on jws-v60x-3, to make it the same like on jws-v60x-4. Then run the test on jws-v60x-3 and it passed again on that machine. So I still don't know, why the test is failing on jws-v60x-4.

Elena

Comment by David Zhao [ 16/Feb/13 ]

Close it for it can not be reproduced with a new win box per easarina's comments.

easarina,

Please feel free to reopen it when you see it again.





[GLASSFISH-18336] Error starting remote, SSH instance after changing all log levels Created: 08/Feb/12  Updated: 15/Feb/13

Status: Open
Project: glassfish
Component/s: logging
Affects Version/s: 3.1.2_b21
Fix Version/s: future release

Type: Bug Priority: Minor
Reporter: lidiam Assignee: naman_mehta
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

DAS on OEL 6 with JDK 7u3, b04
remote instance on Solaris 10 with JDK 7u3, b04
Firefox on Winxp


Attachments: Java Archive File console-common.jar     JPEG File error-starting-instance.JPG     JPEG File logging-error-saving-changes.JPG     Text File logging.properties.das.txt     Text File logging.properties.instance.txt     Text File LoggingHandlers.diff.text     Text File server.log.badfileexception.txt     Text File server.log.das.txt     Text File server.log.instance.txt    
Tags: 312_qa, 312_regression

 Description   

After changing all log levels for a remote instance, on SSH node, to WARNING, instance fails to start with the following error in Admin Console:

An error has occurred
Could not start instance intp on node tuppy (tuppy). Command failed on node tuppy (tuppy): CLI801 Instance is already synchronized Waiting for intp to start ..........Command start-local-instance failed. Error starting instance intp. The server exited prematurely with exit code 1. Before it died, it produced the following output: Launching GlassFish on Felix platform Completed shutdown of GlassFish runtime Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.lang.NullPointerException at com.sun.enterprise.server. .... msg.seeServerLog

Attaching server.log files.

I have tried it several times with b20 and b21 and it always happens in my setup with a remote instance on an SSH node. I does not happen when instance is on a DCOM node or on localhost. Changing just one or two log levels for an instance on SSH node does NOT trigger this error. However, I'm still logging this as a Major issue, so we at least investigate it, since this is not an unlikely scenario, to set all log levels to warning on a production system.

Steps to reproduce:

1. Create a remote SSH node.
2. Create a standalone instance under the new node and start it.
3. In Admin Console go to instance's configuration, Logger Settings, Log Levels.
4. Select checkboxes for all loggers, WARNING for Log Level drop down and click on Change Level. Then click on Save.
5. Go back to instance page and stop it.
6. Start the instance and the above mentioned error is displayed.



 Comments   
Comment by lidiam [ 08/Feb/12 ]

Once I change log levels to INFO for all loggers, and set com.sun.enterprise.server.logging.GFFileHandler level to ALL, instance starts fine once again. However, if I only change all log levels to INFO, instance still fails to start with the same exception. Please note that there is no "Load Default" button on this page, so user will have hard time finding out what were the default settings (it can be seen in default-conifg).

Comment by naman_mehta [ 08/Feb/12 ]

As per the error looks like logging.properties files are missing some entries on SSH node.

Can you send me logging.properties files from ssh machine?
Have you changed log levels through admin console?
Can you share same machine with me so I can check the same?

Comment by lidiam [ 08/Feb/12 ]

Attaching logging.properties.instance.remote.txt file - this is logging.properties file on the remote SSH node, under <glassfish home>/glassfish/nodes/tuppy/intp/config/intp-config directory.

Comment by lidiam [ 08/Feb/12 ]

I performed all operations from Admin Console, as described in steps to reproduce section. I attached logging.properties for the instance on the remote SSH node - the file is the same as on the DAS system, under domain1/config/intp-config (diff shows no difference). I also sent you an email with the machine information.

Comment by naman_mehta [ 08/Feb/12 ]

logging.properties file is correct...

Have you changed log levels through admin console? Can you share same machine with me so I can check the same? Is it reproducible/occasionally?

Comment by lidiam [ 08/Feb/12 ]

I have changed the log levels back to info, in order to see if instance would start then. I already wrote that all operations were performed through Admin Console, including setting the log levels. Please see the section above that starts with "Steps to reproduce". It is reproducible each time all log levels are set to WARNING.

Comment by naman_mehta [ 08/Feb/12 ]

I tried on my local machine with use of ssh node on another machine on linux but couldn't able to reproduce the same. I tried several times but no success.

Then, I tried on same machine and same setup having this issue filed but after several runs it couldn't be reproducible.

Comment by naman_mehta [ 08/Feb/12 ]

Lidia,

Please try yourself again on same machine by creating new node and new instance. If it's reproducible for you then stop using this machine. Let me know, I will debug same on this machine only by doing remote debug.

Comment by naman_mehta [ 08/Feb/12 ]

I would able to identify the issue: Sometimes logging.properties file is missing entries

Initially logging.properties file contains:
"logging.properties" 69 lines, 3756 characters

now after updating log levels from admin console
"logging.properties" 46 lines, 2285 characters

So missing some properties which is causing failure....

Comment by naman_mehta [ 08/Feb/12 ]

When you change log levels through admin console it's going to update logging.proeprties file on DAS and then trying to replicate on the instance machine. But sometimes we are stopping instance before replication is successful and which corrupts the logging.properties file on remote machine. If you check logging.properties file on DAS it's up to date and corrupted on remote machine.

Now, next time when you trying to start instance it's looking for some properties on startup and which are missing there and so it's not coming up.

Solution is to to avoid catastrophic failure in the absence of a user defined configuration property like com.sun.enterprise.server.logging.GFFileHandler.file.

Comment by lidiam [ 08/Feb/12 ]

I tried the following on newly created instance:

1. Changed all log levels to WARNING, waited 25 seconds after Admin Console reported logging changes saved.
Result: instance started fine.

2. Changed all log levels to INFO, waited 20 seconds after Admin Console reported logging changes saved.
Result: instance started fine.

3. Chagned all log levels to SEVERE - could not save the change with the following error in Admin Console:

An error has occurred
An error occurred during replication Failure: Command set-log-levels failed on server instance inbif: remote failure: Could not set logger levels for inbif-config. Failure: Command set-log-levels failed on server instance inbif: remote failure: Could not set logger levels for inbif-config.

The following exception was in the instance's server.log:

[#|2012-02-08T15:05:16.277-0800|SEVERE|oracle-glassfish3.1.2|null|ThreadID=19;
ThreadName=Thread-2;|Cannot read logging.properties file :
java.io.IOException: Bad file number
at java.io.FileInputStream.readBytes(Native Method)

I'm attaching instance's server.log as server.log.badfileexception.txt.

Comment by lidiam [ 09/Feb/12 ]

It looks like stopping of the instance is NOT the cause of bad logging.properties file being written to the remote machine. This looks like an intermittent problem with writing full logging.properties file to the remote machine. Once the "bad" file is written, I can wait for a minute before restarting server and it is not getting corrected. Here is what I tested:

1. Change all log levels and save.
2. Wait 30 seconds after save.
3. Compare size of logging.properties files on DAS and remote system.
4. Restart instance.

While executing the above, logging.properties file got corrupted on the 4th attempt - size differed between DAS and remote and instance failed to start. I tested further and again hit the issue on the 6th attempt (or 10th, 4 + 6 new attempts). The trouble is that once the instance cannot be started I can only fix it by manually copying logging.properties file from DAS to the instance. Other changes, also to logging properties, don't seem to trigger copying the file over.

Comment by lidiam [ 09/Feb/12 ]

I'm attaching logging.properties files from the time of failure:

t# ls -l logging.p*
rw-rr- 1 j2eetest green 3659 Feb 8 15:36 logging.properties.das
rw-rr- 1 j2eetest green 1278 Feb 8 15:36 logging.properties.instance

Comment by lidiam [ 09/Feb/12 ]

Example of error in Admin Console when logging changes cannot be saved due to corrupted logging.properties file.

Comment by lidiam [ 09/Feb/12 ]

Attaching logging.properties files with txt extension for easier viewing.

Comment by Joe Di Pol [ 09/Feb/12 ]

Workaround is to manually copy the logging.properties from the DAS to the instance that can't start.

Comment by naman_mehta [ 09/Feb/12 ]

set-log-level command has capability to update multiple log levels simultaneously.

Usage of command:
./asadmin set-log-levels --target in5-config javax.enterprise.system.container.cmp=INFO:javax.enterprise.resource.webcontainer.jsf.application=INFO:javax.enterprise.system.ssl.security=INFO

Here, all logger name are : separated.

In this scenario, logging back end code will open logging.properties files once update all log levels as passed and closing logging.properties file. So here all log levels are updated in single file operation.

--------------------------------------------------------------------------------------------------------

In current scenario, when user updates all log level through admin console by selecting all logger names, it passes single logger name and log level to back end code then next time passes another logger name and log level. So if glassfish has 46 loggers it has 46 back end calls and due to that file operation is also becomes multiple of 46 and which causes file related exception in back end code.

Instead of this admin gui can consolidate all log levels with : separated as above and pass to back end code which cause single file operation and it avoids file related exceptions.

Comment by naman_mehta [ 09/Feb/12 ]

One more thing I monitored,

If user changes single log level for admin console then also it passes all logger names and levels to the back end code.

So for single log level updates from GUI also making 46 back end calls to set-log-level command.

Comment by Rebecca Parks [ 09/Feb/12 ]

Added to the 3.1.2 Release Notes:

Description

Changing all log levels on a remote SSH server instance before stopping the instance can cause the instance to fail to restart.

Workaround

Manually copy the logging.properties file from the domain administration server (DAS) to the server instance that won’t restart.

Comment by lidiam [ 09/Feb/12 ]

Now that this issue is understood fully I'm upgrading it to P2, after talking to Sudipa. Since logging.properties file is relatively likely to get corrupted, we should fix this issue for 3.1.2 if another show stopper is found for this release. I'm also reassigning to Admin Console, to evaluate the possible fix proposed by Naman.

Comment by Anissa Lam [ 09/Feb/12 ]

Agree that GUI shouldn't call changing of one log level at a time. Thats bad for performance too. I have fixed that in the trunk for 4.0

Log Message:
------------
Improve performance of GUI when changing log levels and to work around a bug in logging, GLASSFISH-18336
Revisions:
----------
52523
Modified Paths:
---------------
trunk/main/appserver/admingui/common/src/main/java/org/glassfish/admingui/common/handlers/LoggingHandlers.java

I am attaching the diff for 3.1.2 in case management believe this needs to be in 3.1.2 too.

Comment by Anissa Lam [ 09/Feb/12 ]

Previous comment from Naman says:
"When you change log levels through admin console it's going to update logging.proeprties file on DAS and then trying to replicate on the instance machine. But sometimes we are stopping instance before replication is successful and which corrupts the logging.properties file on remote machine. "

Console code is using the correct API but of course should be more efficient, should call set-log-levels once instead of in a loop. I fixed this in the trunk now. I feel that the console change is just masking the root problem.
Besides triggered by console, will there be other situation that the replication failed affecting logging.properties ?

I am changing this back to logging for Naman to put in the proper fix, maybe by catching the NPE.
Caused by: java.lang.NullPointerException
at com.sun.enterprise.server.logging.GFFileHandler.postConstruct(GFFileHandler.java:162)

Comment by Anissa Lam [ 10/Feb/12 ]

Attached console-common.jar which has the changes in console for 3.1.2.

To test this:
install OGS promoted build 21.
cd <glassfish-install>/glassfish/modules
mv console-common.jar console-common.jar.ORIG
cp <Download>/console-common.jar .

rm -rf <glassfish-install>/glassfish/domains/domain1/osgi-cache

restart domain.

Comment by lidiam [ 10/Feb/12 ]

Tested patch on existing build. It works like a charm. Tried the scenario 10 times and did not see the issue. Also, now log levels update takes less than a second (before it was taking around 12 seconds). I didn't wait at all between logging levels changes and instance restarts and it all worked.

Comment by naman_mehta [ 10/Feb/12 ]

This issue is coming due to replication error or file is not properly updated on remote machine and instance is not able to coming up as some properties are missing. To avoid this we can make following changes,

Index: src/main/java/com/sun/enterprise/server/logging/GFFileHandler.java
===================================================================
— src/main/java/com/sun/enterprise/server/logging/GFFileHandler.java (revision 52453)
+++ src/main/java/com/sun/enterprise/server/logging/GFFileHandler.java (working copy)
@@ -154,13 +154,25 @@
String recordFieldSeparator;
String recordDateFormat;

+ String logFileProperty = "";
+
public void postConstruct() {

LogManager manager = LogManager.getLogManager();
String cname = getClass().getName();

  • String filename = TranslatedConfigView.getTranslatedValue(manager.getProperty(cname + ".file")).toString();
    + if (manager != null) { + logFileProperty = manager.getProperty(cname + ".file"); + }

+ if (logFileProperty == null || logFileProperty.trim().equals(""))

{ + logFileProperty = env.getInstanceRoot().getAbsolutePath() + File.separator + LOGS_DIR + File.separator + + logFileName; + }

+
+ String filename = TranslatedConfigView.getTranslatedValue(logFileProperty).toString();
+
+
File serverLog = new File(filename);
absoluteServerLogName = filename;
if (!serverLog.isAbsolute()) {

Above changes will check for the required property is present or not if not it would create log files under installation directory. And now server will come up and later the new logging.properties file is replicated to the remote machine and which could solve the problem.

This fix is to avoid to avoid catastrophic failure in the absence of a user defined configuration property like com.sun.enterprise.server.logging.GFFileHandler.file. It's just to avoid instance startup error.

Comment by lidiam [ 10/Feb/12 ]

It seems to me that with the above solution we will totally hide the issue, i.e. we will never know if logging.properties file got truncated/corrupted. I think we should at least print a WARNING level message to server.log indicating that there is a problem.

Comment by Anissa Lam [ 11/Feb/12 ]

The changes in GUI to call set-log-levels once for all loglevels has been checked in to 3.1.2 branch with Joe's request.
This should be in 3.1.2 RC4 build.

Log Message:
------------
Improve performance of GUI when changing log levels and to avoid a bug in logging, GLASSFISH-18336.
Revisions:
----------
52545
Modified Paths:
---------------
branches/3.1.2/admingui/common/src/main/java/org/glassfish/admingui/common/handlers/LoggingHandlers.java

Comment by Rebecca Parks [ 14/Feb/12 ]

Per Sudipa, removed from Release Notes due to the fix.

Comment by Joe Di Pol [ 14/Feb/12 ]

A work-around was implemented in the console, so this bug is not hitting users anymore. But there is still an NPE in the backend that should be fixed. I'm lowering the priority to P4 since there is limited customer impact.

Comment by naman_mehta [ 20/Feb/12 ]

Added backend changes for 4.0.





[GLASSFISH-18318] Install silent option does not store admin password input Created: 03/Feb/12  Updated: 19/Sep/14

Status: Open
Project: glassfish
Component/s: installation
Affects Version/s: 3.1.2_b20
Fix Version/s: 4.1

Type: Bug Priority: Major
Reporter: Alex Pineda Assignee: Snjezana Sevo-Zenzerovic
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL6 system. GF 3.1.2 build20. JDK1.7.0_03. Firefox browser 3.6.17. Typical Install option


Tags: 312_qa, 3_1_2-exclude, 3_1_2-release-note-added, 3_1_2-release-notes

 Description   

With the addition of the Domain Info screen (due to security remediation), the user is given the ability to add an "admin password" that is later used to login to the AdminConsole. The GF Installer provides and option to create a "silent" file that records all the user choices and is only supported in the "Typical" scenario. This "silent" file can later be used to do installation without user interaction.

The issue uncovered is that the option given in the Domain Info (to enter a user admin password) is not being recorded or stored in the "silent" file. The procedure to create the file is as follows:
1. Get the latest build (ogs-3.1.2-b20-unix.sh*)
2. Generate the silent file

  • machine $ ogs-3.1.2-b20-unix.sh -n sfile.txt (file in which all the Install actions are recorded)
    3. Run through the Install Typical option and enter a password in the Domain Info screen (admin123)
    4. Complete the installation steps.
    5. Execute the installation using the "silent" file as follows:
  • machine $ ogs-3.1.2-b20-unix.sh -a sfile.txt -s
    6. After the installation completes, start the domain server
    7. Go to the Admin Console (http://localhost:4848)

You will notice one will be logged into to the Admin Console without any password.

The expected behavior and when executing the same scenario interactively, the AdminConsole login screen is displayed and one has to enter the admin user and admin password.

Reporting this bug as low priority because it's a bit late and perhaps risky to fix. Documenting this issue is sufficient at this time.



 Comments   
Comment by scatari [ 03/Feb/12 ]

Enabling silent installer to recognize passwords is an enhancement requiring extensive changes. Marking this as Release notes item to be documented. Here is what should be documented as a limitation.

"The generated silent file will not contain any passwords and if such files are used for running automated silent installation, then the created GlassFish domain will provide unauthenticated login mechanism".

Comment by Rebecca Parks [ 07/Feb/12 ]

Added to 3.1.2 Release Notes:

Description

The GlassFish Server installer provides an option to create a silent file that records all user choices and is only supported in the Typical scenario. This silent file can later be used to perform installation without user interaction.

The generated silent file does not contain any passwords. If this file is used for running automated silent installation, the created GlassFish Server domain provides an unauthenticated login mechanism.

Workaround

Use interactive installation if you want the GlassFish Server domain to require passwords.

Comment by shreedhar_ganapathy [ 19/Mar/13 ]

-> Tom Mueller to eval if this will be fixed in 4.0

Comment by Tom Mueller [ 19/Mar/13 ]

I confirmed that this problem is there in the OSE installer as well as the OGS installer, however, the only way to get a password prompt via the OSE installer is to use the Custom path, not the Typical path.

For 4.0 OSE, the Custom path through the installer is going to be disabled (see GLASSFISH-19680) so there will be no opportunity to enter a password when using the 4.0 OSE installer, so this bug does not need to be fixed for 4.0.





[GLASSFISH-18289] Restart of DAS produces NullPointerException in server.log after logging changes Created: 01/Feb/12  Updated: 01/Feb/12  Resolved: 01/Feb/12

Status: Resolved
Project: glassfish
Component/s: configuration
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Tom Mueller
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip, DAS on OEL 6 with JRockit 1.6.0_29


Attachments: Text File server.log.txt    
Tags: 312_qa, 312_regression

 Description   

After making changes to server-config, Logger Settings, the following error message is written to server.log with each DAS restart:

[#|2012-01-30T14:01:28.271-0800|SEVERE|glassfish3.1.2|null|_ThreadID=48;_ThreadName=Thread-4;|Config Listener notification got interrupted
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1.call(Transactions.java:290)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1.call(Transactions.java:269)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:139)
at org.jvnet.hk2.config.Transactions$Notifier$1$1.run(Transactions.java:169)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:139)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

This exception is printed regardless if restart is done from Admin Console or CLI, however, it seems to be triggered from Admin Console. Steps to reproduce:

1. Go to server-config, Logger Settings and modify File Rotation Time Limit, to e.g. 30.
2. Go to server and click Restart button. Note exception printed in server.log.
3. Consequent restarts of DAS produce the same exception.

This is not happening if logging changes are done from command line:

1. Once the exception is no longer printed in server.log use CLI to modify File Rotation Time Limit as follows:

in1/config# asadmin set-log-attributes --target server com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes=20
Enter admin password for user "admin">
com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes logging attribute set with value 20.
These logging attributes are set for server.
Command set-log-attributes executed successfully.

2. Restart DAS from command line and note that the exception is NOT printed to server.log.

3. Go to Admin Console, verify logging settings and Restart server from there. The exception is again printed to server.log.

By the way, the exception should not be printed by null logger.



 Comments   
Comment by Anissa Lam [ 01/Feb/12 ]

I followed the direction and I cannot reproduce on my Mac and on Solaris x86. I have also tried with secure-admin enabled and disabled.
Can you consistently reproduce this on this machine ? If so, can you try to see if you can reproduce this on another machine ?

Console called the REST API ":4848management/domain/restart" to perform the restart. Also since the NPE is not from console, I am transferring the bug to configuration for evaluation. I believe thats the component for hk2. If not, please reassign.

Comment by Tom Mueller [ 01/Feb/12 ]

I cannot reproduce this either using the latest build (b20). Please try again with b20 and reopen the bug if you still see the issue.

This problem may be related to issue GLASSFISH-18233 which has been fixed on the trunk but not in 3.1.2.





[GLASSFISH-18287] false error message during an instance creation (the remote DCOM node was created with --nodedir --installdir) Created: 01/Feb/12  Updated: 03/Feb/12  Resolved: 03/Feb/12

Status: Resolved
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Major
Reporter: easarina Assignee: Byron Nevins
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive fix.zip    
Tags: 312_qa, 312_regression, 3_1_2-approved

 Description   

Build 19 Win 2008.

Executed the follow commands:

asadmin --user admin --passwordfile passw
ord.txt create-node-dcom --nodehost bigapp-oblade-2 --installdir C:\export\glass
fish3 --nodedir C:\export\glassfish3\glassfish\nodes temp111
Command create-node-dcom executed successfully.

asadmin --user admin --passwordfile password.txt create-instance --node temp111 in111
Successfully created instance in111 in the DAS configuration, but failed to crea
te the instance files on node temp111 (bigapp-oblade-2).

Command failed on node temp111 (bigapp-oblade-2): Command _create-instance-files
ystem executed successfully.

To complete this operation run the following command locally on host bigapp-obla
de-2 from the GlassFish install location C:\export\glassfish3:

bin/asadmin --host bigapp-oblade-1.us.oracle.com --port 4848 create-local-insta
nce --nodedir C:\export\glassfish3\glassfish\nodes --node temp111 in111
Command create-instance completed with warnings.

=========================================

So it looks like the instance was not created, but really it was. In the cli log I saw such commands:

=========================================================================
01/31/2012 17:02:39 EXIT: 0 asadmin --user admin --passwordfile password.txt create-node-dcom --nodehost bigapp-oblade-2 --installdir C:\export\glassfish3 --nodedir C:\export\glassfish3\glassfish\nodes temp111
01/31/2012 17:03:53 EXIT: 0 asadmin --user admin --passwordfile password.txt create-instance --node temp111 in111
=======================================================

I was able to start this instance and saw the instance files on the remote host bigapp-oblade-2.

So this warning message is wrong and also the instance ports were not published.

When the correspondent node was created without --nodedir and --installdir, this false warning message was not seen. I did not see this false error message, when executed the same test against b14. And did not see this error when I've run the test in ssh mode against b17.



 Comments   
Comment by Byron Nevins [ 01/Feb/12 ]

Nice job finding this.

There is a work-around because it is IMPOSSIBLE to get the return value (integer) from running remote DCOM commands.

Without looking at the code I'm confident that the work-around, which looks for the right files to be created, is not using nodedir.

Should be a straight-forward fix.

idea: maybe it's time to get create-local-instance to write something unique and special to stderr for determining return value?

Comment by Byron Nevins [ 02/Feb/12 ]

The problem is in the code below from Paths.java

getNodeDirAbsoluteUnixStyle() returns the PARENT DIRECTORY of the node dir!!!

===========================================
public final static String getNodeDir(final Node node)

{ if (node == null) // don't do that! throw new NullPointerException(); // forward slashes are fine for ALL supported platforms // JIRA 18287 // note that getNodeDirAbsoluteUnixStyle() returns the PARENT of the node dir!! String nodeDir = node.getNodeDirAbsoluteUnixStyle(); if (nodeDir == null) nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes/" + node.getName(); return nodeDir; }

==========================

The fix is easy:

BEFORE:
if (nodeDir == null)
nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes/" + node.getName();

AFTER:
if (nodeDir == null)
nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes";

nodeDir += node.getName();

====================================

Comment by Byron Nevins [ 02/Feb/12 ]

FIX:

Index: common/src/main/java/com/sun/enterprise/util/cluster/Paths.java
===================================================================
— common/src/main/java/com/sun/enterprise/util/cluster/Paths.java (revision 52337)
+++ common/src/main/java/com/sun/enterprise/util/cluster/Paths.java (working copy)
@@ -57,12 +57,13 @@
throw new NullPointerException();

// forward slashes are fine for ALL supported platforms

  • String nodeDir = node.getNodeDirAbsoluteUnixStyle();
    + String nodeParentDir = node.getNodeDirAbsoluteUnixStyle();
  • if (nodeDir == null)
  • nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes/" + node.getName();
    + if (nodeParentDir == null)
    + nodeParentDir = node.getInstallDirUnixStyle() + "/glassfish/nodes";
  • return nodeDir;
    + nodeParentDir += node.getName();
    + return nodeParentDir;
    }

public final static String getDasPropsPath(final Node node) {

Comment by Byron Nevins [ 02/Feb/12 ]

Actually it is:

nodeParentDir += "/" + node.getName();

Not

nodeParentDir += node.getName();

Comment by scatari [ 02/Feb/12 ]

Byron,
Looking at the suggested code changes from above

BEFORE:
if (nodeDir == null)
nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes/" + node.getName();

AFTER:
if (nodeDir == null)
nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes";

nodeDir += node.getName();

Shouldn't it be like this?

AFTER:
if (nodeDir == null)

{ nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes"; nodeDir += node.getName(); }

as you would like to do nodeDir += node.getName(); only when nodeDif is null
?

Comment by Byron Nevins [ 02/Feb/12 ]

{{{

public final static String getNodeDir(final Node node)

{ if (node == null) // don't do that! throw new NullPointerException(); // forward slashes are fine for ALL supported platforms String nodeParentDir = node.getNodeDirAbsoluteUnixStyle(); if (nodeParentDir == null) nodeParentDir = node.getInstallDirUnixStyle() + "/glassfish/nodes"; return nodeParentDir + "/" + node.getName(); }

}}}

Comment by Byron Nevins [ 02/Feb/12 ]

Sathyan –

No. That's the bug. add the node.getName() in ALL cases.

Comment by Byron Nevins [ 02/Feb/12 ]

Note that all the code above is BEFORE building and testing. I'm trying to get the answers in here quickly.
Just look at the code to get the gist of the problem...

No way is code going to be changed if it isn't perfect...

Comment by Byron Nevins [ 02/Feb/12 ]

BEFORE;

d:\gf\branches\3.1.2\cluster>asadmin create-node-dcom -W d:/pw -w hudson --nodehost vaio --installdir
c:/glassfish3 --nodedir c:/glassfish3/glassfish/nodes v2
Command create-node-dcom executed successfully.

d:\gf\branches\3.1.2\cluster>vid
d:\gf\branches\3.1.2\cluster>asadmin create-instance --node v2 q1
Successfully created instance q1 in the DAS configuration, but failed to create the instance files on node v2 (vaio).

Command failed on node v2 (vaio): Command _create-instance-filesystem executed successfully.

To complete this operation run the following command locally on host vaio from the GlassFish install location c:/glassfish3:

bin/asadmin --host WNEVINS-LAP --port 4848 create-local-instance --nodedir c:/glassfish3/glassfish/nodes --node v2 q1
Command create-instance completed with warnings.

===================================================================================================

AFTER:

d:\gf\branches\3.1.2\cluster\common>asadmin create-instance --node v2 q2
Using DAS host WNEVINS-LAP and port 4848 from existing das.properties for node
v2. To use a different DAS, create a new node using create-node-ssh or
create-node-config. Create the instance with the new node and correct
host and port:
asadmin --host das_host --port das_port create-local-instance --node node_name instance_name.
Command _create-instance-filesystem executed successfully.
Port Assignments for server instance q2:
JMX_SYSTEM_CONNECTOR_PORT=28692
JMS_PROVIDER_PORT=27682
HTTP_LISTENER_PORT=28086
ASADMIN_LISTENER_PORT=24854
JAVA_DEBUGGER_PORT=29015
IIOP_SSL_LISTENER_PORT=23826
IIOP_LISTENER_PORT=23706
OSGI_SHELL_TELNET_PORT=26672
HTTP_SSL_LISTENER_PORT=28187
IIOP_SSL_MUTUALAUTH_PORT=23926
The instance, q2, was created on host vaio
Command create-instance executed successfully.

Comment by Byron Nevins [ 02/Feb/12 ]

Fixed Files for 3.1.2

Don't use for 4.0

Permanent clean fix is adding ducktype methods to Node.java

To see the 2 files:

cd ....\cluster\common
jar xvf fix.zip

===========================
PathsTests is a unit test and svn is freaking out and calling EVERY line as changed.

here are the changes for Paths.java

Index: src/main/java/com/sun/enterprise/util/cluster/Paths.java
===================================================================
— src/main/java/com/sun/enterprise/util/cluster/Paths.java (revision 52389)
+++ src/main/java/com/sun/enterprise/util/cluster/Paths.java (working copy)
@@ -1,7 +1,7 @@
/*

  • DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS HEADER.
    *
  • * Copyright (c) 2011 Oracle and/or its affiliates. All rights reserved.
    + * Copyright (c) 2011-2012 Oracle and/or its affiliates. All rights reserved.
    *
  • The contents of this file are subject to the terms of either the GNU
  • General Public License Version 2 only ("GPL") or the Common Development
    @@ -37,6 +37,7 @@
  • only if the new code is made subject to such option by the copyright
  • holder.
    */
    +
    package com.sun.enterprise.util.cluster;

import com.sun.enterprise.config.serverbeans.Node;
@@ -52,19 +53,22 @@
public final static String DAS_PROPS_SUBPATH = "agent/config/" + DAS_PROPS_FILENAME;

// TODO: the Node class ought to do this!

  • public final static String getNodeDir(final Node node) {
    + public final static String getNodesDir(final Node node) { if (node == null) // don't do that! throw new NullPointerException(); - // forward slashes are fine for ALL supported platforms - String nodeDir = node.getNodeDirAbsoluteUnixStyle(); + String nodesDir = node.getNodeDirAbsoluteUnixStyle(); - if (nodeDir == null) - nodeDir = node.getInstallDirUnixStyle() + "/glassfish/nodes/" + node.getName(); + if (nodesDir == null) + nodesDir = node.getInstallDirUnixStyle() + "/glassfish/nodes"; - return nodeDir; + return nodesDir; }

+ public final static String getNodeDir(final Node node)

{ + return getNodesDir(node) + "/" + node.getName(); + }

+
public final static String getDasPropsPath(final Node node)

{ return getNodeDir(node) + "/" + DAS_PROPS_SUBPATH; }
Comment by Byron Nevins [ 03/Feb/12 ]

Sending common\src\main\java\com\sun\enterprise\util\cluster\Paths.java
Sending common\src\test\java\com\sun\enterprise\util\cluster\PathsTest.java
Transmitting file data ..
Committed revision 52417.





[GLASSFISH-18286] Regression: nodes tree not updated with new lifecycle module Created: 31/Jan/12  Updated: 31/Jan/12  Resolved: 31/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: Anissa Lam
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip


Attachments: JPEG File lfm-tree-not-updated.JPG    
Issue Links:
Duplicate
duplicates GLASSFISH-15997 Intermittent issue : Left tree not re... Open
Tags: 312_qa, 312_regression

 Description   

Nodes tree is not updated when a new lifecycle module is created. Steps to reproduce:

1. Click on Lifecycle Modules, New button.
2. Fill in the required fields and click Save to create a new lifecycle module.
3. New module is displayed in the table, but the tree on the left is never updated with it (screenshot attached).

This used to work in 3.1.1.



 Comments   
Comment by Anissa Lam [ 31/Jan/12 ]

There hasn't been any change related to LifeCycle since 3.1.1.
Are you able to consistently reproduce this ? On which browser ?
I test this on Firefox, Chrome and Safari, cannot reproduce this.
I am marking this as duplicate of GLASSFISH-15997, Intermittent issue that left navigation tree not updated.
Refresh the browser of clicking Home button should fix the issue.





[GLASSFISH-18282] Export Now button doesn't export loadbalancer xml file in IE9 Created: 31/Jan/12  Updated: 31/Jan/12  Resolved: 31/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arunkumar_s Assignee: Jason Lee
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL, IE9


Issue Links:
Duplicate
duplicates GLASSFISH-18281 IE9 and Google Chrome only: Export a ... Open
Tags: 312_qa

 Description   

This issue is browser specific(Only in IE9).

1) Login Admin GUI
2) Open LB and Export Tab and click export button

Page gets refreshed and couldn't able to see the load balancer xml file to download.



 Comments   
Comment by Anissa Lam [ 31/Jan/12 ]

We will address Export LB Config button in GLASSFISH-18281. Marking this as duplicate.





[GLASSFISH-18279] HTTP 500 error when LB name contains Hash (#) Created: 31/Jan/12  Updated: 31/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: arunkumar_s Assignee: srinik76
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL


Tags: 312_qa

 Description   

Steps

Login Admin console and create a LB with name contains #

When trying to create a LB config with invalid characters, Error message comes as "Invalid load-balancer name. The name must start with a letter, number or underscore and may contain only letters, numbers, and these characters: hyphen, period, underscore, hash and semicolon." as expected

If we create a LB with HASH(#) HTTP 500 error message throws up.



 Comments   
Comment by arunkumar_s [ 31/Jan/12 ]

Same issue happens with semi-colon

Comment by Anissa Lam [ 31/Jan/12 ]

Negative testing, lower the priority, will look at this in next release.





[GLASSFISH-18274] [Regression] Custom Install. Installation flow has changed after selecting "Configure again". Errors in port conflict are reported in Domain Info screen. Created: 30/Jan/12  Updated: 07/Feb/12  Resolved: 07/Feb/12

Status: Resolved
Project: glassfish
Component/s: installation
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b21

Type: Bug Priority: Critical
Reporter: Alex Pineda Assignee: scatari
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle Enterprise Linux (OEL)6. JDK1.7.0_02. GF 3.1.2 build19 (ogs-unix.sh). Firefox browser 3.6.25. Install option "Custom Install".


Tags: 312_qa, 3_1_2-approved

 Description   

The test scenario based on the "Custom" Install option to first install a "Server domain" and afterwards, to add a "standalone instance server" or a "clustered server instance". In GF 3.1.2 build16 and prior versions (3.1.1, 3.1), the process was done, after creating the domain, by selecting the "configure again" option and selecting the option of standalone or the clustered server instance, however, in GF 3.1.2 build19 (probably present since build17), the flow has changed and the "Domain Info" screen is displayed which has the options for port, user admin, password, etc.. If one selects "Next", the system complaints of a port conflict and highlights the Admin Port and Admin User. This is clearly a regression and probably a side effect of the recent Security remediation requirements.

A workaround, which we could document is to instruct the user, not to select "Next" in the Domain Info screen, but rather select the "Back" button which then puts the user in the "Configuration" screen and allows them to proceed forward with their selection (of "standalone server" or "clustered server" instances).



 Comments   
Comment by Joe Di Pol [ 31/Jan/12 ]

I did some experimentation and saw two basic bugs:

In all cases:

Select Custom Installation
Select Install and Configure
Specify new installation directory
Select JDK
Uncheck install Update Tool (to save time)
Click Install

Bug 1:

Select Create a server domain
Click Next, accepting defaults (domain is created and started)
Click "Configure again"

Bug: This takes you to the "Domain Info" screen, not back to the Configuration screen
(like it does in 3.1.1)

Bug 2:

Select Create a stand alone server instance
Click Next
Bug: this takes you to "Domain Info" screen not to the Instance Configuration screen (like it does in 3.1.1)
Next, accepting defaults.
Takes you to Instance Configuration
Next, accepting defaults
Configuration fails because domain was never created nor started even though you filled out domain information. So we lead the user down a path that should work, and it fails.

Comment by scatari [ 07/Feb/12 ]

• What is the impact on the customer of the bug?
Visible during advanced installer navigation sequence if the default sequence is skipped.

Is it a regression? Does it meet other bug fix criteria (security, performance, etc.)?
No, the bug has existed for a while, but shows up as we recently enabled "Domain Information" Panel
to adhere to Oracle's security compliance.

• What is the cost/risk of fixing the bug?
Code changes are minimal and isolated.
• Is there an impact on documentation or message strings?
No doc impact. No doc change required
• Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Custom install and Typical install on 1 unix and 1 windows platform
• Which is the targeted build of 3.1.2 for this fix?
3.1.2-b21





[GLASSFISH-18273] Exception in server.log when adding property to application configuration Created: 30/Jan/12  Updated: 20/Sep/12

Status: Open
Project: glassfish
Component/s: deployment
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Anissa Lam
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip


Attachments: Text File server.log.txt    
Tags: 312_qa

 Description   

Steps to reproduce:

1. Go to Domain, Applications Configuration.
2. Click on Add Properties at the bottom of the page and add some property.
3. Click Save. The following exception is printed in server.log:

[#|2012-01-30T14:01:28.271-0800|SEVERE|glassfish3.1.2|null|_ThreadID=48;_ThreadName=Thread-4;|Config Listener notification got interrupted
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:232)
at java.util.concurrent.FutureTask.get(FutureTask.java:91)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1.call(Transactions.java:290)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1.call(Transactions.java:269)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:139)
at org.jvnet.hk2.config.Transactions$Notifier$1$1.run(Transactions.java:169)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:442)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:139)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at java.util.regex.Matcher.getTextLength(Matcher.java:1140)
at java.util.regex.Matcher.reset(Matcher.java:291)
at java.util.regex.Matcher.<init>(Matcher.java:211)
at java.util.regex.Pattern.matcher(Pattern.java:888)
at org.glassfish.deployment.autodeploy.AutoDeployService.replaceTokens(AutoDeployService.java:391)

The property is saved fine, hence logging this as a minor priority.



 Comments   
Comment by Anissa Lam [ 30/Jan/12 ]

Stack tace shows coming from deployment code. transfer to deployment for evaluation.

Comment by Jeremy_Lv [ 20/Sep/12 ]

lidiam:
I try to reproduce it in the latest version of GFV4.0_b54 but it can not be reproduced.
I think it is fixed in the version of GFV4.





[GLASSFISH-18270] Test Connection in LB Config with a unavailable proxy host Error message shows twice Created: 30/Jan/12  Updated: 30/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: arunkumar_s Assignee: Anissa Lam
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL


Tags: 312_qa

 Description   

Steps:

1) Login admin GUI and create a LB with an unavailable proxy hostname /port name.
2) Open the LB config and Test connection.

Error message says "No route to hostNo route to host" . It looks repeated. Error message shouldn't be repeated and it should be clear.






[GLASSFISH-18269] [intermittent] SSLHandshakeException message on deploy; "PortUnification exception. java.lang.NoClassDefFoundError: javax/crypto/SunJCE_b" in the instance log Created: 30/Jan/12  Updated: 25/Apr/14

Status: Open
Project: glassfish
Component/s: security
Affects Version/s: 3.1.2_b19
Fix Version/s: future release

Type: Bug Priority: Minor
Reporter: varunrupela Assignee: JeffTancill
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive all-instances-jstacks.zip     Zip Archive first-test.zip     Zip Archive second-test.zip    
Tags: 312_qa, 3_1_2-exclude

 Description   

GF Build 19
Setup: 10 instance cluster
Platform: OEL6
JDK: JRockit

This issue occurred on the first test (deploy) that was done after a fresh install, creation of the Cluster and Enabling of Instance Access Logs.

All application deploys fail and the cluster becomes unusable.

[Note: It was found while creating a setup for debugging issue 18211]

Output from asadmin deploy:
****
INFO: Command Executed at agent machine agent1: /space/gf-ha/glassfish3/bin/asadmin --user admin deploy --availabilityenabled=true --force=true --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/clusterjsp.ear
Output : Application deployed with name clusterjsp.
WARNING: Command _deploy did not complete successfully on server instance instance102: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
Command deploy completed with warnings.

The second test showed the following output from asadmin deploy:
*****
INFO: Command Executed at agent machine agent1: /space/gf-ha/glassfish3/bin/asadmin --user admin deploy --availabilityenabled=true --force=true --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/clusterjsp.ear
Output : Command deploy failed.

Jan 29, 2012 4:22:29 PM com.sun.dft.glassfish.utils.Utility logCommandOutput
SEVERE: remote failure: Error occurred during deployment: Keys cannot be duplicate. Old value of this key property, nullwill be retained. Please see server.log for more details.
clusterjsp disabled failed
Failure: Command disable failed on server instance instance102: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
*****

instance102 has the following message on deploy:
******
[#|2012-01-29T16:19:00.226-0800|WARNING|glassfish3.1.2|grizzly|_ThreadID=22;_ThreadName=Thread-4;|GRIZZLY0059: PortUnification exception.
java.lang.NoClassDefFoundError: javax/crypto/SunJCE_b
at javax.crypto.Cipher.getInstance(DashoA13*..)
at com.sun.net.ssl.internal.ssl.JsseJce.getCipher(JsseJce.java:180)
at com.sun.net.ssl.internal.ssl.CipherBox.<init>(CipherBox.java:85)
at com.sun.net.ssl.internal.ssl.CipherBox.newCipherBox(CipherBox.java:119)
at com.sun.net.ssl.internal.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:369)
at com.sun.net.ssl.internal.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:407)
at com.sun.net.ssl.internal.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:386)
at com.sun.net.ssl.internal.ssl.CipherSuite.isAvailable(CipherSuite.java:144)
at com.sun.net.ssl.internal.ssl.CipherSuiteList.buildAvailableCache(CipherSuiteList.java:215)
at com.sun.net.ssl.internal.ssl.CipherSuiteList.getDefault(CipherSuiteList.java:239)
at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.initServer(SSLServerSocketImpl.java:130)
at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.<init>(SSLServerSocketImpl.java:118)
at com.sun.net.ssl.internal.ssl.SSLServerSocketFactoryImpl.createServerSocket(SSLServerSocketFactoryImpl.java:52)
at com.sun.grizzly.util.net.jsse.JSSE14SocketFactory.checkConfig(JSSE14SocketFactory.java:443)
at com.sun.grizzly.util.net.jsse.JSSE14SocketFactory.init(JSSE14SocketFactory.java:183)
at com.sun.grizzly.config.SSLConfigHolder.initializeSSL(SSLConfigHolder.java:361)
at com.sun.grizzly.config.SSLConfigHolder.configureSSL(SSLConfigHolder.java:239)
at com.sun.grizzly.config.HttpProtocolFinder.configureSSLIfNeeded(HttpProtocolFinder.java:248)
at com.sun.grizzly.config.HttpProtocolFinder.find(HttpProtocolFinder.java:105)
at com.sun.grizzly.config.ConfigProtocolFinderWrapper.find(ConfigProtocolFinderWrapper.java:72)
at com.sun.grizzly.portunif.PUReadFilter.findProtocol(PUReadFilter.java:522)
at com.sun.grizzly.portunif.PUReadFilter.execute(PUReadFilter.java:193)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]
******

Logs Attached:

  • logs from 2 tests
    Note: The client logs are in ant.output. Other logs are under <test-name>/logs/
  • JStacks of all the instances and the domain.


 Comments   
Comment by Tom Mueller [ 30/Jan/12 ]

Is this problem related to Grizzly?

Comment by oleksiys [ 30/Jan/12 ]

doesn't look so, IMO it's something security or/and jdk related.

Comment by varunrupela [ 30/Jan/12 ]

Marked the issue as intermittent as recreating the cluster and running the tests again did not show the problem.

Comment by Joe Di Pol [ 01/Feb/12 ]

As discussed in Bug Swat this has only been seen once and that case was on JRockit. QA will continue to monitor further tests. At this point this is not considered a 3.1.2 stopper.

Comment by varunrupela [ 02/Feb/12 ]

The same set of machines and environment has been used a number of times (after this error was observed) to re-install GF and re-create the cluster setup. The exact error in the log has not yet been observed.

However, we did observe an output error message with "SSLHandshakeException" on running asadmin commands after an instance showed an OutOfMemory error (while investigating another bug).





[GLASSFISH-18268] NPE in org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97) after instance shutdown is initiated Created: 30/Jan/12  Updated: 19/Apr/12

Status: Open
Project: glassfish
Component/s: failover
Affects Version/s: 3.1.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: sonymanuel Assignee: Mahesh Kannan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL6 + JRockit


Attachments: Text File server.log    
Issue Links:
Dependency
blocks GLASSFISH-18267 Traffic loss during instance start be... Resolved
Related
is related to GLASSFISH-18489 Web HA replicated session store not g... Closed
Tags: 312_qa, 3_1_2-exclude

 Description   

3.1.2 Build 17, JRockit 6u29, OEL6

One of the HA stress scenarios does a stop/start of instance while the traffic is in progress. We are observing this NPE in the instance after shutdown is initiated. The client is getting http response 503. Server log is attached.

[#|2012-01-17T22:52:25.453-0800|SEVERE|glassfish3.1.2|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=29;_ThreadName=Thread-2;|PWC3989: An exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)
at org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:407)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:396)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1052)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1015)
at org.glassfish.web.ha.session.management.ReplicationManagerBase.findSession(ReplicationManagerBase.java:151)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2852)
at org.apache.catalina.connector.Request.getSessionInternal(Request.java:2777)
at org.apache.catalina.connector.Request.unlockSession(Request.java:4201)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:342)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)



 Comments   
Comment by Mahesh Kannan [ 31/Jan/12 ]

Line 97 in ReplicatedDataStore is doing dataStore.load(). The npe is due to dataStore being null.

The only reason that this could happen is that dataStore.destroy() has been called. As Joe commented in 18267, this is happening after the instance is shutdown. When an instance is shutdown, dataStore.destroy() is called which sets the underlying dataStore to null causing the NPE.

I agree that the NPE could be prevented but that requires touching a bunch of files to guard against dataStore being null.

However, this NPE is not causing any data loss.

Comment by Joe Di Pol [ 01/Feb/12 ]

Excluding bug from 3.1.2 since the NPE is on shutdown and not causing test failures.

Comment by jonj27 [ 19/Apr/12 ]

I am receiving the same error on instance shutdown. This is causing any requests to not fail over and requiring me to sign in again on the other node. This is not ideal. There must be a workaround or a fix I can implement perhaps?

Thanks in advance





[GLASSFISH-18267] Traffic loss during instance start between the time 8080 is up and application is loaded Created: 30/Jan/12  Updated: 19/Sep/14  Resolved: 30/May/13

Status: Resolved
Project: glassfish
Component/s: web_container
Affects Version/s: 3.1.2
Fix Version/s: 4.1

Type: Bug Priority: Critical
Reporter: sonymanuel Assignee: oleksiys
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL6 + JRockit


Attachments: File framework.log.1     Text File server.log    
Issue Links:
Dependency
depends on GLASSFISH-18268 NPE in org.shoal.adapter.store.Replic... Open
Related
is related to GLASSFISH-18211 http client sometimes receives a "400... Resolved
Tags: 312_qa, 3_1_2-exclude, 3_1_2-release-note-added, 3_1_2-release-notes, 4_0-release-notes, 4_0-release-notes-completed, 4_0-release-notes-drafted

 Description   

Build : 3.1.2 build 17 , JRockit VM 6u29., OEL6

When running HA Stress tests we are observing traffic loss when a cluster instance is restarting. There is a time gap between the time when port 8080 is up and application loading is done. Loadbalancer probably detects http port is up and starts forwarding traffic. In this particular case, the time is approx. 11 sec and the request rate is 25 reqs/instance. 325 requests failed.

The client receives a 404 during this time :

SEVERE: 4xx or 5xx Response code - 404 HttpClientSession-47926
Jan 17, 2012 10:58:01 PM com.sun.dft.glassfish.stress.ReadWriteTest$MyTestListener postInvoke
SEVERE: <Unable to render embedded object: File (//www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>GlassFish v3 - Error report</title><style type="text/css"><) not found.--H1

{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}

H2

{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}

H3

{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}

BODY

{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}

B

{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}

P

{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}

A

{color : black;}

HR

{color : #525D76;}

--></style> </head><body><h1>HTTP Status 404 - </h1><hr/><p><b>type</b> Status report</p><p><b>message</b></p><p><b>description</b>The requested resource () is not available.</p><hr/><h3>GlassFish Server Open Source Edition 3.1.2-b17</h3></body></html>

Server log :

http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/build17/ha/oel6_stress/stress/com.sun.dft.glassfish.stress.ReadWriteTest/testReadWrite/logs/st-cluster/instance101/logs/server.log

[#|2012-01-17T22:58:00.873-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:28080]|#]

[#|2012-01-17T22:58:00.994-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:28181]|#]

[#|2012-01-17T22:58:01.025-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:24848]|#]

[#|2012-01-17T22:58:01.509-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0171: Created virtual server [server]|#]

[#|2012-01-17T22:58:01.518-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0171: Created virtual server [__asadmin]|#]

[#|2012-01-17T22:58:05.306-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0172: Virtual server [server] loaded default web module []|#]

[#|2012-01-17T22:58:08.619-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=10;_ThreadName=Thread-2;|core.start_container_done|#]

[#|2012-01-17T22:58:09.940-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1002: Security Manager is OFF.|#]

[#|2012-01-17T22:58:10.115-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1010: Entering Security Startup Service|#]

[#|2012-01-17T22:58:10.147-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1143: Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.|#]

[#|2012-01-17T22:58:10.405-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#]

[#|2012-01-17T22:58:10.417-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#]

[#|2012-01-17T22:58:10.539-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.|#]

[#|2012-01-17T22:58:10.591-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1011: Security Service(s) Started Successfully|#]

[#|2012-01-17T22:58:13.758-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=10;_ThreadName=Thread-2;|**GroupServiceProvider:: REGISTERED member event listeners for <group, instance> => <st-cluster, instance101>|#]

[#|2012-01-17T22:58:14.061-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0671: Loading application [ReadWriteServletTest] at [/ReadWriteServletTest]|#]

[#|2012-01-17T22:58:14.076-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application ReadWriteServletTest done in 5,380 ms|#]

[#|2012-01-17T22:58:14.085-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2-b17 (17) startup time : Felix (8,857ms), startup services(21,300ms), total(30,157ms)|#]



 Comments   
Comment by Shing Wai Chan [ 30/Jan/12 ]

I notice the following in the server.log:

exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)
at org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:407)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:396)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1052)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1015)
at org.glassfish.web.ha.session.management.ReplicationManagerBase.findSession(ReplicationManagerBase.java:151)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2852)
at org.apache.catalina.connector.Request.getSessionInternal(Request.java:2777)
at org.apache.catalina.connector.Request.unlockSession(Request.java:4201)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:342)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

Comment by Shing Wai Chan [ 30/Jan/12 ]

Assign to shoal team to investigate the NPE from org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97).

Comment by Joe Fialli [ 30/Jan/12 ]

The reported NPE in shoal is happening AFTER the clustered instance is started and immediately stops. Here are relevant log messages. Shoal GMS has an glassfish PREPARE_SHUTDOWN event handler. As part of that handling,
GMS leaves the cluster. The server initiated shutdown is at 22:52:23.747. I can not tell if someone ran
"asadmin stop-instance" on the instance or if something failed in the startup of the server and then glassfish server
shutdown was initiated. The NPE is not causing the server to shutdown, the NPE is occurring after the instance leaves the gms cluster. The shoal cache code is not handling quick shutdown properly but I do not think that is the main
issue for this bug.

[#|2012-01-17T22:44:08.990-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=21;_ThreadName=Thread-2;|SEC1011: Security Service(s) Started Successfully|#]

[#|2012-01-17T22:44:10.170-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=21;_ThreadName=Thread-2;|**GroupServiceProvider:: REGISTERED member event listeners for <group, instance> => <st-cluster, instance101>|#]

[#|2012-01-17T22:44:10.361-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=21;_ThreadName=Thread-2;|WEB0671: Loading application [ReadWriteServletTest] at [/ReadWriteServletTest]|#]

[#|2012-01-17T22:44:10.568-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=21;_ThreadName=Thread-2;|ReadWriteServletTest was successfully deployed in 2,513 milliseconds.|#]

[#|2012-01-17T22:52:23.747-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=22;_ThreadName=Thread-2;|Server shutdown initiated|#]

[#|2012-01-17T22:52:23.752-0800|INFO|glassfish3.1.2|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=22;_ThreadName=Thread-2;|GMSAD1008: GMSAdapter for member: instance101 group: st-cluster received GlassfishEventType: prepare_shutdown|#]

[#|2012-01-17T22:52:23.757-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1096: member: instance101 is leaving group: st-cluster|#]

[#|2012-01-17T22:52:23.759-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1010: Leaving GMS group: st-cluster with shutdown type set to InstanceShutdown|#]

[#|2012-01-17T22:52:23.774-0800|CONFIG|glassfish3.1.2|ShoalLogger|_ThreadID=23;_ThreadName=Thread-2;|GMS1065: Completed processing outstanding master node messages for member: instance101 group: st-cluster outstandingMessages to process: 1|#]

<deleted some log messages>

[#|2012-01-17T22:52:25.453-0800|SEVERE|glassfish3.1.2|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=29;_ThreadName=Thread-2;|PWC3989: An exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)
at org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:407)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:396)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1052)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1015)
at org.glassfish.web.ha.session.management.ReplicationManagerBase.findSession(ReplicationManagerBase.java:151)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2852)
at org.apache.catalina.connector.Request.getSessionInternal(Request.java:2777)
at org.apache.catalina.connector.Request.unlockSession(Request.java:4201)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:342)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:174)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.ja3DThread-2;|PWC3989: An exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)
at org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:407)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:396)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1052)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1015)
at org.glassfish.web.ha.session.management.ReplicationManagerBase.findSession(ReplicationManagerBase.java:151)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2852)
at org.apache.catalina.connector.Request.getSessionInternal(Request.java:2777)
at org.apache.catalina.connector.Request.unlockSession(Request.java:4201)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:342)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter
Bug tracking, issue tracking and project management software powered by Atlassian JIRA (v4.0.2#472) | Report a problem | Request a feature | Contact administrators

Comment by Joe Fialli [ 30/Jan/12 ]

The NPE in shoal cache in this bug report duplicates GLASSFISH-18268.
The NPE is happening AFTER the instance is shutdown.

[#|2012-01-17T22:52:23.747-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=22;_ThreadName=Thread-2;|Server shutdown initiated|#]

[#|2012-01-17T22:52:23.752-0800|INFO|glassfish3.1.2|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=22;_ThreadName=Thread-2;|GMSAD1008: GMSAdapter for member: instance101 group: st-cluster received GlassfishEventType: prepare_shutdown|#]

[#|2012-01-17T22:52:23.757-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1096: member: instance101 is leaving group: st-cluster|#]

[#|2012-01-17T22:52:23.759-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1010: Leaving GMS group: st-cluster with shutdown type set to InstanceShutdown|#]

[#|2012-01-17T22:52:23.774-0800|CONFIG|glassfish3.1.2|ShoalLogger|_ThreadID=23;_ThreadName=Thread-2;|GMS1065: Completed processing outstanding master node messages for member: instance101 group: st-cluster outstandingMessages to process: 1|#]

[#|2012-01-17T22:52:25.453-0800|SEVERE|glassfish3.1.2|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=29;_ThreadName=Thread-2;|PWC3989: An exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)

Comment by sonymanuel [ 31/Jan/12 ]

To clarify NPE is reported as a separate bug 18268. It has not connection with this particular issue.

Test scenario is as follows :

1. Deploy app.
2. Start http traffic (75 new requests per sec)
3. After 5 minutes, stop a cluster instance (asadmin stop-instance)

  • This shows the NPE in server logs (18268)
    4. After 5 minutes, start instance. During the instance start there is traffic loss (this issue). LB probably detects http port is up and starts forwarding traffic. But the application loading is not complete.
    5. After startup is complete test runs fine for few hours.
Comment by Joe Fialli [ 31/Jan/12 ]

this bug is concerning the http traffic loss.
The NPE in shoal cache is occurring after the server instance is being shutdown.
This issue is already reported as GF-18268 and has nothing to do with the traffic loss.
The instance is already shutdown when the NPE is occurring (as described in 18268).

The NPE is 2 seconds after server shutdown was initiated. This is exactly what GLASSFISH-18268 in the server.log attached to this issue.

To summarize, see that server shutdown is initiated (probably by asdmin stop-instance) and
then the NPE happens 2 seconds later.

#|2012-01-17T22:52:23.747-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=22;_ThreadName=Thread-2;|
Server shutdown initiated|#]

[#|2012-01-17T22:52:23.752-0800|INFO|glassfish3.1.2|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=22;_ThreadName=Thread-2;|GMSAD1008: GMSAdapter for member: instance101 group: st-cluster received GlassfishEventType: prepare_shutdown|#]

[#|2012-01-17T22:52:23.759-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=22;_ThreadName=Thread-2;|GMS1010: Leaving GMS group: st-cluster with shutdown type set to InstanceShutdown|#]

[#|2012-01-17T22:52:23.774-0800|CONFIG|glassfish3.1.2|ShoalLogger|_ThreadID=23;_ThreadName=Thread-2;|GMS1065: Completed processing outstanding master node messages for member: instance101 group: st-cluster outstandingMessages to process: 1|#]

[#|2012-01-17T22:52:25.453-0800|SEVERE|glassfish3.1.2|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=29;_ThreadName=Thread-2;|PWC3989: An exception or error occurred in the container during the request processing
java.lang.NullPointerException
at org.shoal.adapter.store.ReplicatedBackingStore.load(ReplicatedBackingStore.java:97)

Comment by sb110099 [ 31/Jan/12 ]

Upgrading to P2 as discussed with Sony .

Comment by Shing Wai Chan [ 31/Jan/12 ]

While AppServerStartup.run, the following is called in order:
(a) GrizzlyService create a GrizzlyProxy, which creates ContainerMapper and Mapper.
(b) WebContainer.postConstruct is invoked and then calls Connector.start and the above mapper is set.
(c) WebApplication.start is called for each web application

If the request comes before (b) or (c) are done, then ContainerMapper will process the request and hence 404 in this case.

The fix should be to let (b) and (c) finish. Then the request will be handled by CoyoteAdapter. One may need a mechanism to notify that the loading is started and done.

Comment by Shing Wai Chan [ 01/Feb/12 ]

In this case, Grizzly should not process the request before (b) and (c) are completed.

Comment by Joe Di Pol [ 01/Feb/12 ]

As discussed in Bug Swat we will not stop the 3.1.2 release to fix this bug, therefore I am excluding it from the release. Note that these tests (HA stress) were not run on 3.1.1 so it is difficult to know if this is a regression or not, but engineering thinks that the same design problem exists in 3.1.1.

Comment by oleksiys [ 02/Feb/12 ]

not sure I understand why it's assigned to Grizzly...
Grizzly has no idea if any other container is going or not going to be started as part of GF startup.

Comment by Shing Wai Chan [ 02/Feb/12 ]

One can know when other containers are started in AppServerStartup.run.
The issue is in GrizzlyService (a) above. (a) has done the following:
set up mapper
(ii) creates ContainerMapper
(iii) ContainerMapper serves requests.

need to happens before (b) above.
(iii) need to happens after other containers are started.

The fix need to separate and (iii) within (a). That is why I assign the issue to Grizzly team.

Comment by Rebecca Parks [ 07/Feb/12 ]

Added to the 3.1.2 Release Notes:

Description

A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when port 8080 is running and when application loading is complete. During this gap client requests are denied with a 404 error.

Workaround

None. The client must retry the request after application loading is complete.

Comment by oleksiys [ 11/Feb/12 ]

Should be fixed as part of GLASSFISH-18211 fix.

Comment by Tom Mueller [ 16/May/13 ]

Has this issue been resolved in 4.0?

Comment by oleksiys [ 22/May/13 ]

It hasn't. This issue and GLASSFISH-18211 have to be revisited for Glassfish v4.

Comment by oleksiys [ 30/May/13 ]

fixed

Project: glassfish
Revision: 62129
Author: oleksiys
Date: 2013-05-30 01:27:59 UTC

Log Message:
------------
+ fix issue #18267
https://java.net/jira/browse/GLASSFISH-18267

"Traffic loss during instance start between the time 8080 is up and application is loaded"

Startup AJP/JK listener only after receiving Glassfish SERVER_READY event

Comment by Gail Risdal [ 01/Jun/13 ]

The following has been added to the release notes:

Traffic loss during instance start between the time 8080 is up and application is loaded (18267)

Description
A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when port 8080 is running and when application loading is complete. During this gap client requests are denied with a 404 error.

Workaround
None. The client must retry the request after application loading is complete.

Comment by Gail Risdal [ 01/Jun/13 ]

Actually, looks like the write-up was carried forward from the 3.1.2 Release Notes.

Comment by oleksiys [ 03/Jun/13 ]

@Gail

can we change the description a bit like:

Traffic loss during instance start between the time AJP (Apache) connector port is up and application is loaded (18267)

Description
A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error.

Comment by dlaudams [ 03/Jun/13 ]

A workaround:

GlassFish will not start without an enabled listener. To get around this I add a secondary listener on a different port, disable the primary listener, restart/deploy, and then enabled the primary listener when running again.

My deployment sequence looks like this:

asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=false
asadmin undeploy <appname>
asadmin stop-domain domain1
asadmin start-domain domain1
asadmin deploy --name <appname> <earfile>
asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=true

Another benefit is that post-deployment checkout can be done against the secondary port before enabling the primary port.

Comment by Gail Risdal [ 05/Jun/13 ]

Revised the Description in the release notes write-up to now read:

A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when the AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error.

I left the title as it was because that reflects the actual title of the JIRA issue.

Comment by oleksiys [ 05/Jun/13 ]

Ok, thank you Gail!





[GLASSFISH-18262] Password alias with one letter name does not work Created: 27/Jan/12  Updated: 27/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Anissa Lam
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip


Attachments: JPEG File error-password-alias.JPG     Text File server.log.txt    
Tags: 312_qa

 Description   

Password alias that has a name length of one character does not work. Steps to reproduce:

1. Create a password alias with correct password called "j".
2. Go to an existing DCOM node and change password to the new alias. Hit Save. The following error is displayed:

An error has occurred
Warning: some parameters appear to be invalid. Node not updated. To force an update of the node with these parameters rerun the command using the --force option. com.sun.enterprise.util.cluster.windows.process.WindowsException: Logon failure: unknown user name or bad password.

DAS server.log contains:

[#|2012-01-26T17:35:14.091-0800|SEVERE|glassfish3.1.2|org.glassfish.admingui|_ThreadID=90;_ThreadName=Thread-4;|RestResponse.getResponse() gives FAILURE. endpoint = 'https://localhost:4848/management/domain/nodes/node/jedy/update-node-dcom'; attrs = '{windowsuser=$

{user.name}

, windowsdomain=jed-asqe-43, installdir=C:\as\dcomtest\glassfish3, windowspassword=*******, nodehost=jed-asqe-43, force=false}'|#]

When I create a password alias with a two or more letter name, it works fine.






[GLASSFISH-18261] Intermittent: ServletException on screen Created: 27/Jan/12  Updated: 27/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Anissa Lam
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip on OEL6


Attachments: Text File server.log.txt     JPEG File servletException.JPG    
Tags: 312_qa

 Description   

I cannot reproduce this issue, but due to the severity, I'm logging it in case anyone else sees it.

While navigating around Admin Console I got the following exception printed on the screen:

root cause:
java.lang.reflect.InvocationTargetException
root cause:
java.util.ConcurrentModificationException

I see in server.log the following:

Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException while attempting to process a 'beforeCreate' event for 'jmsHosts'.

I was not modifying jms hosts. I created a password alias, then clicked on an existing ssh node, looked at SSH User Authentication, then clicked on existing dcom node. That's when I got this exception.



 Comments   
Comment by Anissa Lam [ 27/Jan/12 ]

Since you cannot reproduce this, I am downgrading this to P4.
Also, you mentioned you are looking at Nodes page, and click on existing DCOM node, but the attached image shows the LB Config tree node being highlighted, and the right screen is about jmsHost creation.
Something really weird is going on.

Comment by lidiam [ 27/Jan/12 ]

I have no load balancers configured and you will notice that Nodes node is simply missing from the tree in the screenshot (should be between Load Balancers and Applications). My existing nodes are displayed under Load Balancers instead of Nodes in the screenshot. Not sure how this is even possible.





[GLASSFISH-18260] RuntimeException on screen when changing pool for a given jdbc resource Created: 27/Jan/12  Updated: 31/Jan/12  Resolved: 28/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b19
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b19.zip
DAS:
OEL6 with Jrockit 1.6.0_29
SSH Node and derby:
Solaris 10, JDK 1.6.0_14
DCOM Node:
Windows XP, JDK 1.6.0_22 , FF9


Attachments: Text File diff.text     XML File domain.xml     JPEG File jdbcResource-runtimeexception.JPG     File resourceEditTabs.inc     Text File server.log.txt    
Tags: 312_qa, 312_regression, 3_1_2-approved

 Description   

I tried switching jdbc resource jdbc/__default to use a custom created JDBC Pool and got a RuntimeException on the screen. The pool got changed. The following was printed to server.log:

Caused by: java.lang.IllegalArgumentException: URI is not absolute
at java.net.URI.toURL(URI.java:1080)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:157)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 80 more

Here are the steps to reproduce:

1. Start derby on a remote machine, tuppy.
2. Go to Admin Console, JDBC Connection Pools and click on New.
3. Enter the following:

  • name: tDerbyPool,
  • Resource Type: javax.sql.DataSource,
  • DB driver vendor: Derby-30.

Click Next. On next page uncheck Isolation Level and enter the following properties:

  • DatabaseName: sun-appserv-samples
  • Password: APP
  • ServerName: tuppy
  • ConnectionAttributes: ;create=true

Click Finish.

4. Go to JDBC Resources, jdbc/__default and change Pool Name to the newly created pool. Hit save - exception is printed on the screen.



 Comments   
Comment by Anissa Lam [ 27/Jan/12 ]

Can you create a new JDBC resource that reference this newly created Pool ?

Comment by lidiam [ 27/Jan/12 ]

Yes, no exception thrown when creating new resource with this pool.

Comment by lidiam [ 27/Jan/12 ]

Even though exception is thrown, the resource is referencing new pool afterwards and works fine. I deployed an application (scrumtoys) to a standalone instance on the remote machine where derby is running (and pool pointing to) and it works fine.

Comment by Anissa Lam [ 27/Jan/12 ]

Agree there shouldn't be exception shown on screen, but this wasn't affecting anything and the intended functionality is performed correctly.
Refreshing the screen will clear the exception and continue without any ill effect.
I am lowering this to P3.

Comment by Anissa Lam [ 27/Jan/12 ]

Is this consistently reproducible ?
Both Suma and I tried and cannot reproduce this. However, the database may not be running at that time. Please include instruction on starting the DB on tuppy.

Comment by lidiam [ 27/Jan/12 ]

It is consistently reproducible on this system. When switching JDBC Pool for an existing JDBC Resource the exception is thrown on the screen. I started database on tuppy prior to executing this test. Here is output from netstat:

tuppy(j2eetest):/export/home/j2eetest/3.1.2/glassfish3/glassfish/nodes/tup/int1/logs# netstat -a | grep 1527
.1527 *. 0 0 49152 0 LISTEN
tuppy.1527 jed-asqe-41.us.oracle.com.7896 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7897 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7898 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7899 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7900 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7901 6912 0 49232 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.7902 6912 0 49232 0 ESTABLISHED
tuppy.43704 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43704 49152 0 49152 0 ESTABLISHED
tuppy.43705 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43705 49152 0 49152 0 ESTABLISHED
tuppy.43706 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43706 49152 0 49152 0 ESTABLISHED
tuppy.43707 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43707 49152 0 49152 0 ESTABLISHED
tuppy.43708 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43708 49152 0 49152 0 ESTABLISHED
tuppy.43709 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43709 49152 0 49152 0 ESTABLISHED
tuppy.1527 jed-asqe-41.us.oracle.com.59139 6912 0 49232 0 ESTABLISHED
tuppy.43741 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43741 49152 0 49152 0 ESTABLISHED
tuppy.43742 tuppy.1527 49152 0 49152 0 ESTABLISHED
tuppy.1527 tuppy.43742 49152 0 49152 0 ESTABLISHED
tuppy(j2eetest):/export/home/j2eetest/3.1.2/glassfish3/glassfish/nodes/tup/int1/logs#

Comment by lidiam [ 27/Jan/12 ]

Attaching domain.xml.

Comment by shaline [ 27/Jan/12 ]

I am seeing this issue as well on b19 on solaris 11. I deleted a JDBC resource and saw the java.lang.IllegalArgumentException, but the resource was deleted, and after that any operation involving a JDBC resource throws a RuntimeException in the Console. This exception started to show up after deleting a existing JDBC resource.

Comment by Anissa Lam [ 27/Jan/12 ]

I can reproduce this problem now. This is not related to particular JDBC pool, or if the DB is running or switching the pool. Finally realize this is related to existence of additional instance/cluster.

I am setting this back to P2.

The bug will occur whenever there is other instance/cluster besides DAS.
All you need to do is press the SAVE button and you will see the exception in screen.
The property sheet will be saved, thus the switching of the pool will happen as user expected, as mentioned in the bug report. However, any change to the property table will be lost, since the exception occurs after saving the property sheet and before saving the property table.

Working on the fix now.

Comment by Anissa Lam [ 27/Jan/12 ]

Looking at the code, this is a more general bug and may affect other resource edit page.
Indeed, I tried and see that this bug affect every resource edit page. Save on any resource when there is other instance/cluster will cause the exception.
This is a must fix for 3.1.2

Comment by Anissa Lam [ 28/Jan/12 ]

I strongly believe there is a duplicate block of code thats due to copy & paste error. These block of code is repeated, and thus causing this issue.
I have tested this thoroughly, with DAS only and with cluster/instance created. Tested it on all resource edit page.
I am attaching the diff, and the file, resourceEditTabs.inc file.
To test this, just save the file as

glassfish/lib/install/applications/__admingui/common/resourceNode/resourceEditTabs.inc

and then restart the server.
I have requested Lidia and Shaline to test this out also.

Here is the change request:

  • What is the impact on the customer of the bug?
    Huge impact. User will not be able to edit any resource if there is other instance/cluster exists besides DAS.
  • What is the cost/risk of fixing the bug?
    I say the risk is min. It is a very localized change, in one file, affecting only the SAVE button of the resource edit page. It will not affect anything else. And this SAVE button is broken anyway without this change.
  • Is there an impact on documentation or message strings?
    No
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Test all resource edit page with and without any instance created.
  • Which is the targeted build of 3.1.2 for this fix?
    3_1_2_b20
Comment by Anissa Lam [ 28/Jan/12 ]

Shaline's email confirmed:

The problem is resolved now.. I tried the below scenarios on b19 with the fix, and the Exception did not show up:
--create and start cluster.
--create a pool and a resource and add both server and cluster targets.
--edit the resource, change the pool names and save.
--edit the pool, advanced pages and save.
----enable/disable resource.
--delete resource
--delete pools.

All these tests passed and no exceptions seen in the server.log or console .
I even Added a Resource using "Add Resource" button, edited the resource and added cluster target. Deleted the resource and there were no exceptions seen as before.

Comment by Anissa Lam [ 28/Jan/12 ]

Fix checked in to 3.1.2 branch. The trunk doesn't not have this dup. block of code and is fine.

Log Message:
------------
GLASSFISH-18260. Remove the duplicate block of code that causes editing of any resource page to throw exception when there is instance or cluster present.

Revisions:
----------
52330

Modified Paths:
---------------
branches/3.1.2/admingui/common/src/main/resources/resourceNode/resourceEditTabs.inc

Comment by lidiam [ 28/Jan/12 ]

After applying the fix exception is no longer printed on the screen when changing JDBC Pool. However, the following exception is printed to the server.log:

[#|2012-01-27T18:02:47.243-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=34;_ThreadName=Thread-4;|Exception while visiting com/sun/gjc/spi/ManagedConnection.class of size 24223
java.lang.NullPointerException
at org.glassfish.hk2.classmodel.reflect.impl.TypesImpl.getType(TypesImpl.java:78)
at org.glassfish.hk2.classmodel.reflect.impl.ModelClassVisitor.visit(ModelClassVisitor.java:119)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.objectweb.asm.ClassReader.accept(Unknown Source)
at org.glassfish.hk2.classmodel.reflect.Parser$5.on(Parser.java:363)
at org.glassfish.hk2.classmodel.reflect.util.JarArchive.onSelectedEntries(JarArchive.java:125)
at org.glassfish.hk2.classmodel.reflect.Parser.doJob(Parser.java:348)
at org.glassfish.hk2.classmodel.reflect.Parser.access$300(Parser.java:70)
at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:307)
at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:296)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

#]
Comment by lidiam [ 28/Jan/12 ]

Tried several more times and the above exception is not printed any more.

Comment by shaline [ 31/Jan/12 ]

Verified this on b20 nightly dated b20-01-30-2012, The java.lang.NullPointerException does show up in the server.log, the very first time we create a resource or edit a resource. But the resource is created and changes made to resource are saved successfully even with this exception in the server.log.





[GLASSFISH-18250] Cannot launch app client application in Chrome Created: 25/Jan/12  Updated: 26/Jan/12  Resolved: 25/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b18
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Anissa Lam
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b18.zip on OEL 6, Chrome 16.0.912.75 m on windows xp, JDK 1.6.0_30


Attachments: JPEG File appclient-error.JPG     PNG File runningOnChrome.png     Text File server.log.txt     Java Archive File showArgsGUI-client.jar    
Tags: 312_qa

 Description   

When accessing app client application from Google Chrome I get "unable to launch application" error with the following exception on the browser side:

java.io.IOException: Server returned HTTP response code: 502 for URL: http://jed-asqe-41:28080/___JWSappclient/___app/showArgsGUI-client/___dyn/___main.jnlp?arg=hey
at sun.reflect.GeneratedConstructorAccessor1.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$6.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at com.sun.deploy.net.BasicHttpRequest.doRequest(Unknown Source)
at com.sun.deploy.net.BasicHttpRequest.doRequest(Unknown Source)
at com.sun.deploy.net.BasicHttpRequest.doGetRequest(Unknown Source)
at com.sun.deploy.net.DownloadEngine.actionDownload(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getCacheEntry(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getCacheEntry(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getResourceCacheEntry(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getResourceCacheEntry(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getResource(Unknown Source)
at com.sun.deploy.net.DownloadEngine.getResource(Unknown Source)
at com.sun.javaws.Launcher.updateFinalLaunchDesc(Unknown Source)
at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
at com.sun.javaws.Launcher.launch(Unknown Source)
at com.sun.javaws.Main.launchApp(Unknown Source)
at com.sun.javaws.Main.continueInSecureThread(Unknown Source)
at com.sun.javaws.Main$1.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Server returned HTTP response code: 502 for URL: http://jed-asqe-41:28080/___JWSappclient/___app/showArgsGUI-client/___dyn/___main.jnlp?arg=hey
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
... 18 more

The same application works fine in Firefox 9. I'm attaching server.log from instance where the application was deployed.



 Comments   
Comment by Anissa Lam [ 25/Jan/12 ]

I am using Chrome 14.0.835.202 on Mac, I cannot reproduce the problem.
runs fine for me.
I am downgrading to P4 as this seems to be single user env. related, and submitter also said there is workaround by using other browser.

Comment by Anissa Lam [ 25/Jan/12 ]

Although it works for me on my Mac, this seems to be a known problem in chrome.

http://www.google.com/support/forum/p/Chrome/thread?tid=0e66a911b9807e74&hl=en
The Best answer there says:
"Doesn't look like this is getting fixed. It's been a known bug since April last year. The Chrome browser doesn't appear to detect the Java Webstart mime type. As a CTO I'm now steering people away from using Google Chrome and recommending Firefox."

nothing GUI can do. closing as won't fix.

Comment by andriy.zhdanov [ 26/Jan/12 ]

Just for information, Chrome on Ubuntu has similar issue in my case, http://localhost:8080/showArgsGUI-client?arg=first just downloads jnlp file, but does not launch an application (even though sample http://docs.oracle.com/javase/tutorialJWS/uiswing/painting/ex6/SwingPaintDemo3.jnlp is launched), however http://localhost:8080/___JWSappclient/___app/showArgsGUI-client/___dyn/___main.jnlp?arg=hey works fine.

So in the case, the problem is just that first url does not have .jnlp extension, this is clearly different than reported problem.





[GLASSFISH-18249] AIX 7.1 the deployment of the timer apps - failed Created: 25/Jan/12  Updated: 26/Jan/12  Resolved: 26/Jan/12

Status: Closed
Project: glassfish
Component/s: security
Affects Version/s: 3.1.2_b18
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: easarina Assignee: kumarjayanti
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 312_qa, 312_regression

 Description   

Executed a deployment test on AIX 7.1 machine against b18, the deployment of all timer apps failed with the follow error messages:

===============================================================
DEPLOYMENT automatic-timer-ejb
[#|2012-01-24T16:58:35.785-0600|INFO|glassfish3.1.2|javax.enterprise.system.core.classloading.com.sun.enterprise.loader|_ThreadID=9;_ThreadName=Thread-7;|enterprise.automatic_timer_ejb.persistence.LogRecord actually got transformed|#]

[#|2012-01-24T16:58:35.802-0600|INFO|glassfish3.1.2|org.eclipse.persistence.session.file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record|_ThreadID=9;_ThreadName=Thread-7;|EclipseLink, version: Eclipse Persistence Services - 2.3.2.v20111125-r10461|#]

[#|2012-01-24T16:58:35.915-0600|INFO|glassfish3.1.2|org.eclipse.persistence.session.file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record|_ThreadID=9;_ThreadName=Thread-7;|file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record login successful|#]

[#|2012-01-24T16:58:35.916-0600|WARNING|glassfish3.1.2|org.eclipse.persistence.session.file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record.server|_ThreadID=9;_ThreadName=Thread-7;|Multiple [2] JMX MBeanServer instances exist, we will use the server at index [0] : [com.sun.enterprise.v3.admin.DynamicInterceptor@56b956b9].|#]

[#|2012-01-24T16:58:35.917-0600|WARNING|glassfish3.1.2|org.eclipse.persistence.session.file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record.server|_ThreadID=9;_ThreadName=Thread-7;|JMX MBeanServer in use: [com.sun.enterprise.v3.admin.DynamicInterceptor@56b956b9] from index [0] |#]

[#|2012-01-24T16:58:35.918-0600|WARNING|glassfish3.1.2|org.eclipse.persistence.session.file:/export/hudson/workspace/deployment-w/glassfish3/glassfish/domains/domain1/applications/automatic-timer-ejb/_log_record.server|_ThreadID=9;_ThreadName=Thread-7;|JMX MBeanServer in use: [com.sun.jmx.mbeanserver.JmxMBeanServer@342b342b] from index [1] |#]

[#|2012-01-24T16:58:35.960-0600|INFO|glassfish3.1.2|javax.enterprise.system.core.security|_ThreadID=9;_ThreadName=Thread-7;|JACC Policy Provider:Failed Permission Check: context (" ejb-timer-service-app/ejb-timer-service-app_internal ") , permission (" (javax.security.jacc.EJBMethodPermission TimerBean countTimersByContainer,Local,long) ") |#]

[#|2012-01-24T16:58:35.961-0600|WARNING|glassfish3.1.2|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=9;_ThreadName=Thread-7;|EJB5184:A system exception occurred during an invocation on EJB TimerBean, method: public int com.sun.ejb.containers.TimerBean.countTimersByContainer(long)|#]

[#|2012-01-24T16:58:35.962-0600|WARNING|glassfish3.1.2|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=9;_ThreadName=Thread-7;|javax.ejb.AccessLocalException: Client not authorized for this invocation
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1888)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at $Proxy201.countTimersByContainer(Unknown Source)
at com.sun.ejb.containers.EJBTimerService.createSchedules(EJBTimerService.java:1310)
at org.glassfish.ejb.startup.EjbDeployer.createAutomaticPersistentTimersForEJB(EjbDeployer.java:592)
at org.glassfish.ejb.startup.EjbDeployer.checkEjbBundleForTimers(EjbDeployer.java:543)
at org.glassfish.ejb.startup.EjbDeployer.event(EjbDeployer.java:516)
at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:128)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:452)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:389)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:66)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:736)

#]

[#|2012-01-24T16:58:35.964-0600|WARNING|glassfish3.1.2|javax.enterprise.system.container.ejb.com.sun.ejb.containers|_ThreadID=9;_ThreadName=Thread-7;|Timer restore or schedule creation error
javax.ejb.EJBAccessException
at com.sun.ejb.containers.BaseContainer.mapLocal3xException(BaseContainer.java:2323)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2096)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:1994)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:221)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at $Proxy201.countTimersByContainer(Unknown Source)
at com.sun.ejb.containers.EJBTimerService.createSchedules(EJBTimerService.java:1310)
at org.glassfish.ejb.startup.EjbDeployer.createAutomaticPersistentTimersForEJB(EjbDeployer.java:592)
at org.glassfish.ejb.startup.EjbDeployer.checkEjbBundleForTimers(EjbDeployer.java:543)
at org.glassfish.ejb.startup.EjbDeployer.event(EjbDeployer.java:516)
at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:128)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:452)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:389)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:66)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:736)
Caused by: javax.ejb.AccessLocalException: Client not authorized for this invocation
at com.sun.ejb.containers.BaseContainer.preInvoke(BaseContainer.java:1888)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
... 35 more

#]

[#|2012-01-24T16:58:35.966-0600|SEVERE|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=9;_ThreadName=Thread-7;|Exception while deploying the app [automatic-timer-ejb]|#]

[#|2012-01-24T16:58:35.967-0600|SEVERE|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=9;_ThreadName=Thread-7;|Failed to create automatic timers for StatelessSessionBean – null
org.glassfish.deployment.common.DeploymentException: Failed to create automatic timers for StatelessSessionBean – null
at com.sun.ejb.containers.EJBTimerService.createEJBException(EJBTimerService.java:297)
at com.sun.ejb.containers.EJBTimerService.recoverAndCreateSchedulesError(EJBTimerService.java:1389)
at com.sun.ejb.containers.EJBTimerService.createSchedules(EJBTimerService.java:1320)
at org.glassfish.ejb.startup.EjbDeployer.createAutomaticPersistentTimersForEJB(EjbDeployer.java:592)
at org.glassfish.ejb.startup.EjbDeployer.checkEjbBundleForTimers(EjbDeployer.java:543)
at org.glassfish.ejb.startup.EjbDeployer.event(EjbDeployer.java:516)
at org.glassfish.kernel.event.EventsImpl.send(EventsImpl.java:128)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:452)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:389)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1291)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:66)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:736)

#]

[#|2012-01-24T16:58:35.995-0600|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=9;_ThreadName=Thread-7;|Exception while deploying the app [automatic-timer-ejb] : Failed to create automatic timers for StatelessSessionBean – null|#]
=================================================================

I've tried to run the same test two times on the different AIX 7.1 machines and in both cases the deployment of the timer apps failed with the same error messages. When I've executed the same test on AIX 6.1 machine, the timer apps were deployed without any issues. Also the timer apps were deployed successfully on other platforms such as: OEL 6, Solaris 11, Win 2008.



 Comments   
Comment by marina vatkina [ 25/Jan/12 ]

Assigning to security for initial evaluation.

Did this test pass on AIX on a previous build?

Comment by easarina [ 25/Jan/12 ]

Last time I've executed the deployment test on AIX 7.1 machine against b16. At that time all timer apps were deployed successfully.

Comment by easarina [ 26/Jan/12 ]

I've re-run the test on the same machines using b19 and don't see this issue any more, so I'm closing this bug.





[GLASSFISH-18241] HA run Solaris 11 - LB reconfiguration thread stops detecting new loadbalancer.xml. This around the time that DAS commands had issues. Created: 24/Jan/12  Updated: 30/Jan/12  Resolved: 30/Jan/12

Status: Closed
Project: glassfish
Component/s: load_balancer
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: kshitiz_saxena
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to GLASSFISH-18240 HA run Solaris 11 - observed DAS comm... Resolved
Tags: 312_qa

 Description   

Please see issue 18240 http://java.net/jira/browse/GLASSFISH-18240 for setup details and for initial description of this issue.

The loadbalancer reconfiguration is observed to have stopped after the point that the 2 DAS related issues were observed. Each test depends on (and triggers) reconfiguration of the loadbalancer (running on the Webserver). LB reconfiguration is triggered after deploying the application that is used by the test.

Its not clear if this reconfiguration problem is caused by the DAS related issues and so this separate bug has been filed to track the reconfiguration problem.

Issue 18240 has the relevant logs attached to it. It also has notes on the specific logs.



 Comments   
Comment by varunrupela [ 30/Jan/12 ]

This issue is not reproducible as a result of fix to issue 18240





[GLASSFISH-18240] HA run Solaris 11 - observed DAS commands take > 5 minutes, SSLHandshakeException on some DAS commands, Blocked Threads on one instance Created: 24/Jan/12  Updated: 30/Jan/12  Resolved: 30/Jan/12

Status: Resolved
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b18
Fix Version/s: 3.1.2_b20

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Archive File grizzly-config.jar     Zip Archive jstacks-jmaps.zip     Zip Archive logs-from-test-before-issue.zip     Zip Archive logs-from-test-when-issue-showed-up.zip     HTML File ws-error-log-after-taking-thread-dump    
Issue Links:
Related
is related to GLASSFISH-18241 HA run Solaris 11 - LB reconfiguratio... Closed
Tags: 312_qa, 3_1_2-approved

 Description   

GF Build 18
Setup: 10 instance cluster
Platform: Solaris 11
JDK: 1.6.0_30

On running of the full HA test suite, One of the MQ tests (GFMQ15LocalTest) showed the following issues:

  • Some of the DAS commands took longer than 300 seconds.
    Note: The tests invoke DAS commands via java.lang.Runtime.exec(). The destroy() method of the resulting java.lang.Process object is invoked if the DAS command takes longer than 300 seconds.
  • Following this, this test's client log shows some admin commands returning with the error: "Failure: Command delete-admin-object failed on server instance instance106: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake". This exception was also observed on many asadmin commands that were run in the tests that followed later in the run..
  • The loadbalancer reconfiguration is observed to have stopped after the point that the above issues appeared. Each test depends on (and triggers) reconfiguration of the loadbalancer (running on the Webserver). LB reconfiguration is triggered after deploying the application that is used by the test. Since its not clear if this reconfiguration problem is related to the above issues, a separate bug will be filed to track it.

More Observations (after the full HA test suite run was complete, below info collected):

  • JStack of the Webserver processes show BLOCKED threads.
  • JStack of instance106 show BLOCKED threads.

Note: These issues showed up while the MQ module of the HA tests was running. MQ tests continued to run after these issues showed up. Tests from the Coherence module also ran before the HA run completed. The JStack, JMap information was collected after completion of the HA tests run.

The following logs are being attached:

  • Logs from a test that was run before the issues appeared.
  • Logs from the test during which the issues were observed.
  • JStack and JMap of 2 instances, the domain and the webserver. JStack was taken on instance106 (as SSLHandshakeException was associated with it) and for instance101 (just to check on one other instance). Also attached a second JStack for instance106. It was taken about 8 minutes after the first one.
  • The MQ Broker process associated with instance106 was found to be running after the completion of all the HA tests. A Jstack and jmap of that broker process is also attached.
  • Webserver log is also attached. A thread dump was saved on to the Webserver instance's error log when running jstack on it.

Note regarding the attached Test logs:

  • The file ant.output contains the client logs for each test. Browsing this file will show all the das commands (look for "30000 milliseconds" to find commands that took longer than 300 seconds to run) & mq commands that were run during the particular test
  • The instance logs are here - st-cluster/<instance-name>/logs/server.log
  • The MQ logs are here - st-cluster/<instance-name>/log.txt
  • The Webserver (i.e. Loadbalancer) logs are under sjsws.

Please send a mail to receive information on:

  • VNC Details of the setup
  • Link to results from the full HA test suite run.
  • Information on the order in which tests are being run.


 Comments   
Comment by Tom Mueller [ 24/Jan/12 ]

Instance106 failed to response to a stop-instance command. The DAS received an SSLHandshakeException when trying to shut down instance106.

The jstack from instance106 shows 10 threads that are blocked with this stack trace:

"http-thread-pool-24849(5)" daemon prio=3 tid=0x01d5a400 nid=0x57 waiting for monitor entry [0xcb5bf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.sun.grizzly.config.GrizzlyEmbeddedHttps$LazySSLInitializationFilter.execute(GrizzlyEmbeddedHttps.java:200)
	- waiting to lock <0xdc8cafd0> (a $Proxy67)
	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
	at java.lang.Thread.run(Thread.java:662)

Please investigate whether there might be a deadlock possibility in the LazySSLInitializationFilter.execute code.

Comment by oleksiys [ 25/Jan/12 ]

yeah, there is non-synchronized access to HashMap: puts and gets... looks like HashMap internal pointers got broken because of that.
I'll need to fix that in Grizzly.

Comment by oleksiys [ 25/Jan/12 ]

fixed in grizzly
http://java.net/jira/browse/GRIZZLY-1182

Comment by oleksiys [ 25/Jan/12 ]

In order to fix the issue we need to integrate Grizzly 1.9.46

  • What is the impact on the customer of the bug?
    This issue is there for a long time. May occur in situation when GF is configured w/ 2+ HTTPS listeners and clients make request to to these listeners at the same time.
  • What is the cost/risk of fixing the bug?
    We need to synchronize assess to Ciphers map. Risk is very low.

How risky is the fix? How much work is the fix? Is the fix complicated?
Low risk.

  • Is there an impact on documentation or message strings?
    No.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Same set of tests QA used to discover the issue.
  • Which is the targeted build of 3.1.2 for this fix?
    b20
Comment by oleksiys [ 26/Jan/12 ]

patch for grizzly 3.1.2

Comment by oleksiys [ 30/Jan/12 ]

resolved.

Project: glassfish
Repository: svn
Revision: 52346
Author: oleksiys
Date: 2012-01-30 18:48:17 UTC
Link:

Log Message:
------------
+ fix issue #18240
http://java.net/jira/browse/GLASSFISH-18240

"HA run Solaris 11 - observed DAS commands take > 5 minutes, SSLHandshakeException on some DAS commands, Blocked Threads on one instance"

Integrating Grizzly 1.9.46

Approved by Joe





[GLASSFISH-18234] Restart required not cleared with instance restart Created: 21/Jan/12  Updated: 26/Jan/12

Status: Open
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b18
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: Byron Nevins
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b18.zip


Attachments: Text File server.log.txt    
Tags: 312_qa, 312_regression, 3_1_2-exclude

 Description   

Restart required is not cleared with instance restart but it is cleared when executing stop and then start instance. Steps to reproduce:

1. Follow steps from issue 18233 to get a clustered instance in restart required state, since it's inconsistent right now.
2. Go to clustered instance page and click on Restart button. After server restarts, the instance is still displayed as requiring restart.
3. Click on Stop and then Start buttons. Now the instance is reported as running.

This does not work in CLI either:

  1. asadmin list-instances
    Enter admin password for user "admin">
    cll01 running; requires restart
    clj01 running
    Command list-instances executed successfully.
  2. asadmin restart-instance cll01
    Enter admin password for user "admin">
    cll01 was restarted.
    Command restart-instance executed successfully.
  3. asadmin list-instances
    Enter admin password for user "admin">
    cll01 running; requires restart
    clj01 running
    Command list-instances executed successfully.

There are no errors in server.log



 Comments   
Comment by Tom Mueller [ 25/Jan/12 ]

The cause of this is that synchronization is not working with restart-instance. The restart-instance command does not pass a limited use token to the _restart-instance command that runs on the instance, so it does not pass it to start-local-instance, so start-local-instance is unable to authenticate with the DAS to do resynchronization. This means that the restart required flag is not cleared on the DAS when the instance is restarted.

Comment by Tom Mueller [ 26/Jan/12 ]

Byron, please take a look at whether there would be a simple fix for this for 3.1.2.
If not, we'll defer this for 3.1.2 and fix it on the trunk only.

Comment by Joe Di Pol [ 26/Jan/12 ]

Since this isn't a 3.1.2 stopper I'm excluding it. If we happen to quickly come up with a simple, low risk fix we can revisit this for 3.1.2.

Comment by Byron Nevins [ 26/Jan/12 ]

1-2 days work to implement a solution.
1 day for thorough testing and adding devtests.
Total = 3 days

Notes -

see NodeRunner.java ~~ line 150 to see how to make a token
JavaClassRunner in common-util needs to be expanded to allow writng lines to stdin
restartinstance.java needs tocall restartrestartinstance with the magic temp. certificate.
restartrestartinstance then calls JavaClassRunner's new ctor (see below) with the "stdinlines"

Why is this hairy?

  • Have to handle the case where the instance was originally started with --_auxinput. I.e. can't just blindly add the follo9wing to the process command line:
    "--_auxinput -"
  • 90% of the code is handling NON-ASADMIN ways of starting:
    java -jar glassfish.jar
    etc., etc.
    Is the auxinput going to work for these non-asadmin calls? Needs plenty of testing.

============

Comment by Byron Nevins [ 26/Jan/12 ]

Definitely exclude this from 3.1.2

Comment by Byron Nevins [ 26/Jan/12 ]

The question was asked:

"Why doesn't restart-instance just do stop-instance followed by start-instance"?

1) Feature – we remember to restart with exactly the same possibly long & complicated list of arguments. A plain start could not do that.

2) if the instance is running in verbose mode, the output window is elegantly re-used (try it). It would be impossible with a plain start at least on Windows.

3) you can start many different (non-asadmin) ways and it will work.

4) This is the big one. If the instance is remote and is using a config node (i.e. no SSH, no DCOM) then once you stop the instance – that's it. You can never start it again from DAS. User has to go to the remote.
THIS IS A HUGE FEATURE!!!

Comment by Byron Nevins [ 26/Jan/12 ]

One final note. When restart was implemented there was no such temporary security token. When that code was added for start-xxx, restart was not included in the change!

Comment by Tom Mueller [ 26/Jan/12 ]

To reproduce:

1. start the domain, change the admin password to something, create a node on another host, create an instance i1

2. start the instance

3. do something to cause the restart-required flag to be set, such as:
asadmin set-log-attributes --target i1-config com.sun.enterprise.server.logging.GFFileHandler.rotationTimelimitInMinutes=3

4. restart the instance:

asadmin restart-instance i1

5. look at the status of the instances:

asadmin list-instance -l

Observer that the restart-required flag is still set on i1.

If you turn on the admin logging (javax.enterprise.system.tools.admin.level=FINE) then you will see log messages in the log that show that a user was not able to login from the host that is running the instance. These are the _synchronize-instance commands that are failing.





[GLASSFISH-18222] [JRockit-intermittent] Observed 100% CPU usage & deploy taking > 5 minutes when running the Coherence HA tests Created: 19/Jan/12  Updated: 26/Mar/13

Status: Open
Project: glassfish
Component/s: web_container
Affects Version/s: None
Fix Version/s: future release

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: Shing Wai Chan
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive coh-test-logs-1.zip     Zip Archive coh-test-logs-2.zip    
Issue Links:
Dependency
blocks GLASSFISH-18190 [JRockit] 100% CPU usage by 2 instanc... Closed
Tags: 312_qa, 3_1_2-exclude, 3_1_2-release-note-added, 3_1_2-release-notes, 3_1_2_review

 Description   

GF Build: 18
Setup: 10 instance cluster (3 machines)
Platform: OEL6
JDK: JRockit
Has this test passed before: No. This is the first time that the combination OEL6 + JRockit + CoherenceTests is being tried out.

Parent Issue: 18160. Observed with build 17. 18222 is a result of further investigation.

Following observed when running Coherence tests (specifically while deploy of the coherence enabled application is on):

  • 100% CPU usage on some machines. "top" was run manually on all the machines involved in the setup if it was observed that one of the test's deploy method was taking unusually long.
  • This was not the case with all Coherence HA tests.
  • In some Coherence HA tests when the invocation of "asadmin deploy" was observed to be taking long (> 5 minutes), some instance logs were observed to contain thread dumps, while others have coherence related exceptions.
  • Tests that hit this issue ended up failing as instances did not seem to be available to service a http request for the application in question.
  • Different instances (1 or more) were observed to be utilizing 100% of the CPU during different tests.
  • In one of the tests a lot of GMS FAILURE_EVENTs have been observed while the deploy is on. Since 100% CPU usage has been observed during the deploy, it is quite likely that GMS alive messages (don't remember the exact terminology) were getting missed.

Logs from 2 different tests have been attached:
a) The first set is that of a test that showed a number of GMS FAILURE_EVENTs. This test happened to run before the test from which the second set of logs was collected.
b) The second set of logs shows thread dumps (see log of st-cluster/instance105) and exceptions (see log of st-cluster/instance106).
Note: The setup was able to continue the tests run after stalling for a bit. The second set of logs have been attached for a test after it had already stalled the tests run for about 2 hrs. A "kill -SIGABRT pid" was done on the instances that were taking 100% CPU (instanc101 & instance104 - second set of logs). Those kill steps seem to have generated thread dumps in the logs. Instances 101 and 104 got restarted after the kill -SIGABRT.

Client logs are in the ant.output file.

  • Deploy for the first set of logs begins at 3:58:33 AM
  • Deploy for the second set of logs begins at 4:10:52 AM

The coherence cache server's logs are also available - see coherence/<machine-name>/cache-server.log.

  • There was only a single storage enable cache server running. It formed the cluster with the storage-disabled GF cache nodes
  • The cache-server.log shows indications of failed members around the time of deploy for both the test cases.

NOTE: A core dump is also available. This was obtained before and for a different test run than the 2 sets of attached test logs. A "kill -SIGABRT pid" in this case had created a core dump rather than a thread dump in the instance logs.

  • Please send us an email and we will send its location.


 Comments   
Comment by Shing Wai Chan [ 20/Jan/12 ]

I notice the following coherence related messages in the instance log files.
Per discussion with James, this is very likely due to a resource or network issue.
So, this may be a result rather a cause of 100% cpu usage.

#|2012-01-19T04:06:40.072-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=25;_ThreadName=Thread-2;|2012-01-19 04:06:40.070/1652.885 Oracle Coherence GE 3.7.0.0 <Error> (thread=Termination Thread, member=n/a): Full Thread Dump

[#|2012-01-19T04:06:40.294-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=25;_ThreadName=Thread-2;|2012-01-19 04:06:40.293/1653.108 Oracle Coherence GE 3.7.0.0 <Error> (thread=http-thread-pool-24849(2), member=n/a): Error while starting cluster: com.tangosol.net.RequestTimeoutException: Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster
MemberSet=ServiceMemberSet(
OldestMember=n/a
ActualMemberSet=MemberSet(Size=9, BitSetCount=2
Member(Id=1, Timestamp=2012-01-19 03:38:42.501, Address=10.133.185.54:8088, MachineId=13878, Location=site:us.oracle.com,machine:jed-asqe-24,process:8642, Role=CoherenceServer)
Member(Id=13, Timestamp=2012-01-19 03:58:48.887, Address=10.133.185.54:8090, MachineId=13878, Location=site:us.oracle.com,machine:jed-asqe-24,process:11176)
Member(Id=14, Timestamp=2012-01-19 03:58:50.606, Address=10.133.185.54:8092, MachineId=13878, Location=site:us.oracle.com,machine:jed-asqe-24,process:9028)
Member(Id=16, Timestamp=2012-01-19 03:59:12.252, Address=10.133.185.20:8090, MachineId=13844, Location=site:us.oracle.com,machine:jed-asqe-25,process:2421)
Member(Id=17, Timestamp=2012-01-19 03:59:19.923, Address=10.133.185.20:8094, MachineId=13844, Location=site:us.oracle.com,machine:jed-asqe-25,process:2408)
Member(Id=18, Timestamp=2012-01-19 04:00:47.551, Address=10.133.185.55:8088, MachineId=13879, Location=site:us.oracle.com,machine:jed-asqe-26,process:15657)
Member(Id=19, Timestamp=2012-01-19 04:01:02.177, Address=10.133.185.55:8090, MachineId=13879, Location=site:us.oracle.com,machine:jed-asqe-26,process:15651)
Member(Id=20, Timestamp=2012-01-19 04:01:04.076, Address=10.133.185.55:8092, MachineId=13879, Location=site:us.oracle.com,machine:jed-asqe-26,process:15690)
Member(Id=21, Timestamp=2012-01-19 04:01:18.349, Address=10.133.185.55:8094, MachineId=13879, Location=site:us.oracle.com,machine:jed-asqe-26,process:15658)
)
MemberId/ServiceVersion/ServiceJoined/MemberState
1/3.7/Thu Jan 19 03:38:42 PST 2012/JOINED,
13/3.7/Thu Jan 19 03:58:48 PST 2012/JOINED,
14/3.7/Thu Jan 19 03:58:50 PST 2012/JOINED,
16/3.7/Thu Jan 19 03:59:13 PST 2012/JOINED,
17/3.7/Thu Jan 19 03:59:19 PST 2012/JOINING,
18/3.7/Thu Jan 19 04:00:47 PST 2012/JOINED,
19/3.7/Thu Jan 19 04:01:02 PST 2012/JOINED,
20/3.7/Thu Jan 19 04:01:04 PST 2012/JOINED,
21/3.7/Thu Jan 19 04:01:18 PST 2012/JOINED
)
)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.onStartupTimeout(Grid.CDB:6)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start(Service.CDB:28)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.start(Grid.CDB:6)
at com.tangosol.coherence.component.net.Cluster.onStart(Cluster.CDB:61)
at com.tangosol.coherence.component.net.Cluster.start(Cluster.CDB:11)
at com.tangosol.coherence.component.util.SafeCluster.startCluster(SafeCluster.CDB:4)
at com.tangosol.coherence.component.util.SafeCluster.restartCluster(SafeCluster.CDB:10)
at com.tangosol.coherence.component.util.SafeCluster.ensureRunningCluster(SafeCluster.CDB:26)
at com.tangosol.coherence.component.util.SafeCluster.start(SafeCluster.CDB:2)
at com.tangosol.net.CacheFactory.ensureCluster(CacheFactory.java:427)
at com.tangosol.coherence.servlet.SessionHelper.checkEdition(SessionHelper.java:295)
at com.tangosol.coherence.servlet.SessionHelper.configure(SessionHelper.java:107)
at com.tangosol.coherence.servlet.SessionHelper.ensureSessionHelper(SessionHelper.java:482)
at com.tangosol.coherence.servlet.glassfish31.CoherenceWebSessionManager.init(CoherenceWebSessionManager.java:133)
at com.tangosol.coherence.servlet.glassfish31.CoherenceWebSessionManager.start(CoherenceWebSessionManager.java:577)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5370)
at com.sun.enterprise.web.WebModule.start(WebModule.java:503)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:917)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:901)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:735)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:2010)
at com.sun.enterprise.web.WebContainer.loadWebModule(WebContainer.java:1661)
at com.sun.enterprise.web.WebApplication.start(WebApplication.java:109)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:302)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:461)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:240)
at org.glassfish.deployment.admin.InstanceDeployCommand.execute(InstanceDeployCommand.java:187)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:348)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:363)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1066)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1200(CommandRunnerImpl.java:95)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1295)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1260)
at com.sun.enterprise.v3.admin.AdminAdapter.doCommand(AdminAdapter.java:461)
at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:212)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:747)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1046)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:105)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:91)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:56)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

Comment by Shing Wai Chan [ 24/Jan/12 ]

Does the same test pass in Oracle JDK?

Comment by varunrupela [ 24/Jan/12 ]

Yes. The coherence tests passed with Oracle JDK. The collected logs from that run do not contain any Thread Dumps and do not indicate delay in execution of DAS commands.

Will send a mail with the results link.

Comment by Shing Wai Chan [ 25/Jan/12 ]

Can you increase the memory for JRockit JVM? I would like to know whether the problems went away after the change.

Comment by varunrupela [ 25/Jan/12 ]

Is there an indication that memory has been an issue ?

Comment by sherryshen [ 30/Jan/12 ]

GF Build: 19
Setup: 10 instance cluster (3 machines)
Platform: OEL6
JDK: JRockit

The execution is hanging on coherence in run 10b,
jed-asqe-28 97% cpu
asqeopteron1,3 2.7% cpu

http://agni-1.us.oracle.com/JSPWiki/Wiki.jsp?page=V312HATest
Varun checked the the log on asqe-log, and found that
both the runs (10a and 10b) have Coherence related failures
due to issue 18222.

*****
[root@jed-asqe-28 coherence-http-runtime]# find . -name "ant.output" | xargs grep "30000"
./com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeTest4/ant.output: [testng] INFO: The asadmin command - /space/gf-ha/glassfish3/bin/asadmin --user admin deploy --availabilityenabled=true --force=true --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/clusterjsp.ear is taking longer than 300000 milliseconds.
./com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeTest24/ant.output: [testng] INFO: The asadmin command - /space/gf-ha/glassfish3/bin/asadmin --user admin deploy --availabilityenabled=true --force=true --target st-cluster /space/gf-ha/agent-repository/web/runtime/SimpleSession.war is taking longer than 300000 milliseconds.
[root@jed-asqe-28 coherence-http-runtime]#
[root@jed-asqe-28 coherence-http-runtime]# find . -name "server.log" | xargs grep "Thread Dump"
./com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeTest4/failoverRuntimeThroughLB/logs/st-cluster/instance101/logs/server.log:[#|2012-01-29T23:36:01.649-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=24;_ThreadName=Thread-4;|2012-01-29 23:36:01.642/437.818 Oracle Coherence GE 3.7.0.0 <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump
*****

http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/build19/ha/oel6nm/all/coherence-http-runtime/com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeTest4/failoverRuntimeThroughLB/logs/st-cluster/instance101/logs/server.log
http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/build19/ha/oel6nm/all/coherence-http-runtime/com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeTest4/failoverRuntimeThroughLB/logs/coherence/jed-asqe-28.us.oracle.com/cache-server.log

Comment by Shing Wai Chan [ 30/Jan/12 ]

In one of the thread dump, we notice the following:

Thread[Cluster|SERVICE_STARTING|STATE_ANNOUNCE|Member(Id=0, Timestamp=2012-01-29 23:30:08.126, Address=10.133.185.4:8090, MachineId=13828, Location=site:us.oracle.com,machine:jed-asqe-28,process:32266),5,Cluster]
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher.drainOverflow(PacketPublisher.CDB:0)
com.tangosol.coherence.component.util.daemon.queueProcessor.packetProcessor.PacketPublisher$InQueue.add(PacketPublisher.CDB:8)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.dispatchMessage(Grid.CDB:62)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.post(Grid.CDB:46)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.Grid.send(Grid.CDB:1)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onTimerAnnouncing(ClusterService.CDB:68)
com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify(ClusterService.CDB:11)
com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:42)
java.lang.Thread.run(Thread.java:662)

Comment by Joe Di Pol [ 01/Feb/12 ]

Current thinking is that this may be a problem with the Coherence code on JRockit. This will not be resolved in time for 3.1.2. We should document this in the release notes as a known issue with JRockit. The work-around is to use the normal Oracle JDK.

Comment by Shing Wai Chan [ 02/Feb/12 ]

If you see the 100% cpu usage again, then please collect the following information. This will help coherence team to debug this further:

  • If this is reproducible, can we take thread dumps every few seconds?
  • Do we know which processes are using up CPU? (using top the pid and then identify what process that pid is)
  • Do we know if this is in user land or kernel? (from top)
Comment by Shing Wai Chan [ 06/Feb/12 ]

An issue is filed on Coherence as follows:
13689509: 100% CPU USAGE IN DEPLOYING COHERENCE 3.7.1 WEB APPLICATION WITH JROCKIT.

Comment by varunrupela [ 07/Feb/12 ]

We are Observing similar thread dumps and GMS related failures in the GF instance server logs with coherence 3.7.1 as well. Will work on collecting the requested frequent thread dumps and then upload the logs.

Comment by Tom Mueller [ 07/Mar/12 ]

Bulk update to set Fix Version to "not determined" for issues that had it set to a version that has already been released.





[GLASSFISH-18211] http client sometimes receives a "400 Bad Request" on running HA LB tests. Created: 19/Jan/12  Updated: 16/Feb/12  Resolved: 16/Feb/12

Status: Resolved
Project: glassfish
Component/s: web_container
Affects Version/s: 3.1.2_b17
Fix Version/s: 3.1.2

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File 18211.diff     Java Archive File grizzly-utils.jar     Java Archive File kernel.jar     Zip Archive patch.zip     Text File server.log     Text File server.log     Zip Archive test-logs.zip     Java Archive File web-core.jar    
Issue Links:
Related
is related to GLASSFISH-18267 Traffic loss during instance start be... Resolved
is related to GLASSFISH-18060 [JRockit-intermittent] ClassCastExcep... Resolved
Tags: 312_qa, 3_1_2-approved

 Description   

GF build 17
Setup: Cluster with 10 instances
Platform: OEL6 with JRockit
Has this passed before: OEL + JRockit combination was last used in GF 3.1 where the LB tests did pass and these "400 Bad Request" response were not seen.

Some of the HA LB tests get a "400 Bad Request" response for some of the requests. Different tests have this issue during different runs. Tests that have the issue in one run have been observed to pass in another run. No pattern has been identified yet to be able to provide a set of steps to reproduce the issue.

This issue was thought to have been related to issue 18060, but is now being filed as a separate issue as issue 18060 has also been observed independent of the "400 Bad Request" response and also based on inputs that the ClassCastException observed in 18060 may not lead to a "400 Bad Request". These issues may still be related though.

Logs are attached from 3 failed tests.

  • ant.output is the client log
  • the instance logs are under test<testName>/st-cluster
  • webserver logs are under test<testName>/sjsws
  • in all three tests, instance102 receives the first request or the first 2 depending on the scenario. The client receives a 200 response from instance102. On sending a subsequent request, the client then receives the "400 Bad request" response. The tests are based on round-robin LB and so it can be safely assumed that this error response is coming from instance103. Now, instance103 also shows the ClassCastException as reported in 18060.
  • The webserver logs do not seem to indicate any obvious issues with the request or indications of an error response.

The same tests have passed on the same platform (and other platforms - windows, solaris) and with the same http client. So it is unlikely that there is an error in the http request it self.

Logs have been attached from some failed https tests. The issue has also been observed with http tests. So, no pattern there either.

The "400 Bad Request" response has only been observed with LB test. Now, some of the LB tests add/remove new http(s) listeners that tests from other HA modules do not. We are trying to run all the LB tests minus these add/remove listener scenarios and see if we still have this issue.

If it helps, do let us know of a way to dump the http request received by the web-container ?



 Comments   
Comment by Shing Wai Chan [ 19/Jan/12 ]

Do we really use "round-robin LB" in the test? Round-robin LB is not supported in 3.1.1. Is it a new requirement?
Do those tests pass when sticky LB is used?

Comment by varunrupela [ 19/Jan/12 ]

"round-robin" is the name of the policy that is used. Please see the policy name in the test<testName>/sjswsj/loadbalancer.xml. It does function as a sticky LB when sessions are in use.

Yes, sticky LB test has passed on this setup. The issue is intermittent and so its possible that it shows up in a sticky LB test as well.

Comment by varunrupela [ 19/Jan/12 ]

Linking this with 18060 as it might be related.

Comment by Amy Roh [ 25/Jan/12 ]

Does this issue occur on Oracle JDK?

Comment by mzh777 [ 25/Jan/12 ]

The results with Oracle JDK 1.6.0_30 shows the issue exits.

Comment by Amy Roh [ 26/Jan/12 ]

I tried a few runs with SQE setup and can't seem to reproduce the errors.

http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy-jrockit/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-nopatch/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy0/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy1/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy2/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy3/summary/lb.html
http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-amy4/summary/lb.html

How often do these errors occur (%)? I need a more stable way to reproduce the errors to debug further.

Comment by mzh777 [ 26/Jan/12 ]

The runs you made have only 17 tests. The 2 runs by SQE (one for JRockit and the other for Oracle JDK) have 103 tests in each and the error string "400 Bad" can be found in both of them. Please update the gf-ha-qe/functional/lb/loadbalancer.testlist to tip.

Comment by Amy Roh [ 26/Jan/12 ]

Varun suggested the setup with 17 tests. Please let me know where can I find the list of 103 tests so I can update the test list.

Comment by Amy Roh [ 27/Jan/12 ]

I ran the full LB test list and all 105 tests passed with no "400 Bad Request" errors.

http://jed-asqe-24.us.oracle.com:1080/job/sherry-module/ws/gf-ha-qe/results-full/summary/lb.html

I'll re-run them again but any pointer to reproduce the error would be helpful.

Comment by Amy Roh [ 27/Jan/12 ]

Varun and Ming,

Can you try running the tests with the attached web-core.jar? When 400 happens, it should log out more info with a heading "GLASSFISH-18211".

Thanks,
Amy

Comment by Amy Roh [ 30/Jan/12 ]

When 400 happens sometimes, the host is null in the MappingData where it should use the default host.

===

host: null
context: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/clusterjsp]
wrapper: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/clusterjsp].StandardWrapper[jsp]
servletName: jsp
contextPath: /clusterjsp
requestPath: /HaJsp.jsp
wrapperPath: /HaJsp.jsp
pathInfo: null
redirectPath: null

===

host: StandardEngine[glassfish-web].StandardHost[server]
context: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/webapps-simple]
wrapper: StandardEngine[glassfish-web].StandardHost[server].StandardContext[/webapps-simple].StandardWrapper[HelloWorldExample]
servletName: HelloWorldExample
contextPath: /webapps-simple
requestPath: /servlet/HelloWorldExample
wrapperPath: /servlet/HelloWorldExample
pathInfo: null
redirectPath: null

Comment by Amy Roh [ 31/Jan/12 ]

400 happens when MappingData.host is null. Default host mapping "server" is present. hosts[pos].name equals the defaultHostName "server", however, hosts[pos].object is NULL, hence, mappingData.host is NULL.

com.sun.grizzly.util.http.mapper.Mapper:863.

 

if (mappingData.host == null) {
    Host[] hosts = this.hosts;
    int pos = findIgnoreCase(hosts, host);
    if ((pos != -1) && (host.equalsIgnoreCase(hosts[pos].name))) {
        mappingData.host = hosts[pos].object;
        hostPos = pos;
        contexts = hosts[pos].contextList.contexts;
        nesting = hosts[pos].contextList.nesting;
    } else {
        if (defaultHostName == null) {
            return;
    }
    pos = findIgnoreCase(hosts, defaultHostName);
    if ((pos != -1) && (defaultHostName.equalsIgnoreCase(hosts[pos].name))) {
        mappingData.host = hosts[pos].object;

// XXX hosts[pos]name equals defaultHostName "server", however, hosts[pos].object is NULL

        hostPos = pos;
        contexts = hosts[pos].contextList.contexts;
        nesting = hosts[pos].contextList.nesting;
    } else {
        return;
    }
}

Comment by Amy Roh [ 01/Feb/12 ]

Looking at the logs, the com.sun.dft.glassfish.lb.RR.http.MultipleFailoverHttpOnly test which fails sometimes follows this scenario.

1. delete-http-listener --target st-cluster http-listener-2
2. deploy --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/webapps-simple.war
3. export-http-lb-config --config gf-ha-qe-lb-config /space/gf-ha/agent-repository//loadbalancer.xml
4. send requests to http://jed-asqe-24.us.oracle.com:80/webapps-simple/session
5. start-cluster st-cluster
6. create-http-listener --target st-cluster http-listener-2
7. undeploy --target st-cluster webapps-simple

The bug report says "We are trying to run all the LB tests minus these add/remove listener scenarios and see if we still have this issue". Has this happened and what is the result?

Comment by varunrupela [ 01/Feb/12 ]

No. We did not get a chance to try a run where there is no change in the listeners.

Are there indications that the issue could be related to addition/removal of listeners ?

Comment by Amy Roh [ 02/Feb/12 ]

Here is the sequence for some tests that fail intermittently.

The app gets deployed successfully.

Both Grizzly and WebContainer get restarted during load balancer configuration.

The first request is sent to instance105 and the client receives a 200 response from instance105.

[testng] ** Sending Request 1 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session
[testng] ** Serving Instance for Request 1:instance105

Between Grizzly startup and WebContainer startup, a second request is sent to the same instance105.

at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.<init>(GrizzlyProxy.java:121)
at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:445)
at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct

A second request to instance105 happens before the Mapper is fully configured and Connector is started.

at com.sun.grizzly.util.http.mapper.Mapper.internalMap(Mapper.java:900)
at com.sun.grizzly.util.http.mapper.Mapper.map(Mapper.java:821)
at com.sun.enterprise.v3.services.impl.ContainerMapper.map(ContainerMapper.java:335)
at com.sun.enterprise.v3.services.impl.ContainerMapper.mapUriWithSemicolon(ContainerMapper.java:323)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)

before the mapper is fully set -

at org.apache.catalina.connector.Connector.start(Connector.java:1481)
at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529)
at org.apache.catalina.startup.Embedded.start(Embedded.java:936)
at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:599)

On all subsequent requests to instances, the request's Mapper contains .object=null because the Mapper wasn't fully configured yet.

[testng] ** Sending Request 2 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=2nd&datavalue=https
[testng] Instance failure detected for instance105

[testng] ** Sending Request 3 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=3rd&datavalue=https
[testng] Instance failure detected for instance109

[testng] ** Sending Request 4 to Webserver > http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=4th&datavalue=https
[testng] Instance failure detected for instance107

[testng] ** Sending Request 5 to Webserver jed-asqe-24.us.oracle.com:80> http://jed-asqe-24.us.oracle.com:80/webapps-simple/session?dataname=5th&datavalue=https

The issue is caused by concurrent thread, which updates the Mapper at the time, another thread is processing request.

Comment by Amy Roh [ 02/Feb/12 ]

This issue is related to 18267 since 400 happens when a request comes in between Grizzly and WebContainer startup before the mapper is set. The request shouldn't come in before the mapper is fully set.

"(a) GrizzlyService create a GrizzlyProxy, which creates ContainerMapper and Mapper.
(b) WebContainer.postConstruct is invoked and then calls Connector.start and the above mapper is set."

Comment by scatari [ 03/Feb/12 ]

As agreed in bug swat, this is not a 3.1.2 stopper, downgrading to P3.

Comment by Amy Roh [ 06/Feb/12 ]

Attach server.log which displays the request comes in between Grizzly and WebContainer start (GrizzlyService.postConstruct & WebContainer.postConstruct) and shows Mapper.mappingData.host=null which causes 400 Bad Request in CoyoteAdapter.doService.

Comment by Amy Roh [ 08/Feb/12 ]

Upon further investigation of the timestamps of the logs, the request that comes in between Grizzly and web container start is actually coming from loadbalancer health-check after the cluster is restarted and shouldn't cause 400 bad request and fail the test in theory.

Comment by varunrupela [ 08/Feb/12 ]

Summary of more findings:

  • A 400 response is being observed "even" when a request is received after web-container and grizzly have initialized correctly and started.
  • The 400 response "does" occur when a request is received between a grizzly startup and the web-container startup but in the case of the tests being run, this can happen only when LB health-check requests land on an instance between grizzly startup and web-container startup.

Below are details noted from results from one of the debug runs (http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/summary/lb.html)

Here are details of logs that show a 400 response when the request is NOT received between grizzly and web-container startup:

Some Patterns that were observed from the run:

a) It seems that changes to instance listener's may be triggering the issue. It is possible that the issue begins to appear after deletion and re-creation of the cluster's (i.e of all the instance's) http or https listener.
Detail: See http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/failed-tests-list for the list of tests that failed in this run. The complete test list is here: http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/loadbalancer.testlist.cvs. instance108 seemed to be the problem instance in this run. Not all instance's serve requests in all the tests. Now, instance108 served the last request correctly in test "com.sun.dft.glassfish.lb.RR.http.OnlyHttpListeners". This test deletes the https listener and re-creates it. The test that follows deletes the http listener and then re-creates it.
b) It seems that once an instance ends up in this state it begins to respond to all requests with a 400 response.
Detail: Once the issue showed up in the "com.sun.dft.glassfish.lb.RR.http.QuiesceInstance", every test where instance108 received a request ended up failing with a 400 response from instance108. This can be verified by looking into the logs of the failed tests listed in http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/failed-tests-list

Here are details of Logs that indicate that the 400 response occurring when a request IS received between a grizzly startup and the web-container startup are a response to the LB Health Check ping:

LB health check requests are auto generated by LB and a 400 response from those does not affect the test's pass/fail status.
Example test that passed but some instance's show a 400 response: http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/com.sun.dft.glassfish.lb.RR.https_routing_on.ChangeHttpsListener/html/index.html

Comment by varunrupela [ 08/Feb/12 ]

The issue needs further evaluation and hence upgraded it to a P2:

Next steps being worked upon:
1. Amy is trying a patch that would provide more information on how the mapper's host entry ends up being null even when the request is NOT received between grizzly and web-container startup.
2. Some pattern seems to have emerged from the failed tests (see new updates to the bug report). We are checking if there is a specific reproducible case that can be created out of it.
3. We are also trying to run the tests on a different OEL6 setup and see if 18211 is reproduced there.

Comment by varunrupela [ 08/Feb/12 ]

The issue was reproduced with build 21 on a second OEL6 setup.

Comment by varunrupela [ 08/Feb/12 ]

Here is info on a pattern that has been observed:

In both the last run and in the earlier saved run (see comment dated 08/Feb/12 04:08 PM) the StickyRoundRobin test was affected by this issue.

In both the runs, the instance that sent the 400 response had previously served a request correctly in the "OnlyHttpListeners" test.

Here is the test list between the two tests along with the main steps that are executed in those tests.

OnlyHttpListeners: deploys the app, deletes the cluster's (all instances) https listener, sends a number (cluster-size*2) requests, re-creates the https listener, undeploys the app
OnlyHttpsListeners: deploys the app, deletes the cluster's (all instances) http listener, sends 1 request, re-creates the https listener, undeploys the app
QuiesceApplication: deploys the app, sends a request, disables the app, sends a few more sticky requests, undeploys the app
QuiesceInstance: deploys the app, sends a request, disables the serving instance, sends a few more sticky requests, undeploys the app
StickyRoundRobin: deploys the app, sends a number (cluster-size*2) of reuqests, undeploys the app

Now, the issue seems to be exposed by tests that cause requests to be sent to all the instances - like the StickyRoundRobin (failed) and the OnlyHttpListener (passed). The test that failed for us is the StickyRoundRobin. Assuming that until OnlyHttpListener completes the cluster setup has no issues, it could be a step after this test and before the StickyRoundRobin test that caused the problem.

  • Is it possible that deletion and re-creation of a listener can somehow result in the mapper.host entry being set to null for some of the instances ? If this null value remains set, then tests that cause requests to be sent to all the instances, can end up seeing a 400 response.
  • Note that the QuieseApplication and QuiesceInstance tests did not affect the instance that sent the 400 response in the StickyRoundRobin test.
Comment by Shing Wai Chan [ 08/Feb/12 ]

The mapper will be modified when a listener is created.

In http://java.net/jira/browse/GLASSFISH-18060, there is an issue about the mapper. It is caused by difference in HK2 events. So, we may like to debug whether it is the same case here.

Comment by Amy Roh [ 08/Feb/12 ]

It is confirmed that mapper.host.object is null after https listener is recreated.

In the com.sun.dft.glassfish.lb.RR.http.OnlyHttpListeners test, the scenario follows

asadmin delete-http-listener http-listener-2
asadmin export-http-lb-config
asadmin create-http-listener http-listener-2

WebContainer#addConnector

        synchronized (mapperUpdateSync) {

            Mapper mapper = null;
            for (Mapper m : habitat.getAllByContract(Mapper.class)) {
                if (m.getPort() == port && m instanceof ContextMapper) {
                    ContextMapper cm = (ContextMapper) m;
                    if (httpListener.getName().equals(cm.getId())) {
                        mapper = m;
                        break;
                    }
                }
            }

            if (start) {
                connector.start();
            }

The newly created connector's mapper.host.object is confirmed to be null. The correct mapper returned by Habitat.getAllByContract(Mapper.class) should have the correct host.object by the time connector is started. Subsequent tests which send requests to the same instance without any restart of listener/instance sometimes fail with 400 Bad Request since host.object is null.

CoyoteAdapter#doService

            connector.requestStartEvent(request.getRequest(),
                request.getHost(), request.getContext());
            Container container = connector.getContainer();
            try {
                request.lockSession();
                if (container.getPipeline().hasNonBasicValves() ||
                        container.hasCustomPipeline()) {
                    container.getPipeline().invoke(request, response);
                } else {
                    // Invoke host directly
                    Host host = request.getHost();
                    if (host == null) {
                        response.sendError(HttpServletResponse.SC_BAD_REQUEST);
                        response.setDetailMessage(sm.getString(
                                "coyoteAdapter.noHost",
                                request.getRequest().getServerName()));
                        return;
                    } 

Comment by Amy Roh [ 08/Feb/12 ]

Assigning to Grizzly team to investigate why Mapper.host.object is null after https listener is recreated.

Comment by scatari [ 09/Feb/12 ]

Can we identity the 3.1.2 build where this issue was not seen? This would help us evaluate Grizzly changes quickly.

Comment by Amy Roh [ 09/Feb/12 ]

The sequence of the mapper events are different at runs. The issue might be related to GLASSFISH-18060. Full log with event debugging info attached.

[#|2012-02-08T18:25:56.733-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|DynamicConfigListener-18211 config changed ADD interface com.sun.grizzly.config.dom.NetworkListener GlassFishConfigBean.com.sun.grizzly.config.dom.NetworkListener http-listener-1|#]

[#|2012-02-08T18:25:56.735-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|WEBCONFIGLISTENER-18211 Web container config changed ADD interface com.sun.grizzly.config.dom.NetworkListener GlassFishConfigBean.com.sun.grizzly.config.dom.NetworkListener http-listener-1|#]

[#|2012-02-08T18:25:56.736-0800|INFO|oracle-glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=34;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:28080]|#]

[#|2012-02-08T18:25:56.747-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1249)
at com.sun.grizzly.util.http.mapper.Mapper.addHost(Mapper.java:214)
at org.glassfish.internal.grizzly.ContextMapper.addHost(ContextMapper.java:96)
at org.glassfish.internal.grizzly.V3Mapper.addHost(V3Mapper.java:103)
at com.sun.enterprise.v3.services.impl.ContainerMapper.configureMapper(ContainerMapper.java:143)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.<init>(GrizzlyProxy.java:121)
at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:451)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.processNetworkListener(DynamicConfigListener.java:156)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.access$100(DynamicConfigListener.java:79)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener$1.changed(DynamicConfigListener.java:105)
at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.changed(DynamicConfigListener.java:94)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

#|2012-02-08T18:25:56.766-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=27;_ThreadName=Thread-2;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1249)
at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:351)
at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:332)
at org.glassfish.internal.grizzly.ContextMapper.addContext(ContextMapper.java:119)
at org.glassfish.internal.grizzly.V3Mapper.addContext(V3Mapper.java:135)
at com.sun.enterprise.v3.services.impl.ContainerMapper.configureMapper(ContainerMapper.java:144)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.configureGrizzly(GrizzlyProxy.java:159)
at com.sun.enterprise.v3.services.impl.GrizzlyProxy.<init>(GrizzlyProxy.java:121)
at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:451)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.processNetworkListener(DynamicConfigListener.java:156)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.access$100(DynamicConfigListener.java:79)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener$1.changed(DynamicConfigListener.java:105)
at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
at com.sun.enterprise.v3.services.impl.DynamicConfigListener.changed(DynamicConfigListener.java:94)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.768-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1249)
at org.apache.catalina.connector.Connector.start(Connector.java:1483)
at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529)
at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263)
at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126)
at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.785-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1249)
at com.sun.grizzly.util.http.mapper.Mapper.addHost(Mapper.java:214)
at org.glassfish.internal.grizzly.ContextMapper.addHost(ContextMapper.java:96)
at org.glassfish.internal.grizzly.V3Mapper.addHost(V3Mapper.java:103)
at org.apache.catalina.connector.MapperListener.registerHost(MapperListener.java:466)
at org.apache.catalina.connector.MapperListener.init(MapperListener.java:203)
at org.apache.catalina.connector.Connector.start(Connector.java:1537)
at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529)
at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263)
at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126)
at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

[#|2012-02-08T18:25:56.787-0800|SEVERE|oracle-glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=34;_ThreadName=Thread-2;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1249)
at com.sun.grizzly.util.http.mapper.Mapper.addContext(Mapper.java:351)
at org.apache.catalina.connector.MapperListener.registerContext(MapperListener.java:550)
at org.apache.catalina.connector.MapperListener.init(MapperListener.java:213)
at org.apache.catalina.connector.Connector.start(Connector.java:1537)
at com.sun.enterprise.web.connector.coyote.PECoyoteConnector.start(PECoyoteConnector.java:529)
at com.sun.enterprise.web.WebContainer.addConnector(WebContainer.java:3263)
at com.sun.enterprise.web.reconfig.WebConfigListener$1.changed(WebConfigListener.java:126)
at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
at com.sun.enterprise.web.reconfig.WebConfigListener.changed(WebConfigListener.java:114)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:401)
at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:391)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Comment by varunrupela [ 09/Feb/12 ]

Question in comment dated - "09/Feb/12 12:45 AM"

  • Can we identity the 3.1.2 build where this issue was not seen? This would help us evaluate Grizzly changes quickly ?

The issue has been observed on OEL6 in all the 3.1.2 runs - first run on OEL6 was with build 16. At that time 18060 was thought to have been causing the 400 responses, but further investigation lead us to file this issue as a separate one.

The issue has not been observed yet on runs done on Solaris 11 and Windows 2008. Further analysis of the cause of the bug is needed to determine if it is platform specific.

Comment by varunrupela [ 09/Feb/12 ]

Removed 18267 from the "depends on" link and adding it as a "related to" link.

Comment by Rajiv Mordani [ 09/Feb/12 ]

Varun, Amy,
If the tests are indeed doing

1. delete-http-listener --target st-cluster http-listener-2
2. deploy --target st-cluster /space/gf-ha/agent-repository/glassfish-samples/webapps-simple.war
3. export-http-lb-config --config gf-ha-qe-lb-config /space/gf-ha/agent-repository//loadbalancer.xml

What is in the http lb config? Doesn't the http lb config put the entire url for the app on the instance including the host and port info which will be null because you have deleted the http listener and then done the export-http-lb-config?

  • Rajiv
Comment by Rajiv Mordani [ 09/Feb/12 ]

Also Varun are you sure nothing has changed in the tests framework etc that is causing this to fail?

Comment by varunrupela [ 09/Feb/12 ]

Some of the tests, delete ONE of the http listeners' while retaining the second one. Thus, ONE listener per instance is still available to serve requests. The resulting loadbalancer.xml that is generated from the export-http-lb-config command contains information about this available listener. See the below link for a sample lb xml.
http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/issue-18211-400response-build19-oracle-jdk/lb/com.sun.dft.glassfish.lb.RR.http.MultipleFailoverHttpOnly/html/index.html

No change in the test framework comes to mind that can lead to the Mapper's host entry to become null.

Comment by oleksiys [ 09/Feb/12 ]

Can you pls. try the attached kernel.jar and check if it makes any difference?

Comment by oleksiys [ 09/Feb/12 ]

Actually, looks like this patch fixes just
http://java.net/jira/browse/GLASSFISH-18267

usecase, so packet loss or issues, which happens during GF startup.

When WebContainer is getting started lazily (when the first webapp is deployed) - there issue is still there.

Comment by sb110099 [ 09/Feb/12 ]

Do you still want us to try the kernel.jar ? Or we can wait for your next update on this .

Thanks,
Sudipa

Comment by oleksiys [ 09/Feb/12 ]

Hi Sudipa,

you can try attached kernel.jar just as part of issue #18267.
Otherwise I'm working on another patch.

WBR,
Alexey.

Comment by oleksiys [ 09/Feb/12 ]

Pls. check the attached patch.zip (contains 2 jars).

Comment by Amy Roh [ 09/Feb/12 ]

I just started the lb test run and results will be ready in about 3 hours.

Comment by varunrupela [ 10/Feb/12 ]

Amy: Thanks for helping out.
The results from the first run look good - the issue did not show up.

Since, the issue is intermittent in nature, we would be doing a few repeat runs with the patch today and then report back.

Comment by varunrupela [ 10/Feb/12 ]

Oleksiy,

When does the web-container start lazily ?

  • Can deletion and re-creation of a listener lead to a web-container waiting to start lazily (on app deploy)?

Please help with some more information:

  • What is the fix contained in the patch ?
  • Once the mapper's host entry is null, until when does it remain null ?
  • What makes the issue intermittent in nature ?
Comment by oleksiys [ 10/Feb/12 ]

Can you pls. add grizzly-utils.jar (attached) to the prev. patch.zip?
It should give us additional information on the problem.

Thanks.

Comment by oleksiys [ 10/Feb/12 ]

Varun,

after you confirmed the issue is still there (after applying patch.zip), looks like WebContainer startup is not an issue (at least not the main one), there should be something more. May be it's related to LB + WebContainer (re)configuration. The latest attached jar should give us more info.

Comment by oleksiys [ 10/Feb/12 ]

pls. apply the latest attached kernel.jar on top of the prev. setup.
thx.

Comment by oleksiys [ 11/Feb/12 ]

patch

Comment by oleksiys [ 11/Feb/12 ]

Looks like the latest patch fixed the issue.

I'll try to share what I found and how it was fixed.
This issue contains of two issues:
a) GLASSFISH-18267, which occurs during WebContainer startup
b) Grizzly and WebContainer thread racing, when they process GF reconfig events (dynamic listeners)

Actually both subissues are related to Grizzly and WebContainer thread racing, when they simultaneously try to update Mapper, but "a)" occurs on WebContainer startup and "b)" when WebContainer is running, but needs to be reconfigured due to GF config changes.

Sub-issue "a)" we fixed by mutual locking of WebContainer startup and Grizzly mapping procedure, so they never happen at the same time, so incoming request will never see Mapper in non-consistent state.

Sub-issue "b)" was caused by fact that in kernel module we exposed Mapper (on Habitat) before actually initializing it. So during dynamic reconfiguration we had a problem that the same Mapper instance was being updated by Grizzly and WebContainer at the same time. We fixed the order of actions in the kernel module, so that we initialize Mapper in Grizzly first and only then share it via Habitat for WebContainer.

Comment by Dhiru Pandey [ 15/Feb/12 ]

This may possibly we an issue on Grizzly 2.0 and GlassFish trunk (aka 4.0) too. Adding this comment after discussion with Alexey.

Here is the commit log for the fix in GlassFish 3.1.2:
Project: glassfish
Repository: svn
Revision: 52549
Author: oleksiys
Date: 2012-02-11 19:48:02 UTC
Link:

Log Message:
------------
+ fix issue #18211
http://java.net/jira/browse/GLASSFISH-18211

"http client sometimes receives a "400 Bad Request" on running HA LB tests"

Approved by Joe

Revisions:
----------
52549

Modified Paths:
---------------
branches/3.1.2/core/kernel/src/main/java/com/sun/enterprise/v3/services/impl/GrizzlyProxy.java
branches/3.1.2/core/kernel/src/main/java/com/sun/enterprise/v3/services/impl/GrizzlyService.java
branches/3.1.2/core/kernel/src/main/java/com/sun/enterprise/v3/services/impl/ContainerMapper.java
branches/3.1.2/web/web-glue/src/main/java/com/sun/enterprise/web/WebContainer.java

Comment by oleksiys [ 16/Feb/12 ]

resolved for 3.1.2





[GLASSFISH-18192] [OLH] 404 error displayed on Domain Logs Help page Created: 13/Jan/12  Updated: 20/Jan/12  Resolved: 20/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b17
Fix Version/s: 3.1.2_b18, 4.0

Type: Bug Priority: Major
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17.zip


Attachments: JPEG File 404-help.JPG     Text File helpkeyDiff.text    
Tags: 312_gui_new, 312_qa, 312_verified, 3_1_2-approved

 Description   

When clicking Help button on Domain Logs page a 404 error is displayed on the help page (see attached screenshot).



 Comments   
Comment by Paul Davies [ 13/Jan/12 ]

The problem appears to be that the Domain Logs page in the administration console is not locating the OLH page.

The OLH page for the Domain Logs page is present at:

resource/common/en/help/ref-domainlogstab.html.

However, the URL in the Help Viewer is as follows:

http://localhost:4848/common/help/help.jsf

I would have expected the URL to be as follows:

http://localhost:4848/common/help/help.jsf?contextRef=/resource/common/en/help/ref-domainlogstab.html

Comment by Anissa Lam [ 13/Jan/12 ]

The issue is there is no helpkey specified for this page, and the Helplinks.properties file doesn't have the key.
The helpkey and the page name is specified in Helplinks.properties file.
eg.
serverInstDomainLogs=ref-domainlogstab.html

Usually the dev team will specify the help key and then the doc team fill in the html page name in the properties file, since dev team will not know that. So, for this page, somehow both dev and doc team missed it.

The fix is to specify this helpkey in the page and also add the above line to Helplinks.properties file.

Comment by Anissa Lam [ 13/Jan/12 ]

This isn't a show stopper bug, but having Error 404 is pretty bad image for the product.
The fix is very trivial and basically no risk.

  • What is the impact on the customer of the bug?
    User cannot get to the context sensitive online help for this page. The help window pops up with a 404 error.
  • What is the cost/risk of fixing the bug?
    basically no risk. specify the helpkey and add that to a properties file.
  • Is there an impact on documentation or message strings?
    No impact. OnlineHelp Doc set for this page has already written and available.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Just this Domain Log page to ensure everything is the same, and can bring up the help page.
  • Which is the targeted build of 3.1.2 for this fix?
    3_1_2_b18

Diff attached.

Comment by Anissa Lam [ 13/Jan/12 ]

Fixed in 3.1.2 branch.

Log Message:
------------
GLASSFISH-18192. Fix OLH helpkey for DomainLog page.
Approved by Joe.

Revisions:
----------
52116

Modified Paths:
---------------
branches/3.1.2/admingui/common/src/main/resources/org/glassfish/common/admingui/Helplinks.properties
branches/3.1.2/admingui/common/src/main/resources/appServer/domainLogs.jsf

Comment by lidiam [ 19/Jan/12 ]

Verified in ogs-3.1.2-b18.zip.

Comment by Anissa Lam [ 20/Jan/12 ]

reopen to add that this is fixed in the trunk.

Comment by Anissa Lam [ 20/Jan/12 ]

close issue again.





[GLASSFISH-18190] [JRockit] 100% CPU usage by 2 instances and DAS commands that apply to all instances are unusable after running HA test suite. Created: 13/Jan/12  Updated: 24/Jan/12  Resolved: 20/Jan/12

Status: Closed
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b17
Fix Version/s: 3.1.2_b18

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: varunrupela
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File instance109-activethreads.txt     Text File instance109-thread-dump.log    
Issue Links:
Dependency
depends on GLASSFISH-18222 [JRockit-intermittent] Observed 100% ... Open
Tags: 312_qa, 3_1_2-review

 Description   

GF Build 17
Platform: OEL6
JDK: JRockit 64-bit
Setup: 10 instance cluster on 3 machines.

The issue was noticed towards the end of the HA test suite run. Observations:

  • none of the java processes on one of the machines (3 machine setup) were responding.
  • "top" showed 100% CPU usage by instance105 and instance109 together.
  • jstack was not responding (perhaps as CPU usage was full)
  • one of the "kill -3" seemed to have worked on instance109 and the thread dump is attached
  • also tried SIGABRT as found at http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/sys_hangs.html and that killed instance105, but i can't find the core dump file
  • DAS commands like deploy/undeploy or start/stop-cluster would take longer than 5 minutes and the HA test suite would abort them.

We are continuing to debug the issue. One of things that will be tried is to do a repeated deply/undeploy and start/stop-cluster test (with and without coherence) and see if we are able to reproduce the issue.



 Comments   
Comment by sb110099 [ 13/Jan/12 ]

Added tag for 3.1.2 review. This is being observed since b17 and is currently holding up HA test runs.

Thanks,
Sudipa

Comment by Tom Mueller [ 13/Jan/12 ]

The attached instance109-activethreads.txt file is an excerpt from the thread dump that shows just the threads that are not waiting in a park, poll, or wait method.

There are 6 threads. The last 3 are probably related to writing the thread dump. The interesting ones are the first 3.

The "http-thread-pool-24851(2)" thread is in the process of running an InstanceDeployCommand, i.e., it is deploying an application to the instance. This thread is calling com.tangosol.coherence.servlet.SessionHelper.configure which eventually calls com.tangosol.coherence.component.util.daemon.queueProcessor.Service.start.

The "Cluster|Member" thread starts with a com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.onNotify which may have been triggered by the start from the previous thread. This thread eventually calls "com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.ClusterService.setAcceptingClients"

The other thread is "ContainerBackgroundProcessor" which is in the "org.apache.catalina.core.ContainerBase.findChildren" method. Line 986 in that file is a synchronized statement, so this thread might be blocked.

Since two of these suspicious threads are in Coherence code, it will be interesting to see the results of trying to reproduce this without Coherence.

Comment by varunrupela [ 17/Jan/12 ]

The setup is currently under use for analyzing failures in HA tests. Will work on reproducing the issue (without using Coherence) once the setup is available.

Comment by Tom Mueller [ 18/Jan/12 ]

Assigning to the submitter while this waits to be reproduced.

Comment by varunrupela [ 19/Jan/12 ]

We have been able to reproduce the 100% CPU usage via the coherence tests. A separate issue has been opened for that - issue 18222 and linked with this issue.

We are still investigating this issue and need to be sure that the tests run goes through when coherence is not involved. In the process if other related issues are found then those too will be linked with this parent issue.

QE will close this Parent issue once its linked issues are fixed.

Comment by varunrupela [ 20/Jan/12 ]

This issue has been reproduced when running the Coherence tests with JRockit. Details are on issue 18222.

We have decided to not pursue further investigation as the first set of logs also pointed to Coherence.

Comment by varunrupela [ 20/Jan/12 ]

Duplicate of 18222

Comment by Shing Wai Chan [ 24/Jan/12 ]

Per discussion with Coherence team, the Coherence exception is due to the resource or network issue. In other words, the Coherence exception is the result of the 100% cpu usage, not the cause of it.





[GLASSFISH-18185] 'update-node-ssh' command hangs when ssh port is not provided. Created: 13/Jan/12  Updated: 25/Jan/12

Status: Open
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b17
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: Byron Nevins
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17.zip


Attachments: JPEG File dcom-to-ssh-error.JPG     Text File server.log.txt    
Issue Links:
Related
is related to GLASSFISH-18209 Endless SSH Network Timeout Open
Tags: 312_gui_new, 312_qa, 3_1_2-exclude, 3_1_2-release-note-added, 3_1_2-release-notes

 Description   

Currently it's not possible to convert a DCOM node to an SSH node in Admin Console. For an existing DCOM node, when user changes to SSH (and selects password authentication in my case), and hits Save, the long running process popup is there for a long time, about 15 minutes. Then the following error is displayed:

An error has occurred
Check server log for more information.

The server.log file contains:

[#|2012-01-12T17:46:55.953-0800|WARNING|glassfish3.1.2|javax.enterprise.system.t
ools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=97;_ThreadName=Thread-2
;|Could not connect to host jed-asqe-43 using SSH.: There was a problem while co
nnecting to jed-asqe-43:135: Operation interrupted: host=jed-asqe-43 port=135 us
er=j2eetest password=<concealed> keyFile=/export/home/j2eetest/.ssh/id_rsa keyPa
ssPhrase=<concealed> authType=null knownHostFile=/export/home/j2eetest/.ssh/know
n_hosts|#]

[#|2012-01-12T17:46:55.953-0800|SEVERE|glassfish3.1.2|org.glassfish.admingui|_Th
readID=98;_ThreadName=Thread-2;|java.io.InterruptedIOException: Operation interr
upted;
java.io.InterruptedIOException: Operation interrupted;
restRequest: endpoint=https://localhost:4848/management/domain/nodes/node/jedy/u
pdate-node-ssh
attrs={sshpassword=*******, installdir=C:\as\dcomtest\glassfish3, nodehost=jed-a
sqe-43, sshuser=${user.name}}
method=POST|#]

The conversion works in CLI with the following command:

asadmin update-node-ssh --nodehost <host name> --sshport 22 --sshuser <user name> <node name>

However, if I execute the following in CLI, to convert DCOM to SSH node, it also hangs and then fails:

asadmin update-node-ssh --nodehost <host name> <node name>

Thus my guess is that Admin Console does not pass along the other two options (that should not be required). Assigning to Admin Console to verify and pass on to CLI, if this is the case. I understand this will most likely not get fixed for this release.



 Comments   
Comment by Anissa Lam [ 13/Jan/12 ]

As shown in the log Lidia pasted, the REST request sent is:

restRequest: endpoint=https://localhost:4848/management/domain/nodes/node/jedy/update-node-ssh
attrs={sshpassword=*******, installdir=C:\as\dcomtest\glassfish3, nodehost=jed-asqe-43, sshuser=${user.name}}
method=POST|#]
so, console is sending in all the info that user enters. At that time, Lidia probably didn't set the ssh-port. I verified that if sshport is specified, it will be sent in as well. I believed console is doing the correct thing.

Transfer to backend for evaluation.

Comment by lidiam [ 13/Jan/12 ]

That's correct, I did not set the ssh port since the default is always set to 22 and there was no need to change that.

Interesting thing to note is that if I create a new SSH node ssh port field is prepopulated to 22. If I choose to convert CONFIG node in Admin Console, ssh port is populated with 22, however, when I choose to convert DCOM node to SSH, ssh port field is not populated, so there is an inconsistency here. In fact if I enter ssh port in Admin Console when converting DCOM node to SSH, it works fine - we can document it as a workaround.

There are two issues here then:

1. ssh port is not populated when switching from DCOM to SSH node.
2. update-node-ssh command hangs when ssh port is not provided.

Comment by Anissa Lam [ 13/Jan/12 ]

I will file a separate P4 bug about sshport not populated when converting from DCOM to SSH.

I changed the summary of this bug to correctly reflect the issue.

This affects both CLI and GUI. I don't know how often user will convert DCOM node to SSH node, but if we decide to release note this, the work around is

  • ensure 'sshport' param is specified if using CLI
  • fill in port number (default is 22) when doing that in GUI.
Comment by Byron Nevins [ 18/Jan/12 ]

The problem is that a DCOM node has the "sshport" set to 135. When you run update-node-ssh it can't tell apart these 2 scenarios:

1) You're running the command on an existing ssh node that happens to use port 135 instead of 2
2) You're converting from a DCOM node that always has 135 set as the port number.

============

This is a gray area. Technically the software did precisely what you asked it to do – it updated the node config with the data you gave it and only the data you gave it. That's how it was designed.

It should be fixed in 4.0.

Recommended Fix

DCOM shouldn't bother with the port setting of 135. That's in a much lower abstraction - we never have to deal with the port number. DCOM should simply never use the sshport field.





[GLASSFISH-18183] Download log files - DAS logs not downloaded consistently Created: 13/Jan/12  Updated: 13/Jan/12

Status: Open
Project: glassfish
Component/s: logging
Affects Version/s: 3.1.2_b17
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: naman_mehta
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17.zip


Tags: 312_qa

 Description   

DAS log files are downloaded when downloading cluster log files but are not downloaded when downloading standalone instance log files. This behaviour should be consistent - either the DAS log files should always be downloaded or never.






[GLASSFISH-18182] Error message too long, hard to read in Admin Console Created: 13/Jan/12  Updated: 13/Jan/12

Status: Open
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b17
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Byron Nevins
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

build ogs-3.1.2-b17.zip


Attachments: JPEG File node-create-error.JPG    
Tags: 312_qa

 Description   

Currently when user tries to install a node on a remote host to a directory where glassfish is already installed the following is printed in Admin Console:

An error has occurred
Successfully connected to j2eetest@tuppy using keyfile /export/home/j2eetest/.ssh/id_rsa Command install-node-ssh failed. Ignoring unrecognized element schedules at Line number = 57 Column number = 18 System Id = file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml Public Id = null Location Uri= file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml CharacterOffset = 3425 Ignoring unrecognized element backup-configs at Line number = 62 Column number = 23 System Id = file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml Public Id = null Location Uri= file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml CharacterOffset = 3634 The remote installation directory, /export/home/j2eetest/3.1.2/glassfish3, already exists. Use the --force option to overwrite it.

It is hard to see the actual cause of the problem. We should, 1. in the least print the last sentence on a line by itself but 2. ideally not include the information in between. Hence it would be:

1.
An error has occurred
Successfully connected to j2eetest@tuppy using keyfile /export/home/j2eetest/.ssh/id_rsa Command install-node-ssh failed. Ignoring unrecognized element schedules at Line number = 57 Column number = 18 System Id = file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml Public Id = null Location Uri= file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml CharacterOffset = 3425 Ignoring unrecognized element backup-configs at Line number = 62 Column number = 23 System Id = file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml Public Id = null Location Uri= file:/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/config/domain.xml CharacterOffset = 3634

The remote installation directory, /export/home/j2eetest/3.1.2/glassfish3, already exists. Use the --force option to overwrite it.

2.
An error has occurred

Successfully connected to j2eetest@tuppy using keyfile /export/home/j2eetest/.ssh/id_rsa Command install-node-ssh failed.

The remote installation directory, /export/home/j2eetest/3.1.2/glassfish3, already exists. Use the --force option to overwrite it.

I'm attaching screenshot of the error message. I understand it is too late to fix this issue for this release, hence logging as minor. This issue surfaced after fixing http://java.net/jira/browse/GLASSFISH-18037.






[GLASSFISH-18181] [OLH] Change text on Domain Logs page Created: 12/Jan/12  Updated: 25/Jan/12  Resolved: 25/Jan/12

Status: Resolved
Project: glassfish
Component/s: docs
Affects Version/s: 3.1.2_b17
Fix Version/s: 3.1.2_b19

Type: Bug Priority: Major
Reporter: lidiam Assignee: Paul Davies
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17.zip


Tags: 312_gui_new, 312_qa

 Description   

Currently the Domain Logs online help page reads:

Note:

The ZIP archive also includes the log file for the domain administration server (DAS), regardless of whether you specify an instance or a cluster.

The above is not correct. DAS logs files are only downloaded with cluster logs but are NOT downloaded with standalone instance logs. I presume this is by design but will ping Anissa for confirmation.



 Comments   
Comment by Paul Davies [ 13/Jan/12 ]

Too late to fix for 3.1.2.

The corrected statement that the DAS log log files are downloaded with cluster logs but not instance logs should also be added to the CLI help for collect-log-files.

Comment by Paul Davies [ 25/Jan/12 ]

Removed the 3_1_2-release-notes tag. This issue was fixed in revision 52253, so does not need to be added to the Release Notes.

Comment by Paul Davies [ 25/Jan/12 ]

Fix committed in revision 52253.





[GLASSFISH-18172] Installer failure with JRockit JDK - java.lang.NoClassDefFoundError: org/openinstaller/provider/conf/ConfigControl Created: 11/Jan/12  Updated: 19/Jan/12  Resolved: 12/Jan/12

Status: Resolved
Project: glassfish
Component/s: installation
Affects Version/s: 3.1.2_b17
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Alex Pineda Assignee: scatari
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Oracle Enterprise Linux 6, JRockit JDK 1.6.0_29, GF 3.1.2 build17. Firefox 3.6.x browser.


Issue Links:
Duplicate
duplicates GLASSFISH-18171 Not able to install on Mac OS X Resolved
Tags: 312_qa

 Description   

The following scenario works with SunJDK 1.6.0_30. The problem is only seen with JRockit JDK 1.6.0_29.
In trying to do a silent install (from a Hudson job) that executes our Smoke test, as
o machine$ latest-glassfish-unix.sh -s silentInstallFile -s -j $JAVA_HOME

the following error is displayed:
Extracting the installer archive...
Extracting the installer runtime...
Extracting the installer resources...
Extracting the installer metadata...

Welcome to GlassFish installer

Using the user defined JAVA_HOME : /export/hudson/tools/jr-jdk1.6.0_29
Entering setup...
java.lang.NoClassDefFoundError: org/openinstaller/provider/conf/ConfigControl
at org.openinstaller.core.Orchestrator.main(Orchestrator.java:468)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.openinstaller.core.EngineBootstrap.main(EngineBootstrap.java:208)
SEVERE INTERNAL ERROR: org/openinstaller/provider/conf/ConfigControl

I can provide the link to the internal Test Hudson job.



 Comments   
Comment by Alex Pineda [ 11/Jan/12 ]

One more thing. This method of installation and scenario worked with GF 3.1.2 build 16.

Comment by scatari [ 12/Jan/12 ]

I think this is a dup of a bug that I filed earlier http://java.net/jira/browse/GLASSFISH-18171. Please verify that this doesn't happen on B16.

Comment by Alex Pineda [ 12/Jan/12 ]

As commented earlier and before the bug was marked as a dup. The problem was not seen on GF 3.1.2 build16.

Comment by Alex Pineda [ 19/Jan/12 ]

Verified fix in build18. Issue is resolved.





[GLASSFISH-18170] [intermittent] [SC-RM] [Solaris 11] HA Failover test results in the fault "SequenceAcknowledgement violates the cumulative Acknowledgement invariant" Created: 11/Jan/12  Updated: 19/Sep/14

Status: Open
Project: glassfish
Component/s: web_services
Affects Version/s: None
Fix Version/s: 4.1

Type: Bug Priority: Major
Reporter: varunrupela Assignee: Martin Grebac
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive 18170-oel6-logs.zip     Zip Archive 18170-solaris11-logs.zip     Zip Archive windows-run-logs.zip    
Tags: 312_qa, 3_1_2-exclude, metro2_2-waived

 Description   

GF 3.1.2 Build Used: 16
Setup: Cluster with 10 instances
Platform: Solaris 11
Has this passed before: Yes, on Solaris 10 with GF 3.1. Tests were not run on Solaris with GF 3.1.1. The test also passed on OEL6 with GF 3.1.2 build 17

Scenario:

  • Start a RM sequence with secure conversation enabled.
  • Send 3 messages of the sequence
  • Kill the serving instance
  • Send more messages that are part of the same sequence, after other instances have detected failure in the first serving instance.
  • Expectation: The instance to which the messages failover, is able to retrieve the sequence from its replica cache and continue the request/response without errors.

Observations:

  • The instance to which the messages failover, generates a fault - "The SequenceAcknowledgement violates the cumulative Acknowledgement invariant."
  • The following two differences were observed between the runs on Solaris 11 and on OEL6:

a) It appears that on Solaris 11, the 4th message that is sent from the client (see ant.output for client logs - look for "kill" and then scroll down to find the 4th message), seems to be received in the RM tube after a delay of about 18 seconds. While on OEL6, there isn't much of a difference. Below are logs from the Solaris 11 run.

        • ant.output (client log)
          [testng] Jan 10, 2012 6:40:27 AM com.sun.xml.ws.dump.MessageDumper dump
          [testng] INFO: Request message received in Tube [ com.sun.xml.ws.rx.rm.runtime.ClientTube ] Instance [ 2 ] Engine [ Metro/2.2-b12 (branches/2.2-6931; 2011-12-16T20:16:55+0000) JAXWS-RI/2.2.6-promoted-b14 JAXWS/2.2 svn-revision#unknown: Stub for http://ejp5363-vm1.india.sun.com:8080/symmetricbinding-rm/HelloWorldService ] Thread [ main ]:
          ****
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:44.098-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=29;_ThreadName=Thread-2;|Request message received in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 21 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(2) ]:
          ****

b) The other difference is related to the error that occurred. On Solaris 11, the response to Message number 4 contained an incorrect sequence acknowledgement range of 1..3 instead of 1..4. While on the run on OEL6 it was correct. Below is a log snippet from the Solaris 11 run.

        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:54.123-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=29;_ThreadName=Thread-2;|Response message processed in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 21 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(2) ]:
          ....
          <ns2:AcknowledgementRange Upper="3" Lower="1"/>
          </ns2:SequenceAcknowledgement>
          ****

An attempt at Analysis:

  • The incorrect sequence acknowledgement range seems to be related to the fault that is finally generated. Could it be caused due to slowness in handling Message number 4 (i.e. the first failed-over request on instance109 after instance101 failed) ? Its possible that it caused a back log of ack requests and so instance109 was unable to process the message correctly ?

Notes about the logs:

  • Logs will be attached to the bug. For instance logs see under st-cluster.
  • On the Solaris 11 run, the first serving instance was instance101 & the replica instance was instance109.
  • On the OEL6 run, the first serving instance was instance101 & the replica instance was instance105.


 Comments   
Comment by varunrupela [ 11/Jan/12 ]

Below are some detailed observations from the instance logs of the run on solaris 11:

  • The instance to which the messages failover (instance109), shows indications of receipt of the 4th request in its server log only after about 18 seconds of failure detection of the first serving instance (instance101).
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:44.098-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=29;_ThreadName=Thread-2;|Request message received in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 21 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(2) ]:
          ****

Between the time of failure detection and this point there seems to be some HA activity that is on.

        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:34.723-0800|FINER|glassfish3.1.2|com.sun.metro.commons|_ThreadID=27;_ThreadName=Thread-2;ClassName=[com.sun.xml.ws.commons.ha.HaContext] ;MethodName=validateRequest;|[METRO-HA] Thread[http-thread-pool-28083(3),5,grizzly-kernel] : Initialized from packet - replaced old HaState {packet=null, haInfo=null}

          with a new HaState{packet=com.sun.xml.ws.api.message.Packet@c424a4
          ****

  • The sequence data then seems to be retrieved from instance109's replica cache and the received message is processed.
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:45.156-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=33;_ThreadName=Thread-2;|Request message processed in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 30 ] Engine [ rm-server-tube-communicator ] Thread [ rm-server-tube-communicator-fiber-executor-thread-1 ]:
          ****
  • This is followed by instance109 receiving a number of AckRequested messages from the client.
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:45.156-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=33;_ThreadName=Thread-2;|Request message processed in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 30 ] Engine [ rm-server-tube-communicator ] Thread [ rm-server-tube-communicator-fiber-executor-thread-1 ]:
          ****
  • And then the Response message is received in tube:
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:45.252-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=33;_ThreadName=Thread-2;|Response message received in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 30 ] Engine [ rm-server-tube-communicator ] Thread [ rm-server-tube-communicator-fiber-executor-thread-1 ]:
          ****
  • The response message is processed but perhaps not completely. The Ack Range of messages received on the server side (Application Destination) still shows 1..3, while it should have updated to 1..4:
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:40:54.123-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=29;_ThreadName=Thread-2;|Response message processed in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 21 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(2) ]:
          ....
          <ns2:AcknowledgementRange Upper="3" Lower="1"/>
          </ns2:SequenceAcknowledgement>
          ****
          Possible the expected ack range for responses (sent from server side to test client) is also incorrectly 1..3, while the test client will soon end up receiving a 4th response
  • This is followed by a number of backed up AckRequested messages received at the server end and responded to.
  • Then comes in the next message in the sequence and with a sequence acknowledgement range of 1..4 (i think this indicates responses received on the App Source i.e. the test client side):
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:41:20.409-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=30;_ThreadName=Thread-2;|Request message received in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 29 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(1) ]:
          ....
          <ns5:AcknowledgementRange Upper="4" Lower="1"/>
          </ns5:SequenceAcknowledgement>
          ****
  • The below message then indicates an issue with the received message:
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:41:20.473-0800|WARNING|glassfish3.1.2|com.sun.metro.rx|_ThreadID=30;_ThreadName=Thread-2;|WSRM1125: "Message number [ 4 ] not found among the unacknowledged message numbers on a sequence [ uuid:b2f8149e-751b-4082-9801-ac6c92dffed8 ]."|#]
          ****
          I think this happens as the expected ack range for responses (sent from server side to test client) was only 1..3, while the test client indicates that a 4th response was also received.
  • A fault is then generated:
        • instance109's server.log_2012-01-10T06-41-21
          [#|2012-01-10T06:41:20.501-0800|INFO|glassfish3.1.2|com.sun.xml.ws.rx.rm.runtime.ServerTube|_ThreadID=30;_ThreadName=Thread-2;|Response message processed in Tube [ com.sun.xml.ws.rx.rm.runtime.ServerTube ] Instance [ 29 ] Engine [ com.sun.xml.ws.server.WSEndpointImpl@5fdc60 ] Thread [ http-thread-pool-28083(1) ]:
          ....
          <faultstring xml:lang="en">The SequenceAcknowledgement violates the cumulative Acknowledgement invariant.</faultstring>
          </SOAP-ENV:Fault>
          </S:Body>
          ****
Comment by Martin Grebac [ 11/Jan/12 ]

Hi Varun, is this the only failing test on Solaris 11? Is it a regression over 3.1.1?

Comment by varunrupela [ 11/Jan/12 ]

Yes. It is failing only on Solaris 11.
We ran the metro-HA tests on Solaris 10 with GF 3.1 and they had passed. Tests were not run on Solaris with GF 3.1.1.

Comment by sb110099 [ 19/Jan/12 ]

Just to clarify, Solaris 11 is a new platform in 3.1.2 .

Thanks,
Sudipa

Comment by varunrupela [ 20/Jan/12 ]

Observing the same failure on Windows 2008.
Logs have been attached.

  • In this case the first serving instance is instance102 and the failover instance is instance109.

The test has passed before with GF 3.1 on Windows 2008.

Comment by Martin Grebac [ 24/Jan/12 ]

Did the tests with 3.1 run on the same JDK? Is there an environment where we could debug this?

Comment by varunrupela [ 25/Jan/12 ]

A different JDK was used with GF 3.1 (JDK1.6.0_23)

Will send you mail with information on the available setup.

Comment by Martin Grebac [ 31/Jan/12 ]

Updating the bug as per discussion with Joe, adding 312 exclude keyword, removing review keyword since the issue is not a regression over 3.1/3.1.1 but an intermittent platform specific issue which makes it hard to debug.

We're not yet aware of what's the root cause, suspect it happens when message arrives to replica instance before the replica data is loaded and due to potential synch. issue the RM session data is overwritten.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

Comment by Marek Potociar [ 08/Mar/13 ]

Assigning to Martin to reassign as appropriate since I no longer own the domain.

Comment by Martin Grebac [ 13/Mar/13 ]

Should be targeted to 4.0.1





[GLASSFISH-18169] Log not displayed in Raw Log Viewer for instance on a remote config node Created: 11/Jan/12  Updated: 17/Apr/14

Status: Reopened
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: future release

Type: Bug Priority: Major
Reporter: lidiam Assignee: andriy.zhdanov
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17-01_09_2012.zip, DAS on Windows XP, config node on solaris


Attachments: JPEG File empty-raw-log.JPG     Text File server.log.txt    
Tags: 312_gui_new, 312_qa, 3_1_2-exclude

 Description   

Log is not displayed in Raw Log Viewer for an instance that's running on a remote CONFIG node. See attached screenshot and server.log, that contains the following exception:

[#|2012-01-10T17:23:57.281-0800|SEVERE|glassfish3.1.2|com.sun.jersey.spi.contain
er.ContainerResponse|_ThreadID=63;_ThreadName=Thread-2;|The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.NullPointerException
at com.sun.enterprise.server.logging.logviewer.backend.LogFilter.getLogF
ileForGivenTarget(LogFilter.java:320)
at org.glassfish.admin.rest.resources.custom.LogViewerResource.get(LogVi
ewerResource.java:121)
at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMe
thodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMeth
odDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchPr
ovider.java:205)

Steps to reproduce:

1. Create a CONFIG node on a remote machine in Admin Console (enter name and host only to make it easy).
2. Log in to the remote machine and create standalone instance locally via:

asadmin --host <das host> create-local-instance inty1

3. Start local instance on the remote machine via:

asadmin start-local-intsnce --host <das host> inty1

4. Go to Admin Console and navigate to the Instance General page. Click on View Raw Log - the server.log file content is not displayed.



 Comments   
Comment by Anissa Lam [ 11/Jan/12 ]

Assign to Andriy so he can evaluate the issue.
However, it has passed HCF, we will not be fixing this for 3.1.2. This is based on the reason that this seems to be a corner/not so common case.
The feature works on

  • instance with localhost CONFIG nodes
  • remote instance with SSH nodes
  • remote instance with DCOM nodes

The case that is not working is for case of CONFIG node on remote system, where user needs to login to the remote system to create a local instance and also start that local instance on that machine.

I am adding the exclude tag. If you think otherwise, please bring that up to Joe and the GUI team.

Andriy, if this is a backend bug, please reassign to logging. thanks

Comment by andriy.zhdanov [ 26/Jan/12 ]

I think this can not be supported for remote config nodes - as far as I understand, remote config nodes do not support any communication.

Comment by lidiam [ 27/Jan/12 ]

If we cannot display content of a log file for instances on remote, CONFIG nodes, then the View Raw Log button should not be displayed for those instances.

Comment by Anissa Lam [ 12/Feb/13 ]

Issues need to be addressed before 4.0 HCF (3/25)

Comment by Anissa Lam [ 15/Feb/13 ]

Move to 4.0.1 according to project triage guidelines.





[GLASSFISH-18161] Exception trying to create cluster in web distribution Created: 10/Jan/12  Updated: 13/Jan/12  Resolved: 10/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17

Type: Bug Priority: Critical
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-web-b17-01_09_2012.zip


Attachments: Text File diff.text     Text File server.log.txt     JPEG File syntaxException.JPG    
Tags: 312_qa, 312_regression, 312_verified, 3_1_2-approved

 Description   

Start Admin Console and click on Clusters, New Cluster. The following exception is printed on screen:

class com.sun.jsftemplating.layout.SyntaxException

The server.log file contains:

[#|2012-01-09T19:00:41.191-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.st
d.com.sun.enterprise.server.logging|_ThreadID=26;_ThreadName=Thread-2;|java.io.F
ileNotFoundException: /jms/jmsHandlers.inc
at com.sun.jsftemplating.util.IncludeInputStream.startInclude(IncludeInp
utStream.java:190)
at com.sun.jsftemplating.util.IncludeInputStream.read(IncludeInputStream
.java:80)
at com.sun.jsftemplating.util.IncludeInputStream.read(IncludeInputStream
.java:121)

Will attach screenshot and server.log.



 Comments   
Comment by Anissa Lam [ 10/Jan/12 ]

jms plugin is not included in the web distribution. The jms handler file needs to be in the cluster plugin.

What is the impact on the customer of the bug?
big impact. user will not be able to create cluster in web distribution.

What is the cost/risk of fixing the bug?
very min. Just move the file to the console cluster module

Is there an impact on documentation or message strings?
No

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
verify that cluster can be created in both web and glassfish dist.

Which is the targeted build of 3.1.2 for this fix?
3_1_2_b17

Diff is attached.
admingui/jms-plugin/src/main/resources/jmsHandlers.inc is moved to
admingui/cluster/src/main/resources/shared/jmsHandlers.inc

Comment by Jason Lee [ 10/Jan/12 ]

Change looks good to me.

Comment by marina vatkina [ 10/Jan/12 ]

JMS is not part of the WEB profile in Java EE 6

Comment by Anissa Lam [ 10/Jan/12 ]

Fix checked into 3.1.2 branch.

Log Message:
------------
GLASSFISH-18161. Move the jms handler to console cluster plugin package, since jms plugin is not available in Web profile.
Approved by Joe.

Revisions:
----------
51996
Modified Paths:
---------------
branches/3.1.2/admingui/cluster/src/main/resources/cluster/clusterNew.jsf
Added Paths:
------------
branches/3.1.2/admingui/cluster/src/main/resources/shared/jmsHandlers.inc

============

Revisions:
----------
51997
Modified Paths:
---------------
branches/3.1.2/admingui/jms-plugin/src/main/resources/jmsAvailabilityService.jsf

Comment by lidiam [ 13/Jan/12 ]

verified in build ogs-3.1.2-web-b17.zip





[GLASSFISH-18160] View Raw Log button missing on clustered instance page Created: 10/Jan/12  Updated: 11/Jan/12  Resolved: 11/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17

Type: Bug Priority: Major
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b17-01_09_2012.zip


Attachments: Text File log-diff.text     JPEG File raw-log-button-missing.JPG    
Tags: 312_gui_new, 312_qa, 312_verified, 3_1_2-approved

 Description   

View Raw Log button is missing on clustered instance page (see attached file).



 Comments   
Comment by Anissa Lam [ 10/Jan/12 ]

The fix below fixes the following:

  • add View Log Raw button in clustered instance page
  • for standalone instance, the View Log Raw button is enabled even when the instance is stopped.
  • The View Low Raw button is using the same logviewer window, thus prevent user to view the raw log and the formatted log at the same time.
  • What is the impact on the customer of the bug?
    Nothing really serious, but much better user experience if the above is fixed.

What is the cost/risk of fixing the bug?
very min. Those are very simple changes.

Is there an impact on documentation or message strings?
No

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
The test plan related to view raw log

Which is the targeted build of 3.1.2 for this fix?
3_1_2_b17

Diff is attached.

Comment by Anissa Lam [ 11/Jan/12 ]

Fixed in 3.1.2 branch.

Log Message:
------------
GLASSFISH-18160. Changes related to View Log Raw buttons.

  • button is enabled only when instance is running; add button to cluster instance; separate raw log and formated log window.

Revisions:
----------
51998

Modified Paths:
---------------
branches/3.1.2/admingui/cluster/src/main/resources/standalone/standaloneInstanceGeneral.jsf
branches/3.1.2/admingui/cluster/src/main/resources/cluster/clusterInstanceEdit.jsf
branches/3.1.2/admingui/common/src/main/resources/appServer/serverInstGeneralPe.jsf

Comment by lidiam [ 11/Jan/12 ]

Verified in build ogs-3.1.2-b17.zip.





[GLASSFISH-18158] Intermittent: long running process message never goes away Created: 09/Jan/12  Updated: 17/Jan/12  Resolved: 15/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b18

Type: Bug Priority: Critical
Reporter: lidiam Assignee: Jason Lee
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

build ogs-3.1.2-b17-01_09_2012.zip on windows xp, chrome


Attachments: File .instancestate     JPEG File firebug-errors.JPG     JPEG File long-running-process.JPG     Text File server.log.txt    
Issue Links:
Duplicate
is duplicated by GLASSFISH-18201 [NLS]A long-running process occurs af... Closed
Tags: 312_qa, 312_regression, 312_verified, 3_1_2-approved

 Description   

This issues has been happening for a few days now, though still inconsistent: a long running process message pops up out of the blue and does not go away. Possible steps to reproduce:

1. Start (or restart) domain1.
2. Go to Admin Console and login.
3. Click on server node, then some other node.

The message usually pops up on the server node. The sever.log file ends with the following while the message is displayed:

[#|2012-01-09T15:04:51.750-0800|INFO|glassfish3.1.2|org.glassfish.admingui|_Thre
adID=19;_ThreadName=Thread-2;|Admin Console: Initializing Session Attributes...|
#]

[#|2012-01-09T15:05:08.750-0800|INFO|glassfish3.1.2|null|_ThreadID=20;_ThreadNam
e=Thread-2;|unable to read instance state file C:\as\glassfish3\glassfish\domain
s\domain1\config\.instancestate, recreating|#]

Reloading the page clears this problem. Attaching screenshot, server.log and .instancestate files.



 Comments   
Comment by Anissa Lam [ 09/Jan/12 ]

When console is waiting for REST request to come back, and it takes a long time, then this pops up, thats the design.
Since the log is from backend, transfer to 'admin' for evaluation and explains what its trying to do.

Comment by Anissa Lam [ 10/Jan/12 ]

Lidia, if you click the 'x' in the corner of that popup, do you notice any issue on console regarding that page ?

Comment by lidiam [ 10/Jan/12 ]

It just happened again, and if I click 'X' the popup goes away and I can use Admin Console as usual.

Comment by Anissa Lam [ 10/Jan/12 ]

Jason mentioned that he may 'know' something. transfer back to Jason to take a look.

Comment by Jason Lee [ 10/Jan/12 ]

All I "know" is that I saw something similar on another page, made some JS tweaks (which have been committed), and it went away. I don't know anything about this particular issue, nor can I reproduce it.

Comment by lidiam [ 10/Jan/12 ]

I see this happening on localhost as well as on a remote DAS. I just had it happen again on localhost on windows (again on server node) after Console was idle for an hour or so. This time server.log ends with:

[#|2012-01-09T19:00:50.218-0800|INFO|glassfish3.1.2|org.glassfish.admingui|_Thre
adID=25;_ThreadName=Thread-2;|Redirecting to /common/index.jsf|#]

[#|2012-01-09T19:00:50.250-0800|INFO|glassfish3.1.2|org.glassfish.admingui|_Thre
adID=21;_ThreadName=Thread-2;|Admin Console: Initializing Session Attributes...|

Thus there is nothing in the server.log to point to the cause of this issue.

I also got the same problem accessing a remote DAS on solaris, but this time on the Clusters node. That server.log also ends with:

[#|2012-01-09T18:13:16.082-0800|INFO|glassfish3.1.2|javax.enterprise.system.cont
ainer.ejb.com.sun.ejb.containers|_ThreadID=24;_ThreadName=Thread-2;|Created EjbT
hreadPoolExecutor with thread-core-pool-size 16 thread-max-pool-size 32 thread-k
eep-alive-seconds 60 thread-queue-capacity 2147483647 allow-core-thread-timeout
false |#]

[#|2012-01-09T18:13:21.002-0800|INFO|glassfish3.1.2|com.sun.jersey.server.impl.a
pplication.WebApplicationImpl|_ThreadID=24;_ThreadName=Thread-2;|Initiating Jers
ey application, version 'Jersey: 1.11 12/09/2011 10:27 AM'|#]

[#|2012-01-09T18:13:24.109-0800|INFO|glassfish3.1.2|javax.enterprise.system.tool
s.admin.org.glassfish.admin.rest.adapter|_ThreadID=24;_ThreadName=Thread-2;|REST
00001: Listening to REST requests at context: /management/domain|#]

[#|2012-01-09T18:13:24.201-0800|INFO|glassfish3.1.2|org.glassfish.admingui|_Thre
adID=21;_ThreadName=Thread-2;|Redirecting to /j_security_check|#]

[#|2012-01-09T18:13:25.467-0800|INFO|glassfish3.1.2|org.glassfish.admingui|_Thre
adID=17;_ThreadName=Thread-2;|Admin Console: Initializing Session Attributes...|
#]

Comment by Jason Lee [ 10/Jan/12 ]

We have not been able to reproduce this on a development machine. Given the late hour, I'm going to lower this to a P4 and look at it in any future 3.x release.

Comment by lidiam [ 11/Jan/12 ]

This issue seems to happen every time I stop and start domain. I got it again in Firefox but there were no errors in firebug at first. After a couple minutes a whole bunch were printed - screenshot attached. The long running process message is still displayed, however, so not sure these errors have anything to do with the problem.

Comment by Anissa Lam [ 12/Jan/12 ]

We need to address this for 3.1.2, bump this to P2.

I am able to reproduce this consistently by:

  • start domain, launch console
  • go to Domain -> Admin password tab, add password and Save
  • Save Successful msg shows up, and then the "Long Running process" popup and never goes away.

I told Jason about this, and tested a patch that he provides to me. With that patch, I can't reproduce this.

Comment by Jason Lee [ 12/Jan/12 ]

We finally have a reproducible test case, so here's what I found in:

On the Admin Password tab, you enter the new password, then click save. The JS starts the timer to display the panel, then submits the ajax request. The response comes back really quickly, and the timer is turned off. Next, either by explicit logic or a after-so-many-ajax-requests counter, reloadHeaderFrame() is called, which fires off an ajax request.  This starts the timer to display the panel.  The callback function for the ajax request, though, does NOT clear the timer, so the panel is displayed, but there's nothing else happening on the page that will remove the panel, so it sits there. I modified reloadHeaderFrame() to tell the ajax…engine NOT to display the header (we really don't need that for this asynchronous, system-requested updated), so the problem goes away.

Here's the diff:

Index: common/src/main/resources/js/adminjsf.js
===================================================================
--- common/src/main/resources/js/adminjsf.js    (revision 52065)
+++ common/src/main/resources/js/adminjsf.js    (working copy)
@@ -1855,9 +1855,7 @@
 
 function reloadHeaderFrame() {
     var mastheadForm = document.getElementById('af');
-    admingui.ajax.postAjaxRequest(mastheadForm, {
-        render: 'af'
-    }, 'af');
+    admingui.ajax.postAjaxRequest(mastheadForm, { render: 'af' }, 'af', false);
 }
 
 admingui.deploy = {
@@ -2243,8 +2241,10 @@
         admingui.nav.selectTreeNodeWithURL(o.argument.url);
     },
 
-    postAjaxRequest : function (component, args, respTarget) {
-        admingui.ajax.ajaxStart();
+    postAjaxRequest : function (component, args, respTarget, displayLoading) {
+        if (displayLoading !== false) {
+            admingui.ajax.ajaxStart();
+        }
         if ((respTarget === null) || (typeof(respTarget) === 'undefined')) {
             respTarget = 'content';
         }
Comment by Jason Lee [ 12/Jan/12 ]
  • What is the impact on the customer of the bug?

The user may or may not see this bug. In situations like Anissa described, it happens pretty much every time. In those like the reporter described, it's intermittent. When it DOES show up, though, it interrupts the workflow, and probably confuses the user, resulting in a very poor and easily avoidable user experience.

  • What is the cost/risk of fixing the bug?

Very simple fix. Took less than minutes to fix and test once the test case was identified.

  • Is there an impact on documentation or message strings?

None.

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

Console dev tests should be sufficient.

  • Which is the targeted build of 3.1.2 for this fix?

b18

Comment by Jason Lee [ 12/Jan/12 ]

Fix committed (r52070).

Comment by lidiam [ 15/Jan/12 ]

This is still happening in build ogs-3.1.2-b18-01_13_2012.zip. After every restart of the DAS and if Admin Console was not used for a long time.

Comment by lidiam [ 15/Jan/12 ]

I'm running glassfish winxp lab machine, secure admin is not enabled. It consistently happens after each DAS restart. It always happens after about 15 - 20 seconds of login to Admin Console, e.g.:

1. Stop and start DAS from command line.
2. Login to Admin Console.
3. Count to 5 and click on server node.
4. Count to 5 and click on Domain node.
5. Count to 5 and click on Clusters node.
6. Count to 5 and the long running popup shows up.

I tried with different nodes and the same thing happens. I tried with login to Admin Console, waiting about 25 seconds and then clicking on a node. Within a second or two the popup shows. I restarted DAS, logged in and waited 18 seconds. Then clicked Standalone Instances, again with second or two I got the popup. I stopped/started DAS, logged in and waited 12 seconds. I clicked on Clusters, waited about 15 seconds and clicked on HTTP Load Balancers. Got the popup within a second again. I did one more test using Restart button on server page. Logged in and waited 22 seconds. Clicked on clusters and within 2 seconds got the popup.

Comment by Jason Lee [ 15/Jan/12 ]

Reclosing this issue. While debugging on the test machine provided, I was able to reproduce the issue given the instructions provided, but I noticed that the JS was out of date. After I cleared the browser cache and restarted the server, I was no longer able to reproduce the issue.

Comment by lidiam [ 17/Jan/12 ]

Verified on build ogs-3.1.2-b18-01_13_2012.zip.





[GLASSFISH-18156] Load defaults in Edit Health checker of LB Config removes URL value from '/' to '' Created: 09/Jan/12  Updated: 10/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: arunkumar_s Assignee: srinik76
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL


Tags: 312_qa

 Description   

Steps:

1) Login into glassfish and create a LB config with some target.
2) Open the LB and Edit Health checker under target tab of the cluster/instance
3) Click Load Defaults

The Default value of the URL field in Health checker section changes from '/' to ''



 Comments   
Comment by Anissa Lam [ 09/Jan/12 ]

There is no default value for the url attribute in HealthChecker config.
However, the console convention is that if there is no default value, that field should stay the same, not being emptied out.
So, GUI code should fix that.

However, if you think this field should have "/" as default value, you need to file a bug against backend.

Comment by Anissa Lam [ 10/Jan/12 ]

Given the fact that there is really no default value for this attribute, and the fact that there is only 3 attributes show on the screen, that user will very easily notice this change, I am lowering this to P4.
Unless Srini can find an easy fix, get this reviewed and approved by Joe before the HCF tonight, we will look into this in next v3 release.





[GLASSFISH-18153] asadmin commands link not working in help doc Created: 09/Jan/12  Updated: 09/Jan/12  Resolved: 09/Jan/12

Status: Closed
Project: glassfish
Component/s: docs
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arunkumar_s Assignee: Paul Davies
Resolution: Works as designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL


Tags: 312_qa

 Description   

Steps:

1) Login Admin GUI
2) Click any Topic (eg.Clusters) and click help
3) Click any one of the equivalent asadmin sub commands in the right pane

Link pointing to oracle.com shows 404 message in the browser.



 Comments   
Comment by Paul Davies [ 09/Jan/12 ]

These links will return error 404 until the targets of the links are published. The targets of the links will be published only when GlassFish 3.1.2 is released and not before then.





[GLASSFISH-18148] Unable to access AdminConsole on Win2008 with JDK1.7.0_02 - works with JDK1.6.0_30 Created: 07/Jan/12  Updated: 08/Jan/12  Resolved: 08/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17

Type: Bug Priority: Critical
Reporter: Alex Pineda Assignee: Anissa Lam
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 2008 R2 system with 64bit OS. GF 3.1.2 build16. JDK1.7.0_02. Firefox browser 3.6.25. Default GF installation


Attachments: Text File server.log    
Issue Links:
Duplicate
duplicates GLASSFISH-18137 Exception thrown when running on jdk ... Resolved
duplicates GLASSFISH-18133 JDK 1.7.0_02 - Nodes/instances creati... Resolved
is duplicated by GLASSFISH-18125 Cannot access admin console on clean ... Resolved
Tags: 312_qa

 Description   

This scenario works with JDK 1.6.0_30 without issues.

The issue is seen after installing GF3.1.2 build16 with JDK1.7.0_02 and attempting to get into the AdminConsole. Specific to http://localhost:4848. The installation is done with all default values (ports and no password), but when to attempting to login, the Admin Console username/password login screen is displayed and since no password was set, unable to login. In the server log the following is reported:

[#|2012-01-07T09:57:55.795-0800|SEVERE|glassfish3.1.2|org.glassfish.admin.rest.LazyJerseyInit|_ThreadID=72;_ThreadName=Thread-2;|java.lang.ClassNotFoundException: org.glassfish.admin.rest.resources.generatedASM.DomainResource not found by org.glassfish.main.admin.rest-service [221]
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDele
gation(BundleWiringImpl.java:1460)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringIm
pl.java:72)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadCla
ss(BundleWiringImpl.java:1843)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:186)
at org.glassfish.admin.rest.LazyJerseyInit.getResourcesConfigForManageme
nt(LazyJerseyInit.java:259)
at org.glassfish.admin.rest.adapter.RestManagementAdapter.getResourcesCo
nfig (RestManagementAdapter.java:62)
at org.glassfish.admin.rest.adapter.RestAdapter.exposeContext(RestAdapte
r.java:340)
at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java
:148)
at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java
:179)
at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java
:117)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(Container
Mapper.java:238)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:8
49)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFil
ter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultPro
tocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.jav
a:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.jav
a:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java
:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextT
ask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.
java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadP
ool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool
.java:513)
at java.lang.Thread.run(Thread.java:722) |#]

Attaching the server log as to provide more detail.



 Comments   
Comment by Anissa Lam [ 08/Jan/12 ]

We are seeing issues with JDK7u2. I filed GLASSFISH-18137 which is closed a duplicate of 18133.
Tom fixed 18133 on 1/6, and with that fix, Mitesh confirmed that the issues go away for him on his windows system. (Before the fix, he reproduced that consistently).
exception shown in the server.log is the same like the other two. I am marking this as duplicate.
Please try 1/7 nightly build, you can reopen if problem persists.





[GLASSFISH-18136] Blocking: RuntimeException modifying EJB Timer Service Created: 06/Jan/12  Updated: 11/Jan/12  Resolved: 06/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17

Type: Bug Priority: Critical
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Attachments: JPEG File runtime-exception.JPG     Text File server.log.txt    
Issue Links:
Duplicate
duplicates GLASSFISH-18107 error message appearing after 'save' ... Resolved
Tags: 312_blocking, 312_qa, 312_regression, 312_verified

 Description   

Saving of EJB Timer Service page causes RuntimeException on the screen. Steps to reproduce:

1. Create a cluster.
2. Go to cluster configuration, EJB Container -> EJB Timer Settings.
3. Hit Save, no need to enter anything, and the RuntimeException is displayed. DAS server.log contains the following:

[#|2012-01-05T16:52:57.096-0800|WARNING|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=112;_ThreadName=Thread-2;|java.lang.RuntimeException while attempting to process a 'command' event for 'saveButton'.
java.lang.RuntimeException: java.lang.RuntimeException while attempting to process a 'command' event for 'saveButton'.
...
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
at org.glassfish.admingui.common.util.RestUtil.parseResponse(RestUtil.java:310)

This is a regression and it is blocking testing of listing ejb timers.



 Comments   
Comment by Anissa Lam [ 06/Jan/12 ]

This is dup. of GLASSFISH-18107.

Comment by lidiam [ 11/Jan/12 ]

Verified in build ogs-3.1.2-b17.zip.





[GLASSFISH-18135] Blocking: Log not displayed in raw log viewer Created: 06/Jan/12  Updated: 11/Jan/12  Resolved: 09/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17, 4.0

Type: Bug Priority: Critical
Reporter: lidiam Assignee: andriy.zhdanov
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip, DAS on solaris, console on WinXP, Chrome.


Attachments: JPEG File rawlogempty.JPG     File working.diff    
Tags: 312_blocking, 312_qa, 312_regression, 312_verified, 3_1_2-approved

 Description   

Raw Log Viewer does not display server.log. Steps to reproduce:

1. In Admin Console go to server node.
2. Click on Raw Log Viewer button - log is not displayed in the newly opened window.

This happens for DAS or any instance. It is a regression and it's blocking from further testing raw log viewer. I tried in both Chrome and Firefox with the same result.



 Comments   
Comment by Anissa Lam [ 06/Jan/12 ]

Here is what i see.
server installed on host1, enable secure admin.

If you launch gui using localhost:4848, then it will ask for admin username/password.
If you luanch gui using hostname:4848, then it won't ask for admin username/password, but didn't show anything.
Are you sure this is a regression ? If so, which build you see this working, that will help.

Comment by Anissa Lam [ 06/Jan/12 ]

accidentally mark fixed. reopen so Andriy can take a look.

Comment by andriy.zhdanov [ 09/Jan/12 ]
  • What is the impact on the customer of the bug?

Minimal.

  • What is the cost/risk of fixing the bug?

No risk.

  • Is there an impact on documentation or message strings?

No

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

View Raw Log

  • Which is the targeted build of 3.1.2 for this fix?

3_1_2_b17

Comment by andriy.zhdanov [ 09/Jan/12 ]

Committed revision 51981
Committed revision 51982

Comment by lidiam [ 11/Jan/12 ]

Verified in build ogs-3.1.2-b17.zip





[GLASSFISH-18130] Test Connection in LB Config fails with 404 error message Created: 05/Jan/12  Updated: 09/Jan/12  Resolved: 09/Jan/12

Status: Closed
Project: glassfish
Component/s: load_balancer
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arunkumar_s Assignee: kshitiz_saxena
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL 5 , OGS 3.1.2 b16(zip) , Glassfish_lb configurator 3.1.2 b01 , Oracle iplanet 7.0 u9+


Tags: 312_qa

 Description   

Steps:

1) Login to glassfish and create a LB
2) Click test connection

Error message

Error An error has occurred
The web server which hosts the load balancer returned response "404 Not found" to the apply change request URL https://ejp-172x-174.in.oracle.com:443/lbconfigupdate. Please check the configuration and the SSL certificates.The web server which hosts the load balancer returned response "404 Not found" to the apply change request URL https://ejp-172x-174.in.oracle.com:443/lbconfigupdate. Please check the configuration and the SSL certificates.

PS: Configured the lb configurator using the steps in Glassfish 17304



 Comments   
Comment by Anissa Lam [ 05/Jan/12 ]

So, what exactly is the problem here ? Are you saying the URL generated to test the connection is not correct ?
Sounds like the web server is responding with 404, and GUI display the error correctly.

Since the error is from backend, transfer for initial evaluation.

Comment by Anissa Lam [ 07/Jan/12 ]

Sounds like GUI is showing whatever error the backend returns.
Transfer to load-balancer for initial evaluation.

Comment by arunkumar_s [ 09/Jan/12 ]

Looks URL is correct, but couldn't able to test connection.

Comment by arunkumar_s [ 09/Jan/12 ]

Not reproducible.

Comment by kshitiz_saxena [ 09/Jan/12 ]

This issue is not reproducible.





[GLASSFISH-18129] Modifying values in LB Config throws Java Runtime Exception Created: 05/Jan/12  Updated: 05/Jan/12  Resolved: 05/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: arunkumar_s Assignee: Anissa Lam
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OEL 5 , OGS 3.1.2 b16 (zip)


Issue Links:
Duplicate
duplicates GLASSFISH-18107 error message appearing after 'save' ... Resolved
Tags: 312_qa

 Description   

Steps to reproduce

1) Open glassfish and create a LB
2) Open the Load balancer and modify device values and click save

Issue --> java.lang.RuntimeException thrown

Server log shows:

[#|2012-01-05T18:17:14.685+0530|WARNING|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.lifecycle|_ThreadID=25;_ThreadName=Thread-2;|java.lang.reflect.InvocationTargetException while attempting to process a 'command' event for 'saveButton'.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException while attempting to process a 'command' event for 'saveButton'.
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:422)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:394)
at com.sun.jsftemplating.layout.event.CommandActionListener.invokeCommandHandlers(CommandActionListener.java:150)
at com.sun.jsftemplating.layout.event.CommandActionListener.processAction(CommandActionListener.java:98)
at javax.faces.event.ActionEvent.processListener(ActionEvent.java:88)
at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:769)
at javax.faces.component.UICommand.broadcast(UICommand.java:300)
at com.sun.webui.jsf.component.WebuiCommand.broadcast(WebuiCommand.java:166)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:794)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1259)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:593)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:232)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jsftemplating.layout.descriptors.handler.Handler.invoke(Handler.java:442)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHandlers(LayoutElementBase.java:420)
... 41 more
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Map
at org.glassfish.admingui.common.util.RestUtil.parseResponse(RestUtil.java:310)
at org.glassfish.admingui.common.util.RestUtil.restRequest(RestUtil.java:218)
at org.glassfish.admingui.common.handlers.RestApiHandlers.restRequest(RestApiHandlers.java:221)
... 46 more



 Comments   
Comment by Anissa Lam [ 05/Jan/12 ]

This has been fixed since 1/4/2012 nightly build.





[GLASSFISH-18124] Cannot start cluster - synchronization fails Created: 05/Jan/12  Updated: 09/Jan/12  Resolved: 09/Jan/12

Status: Resolved
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: Tom Mueller
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip on solaris


Attachments: Text File server.log.txt     JPEG File synchronization-error.JPG    
Tags: 312_gui_new, 312_qa

 Description   

I have a cluster with two instances: one on localhost, another one on a remote solaris machine on an SSH node. This cluster was running in the past. I can no longer start the cluster with the following error:

cll2: Could not start instance cll2 on node localhost-domain1 (localhost). Command failed on node localhost-domain1 (localhost): Previous synchronization failed at Jan 4, 2012 5:28:26 PM Will perform full synchronization. Removing all cached state for instance cll2. Command start-local-instance failed. CLI802 Synchronization failed for directory config, caused by: remote failure: Command timed out. Unable to acquire a lock to access the domain. Another command acquired exclusive access to the domain on Wed, 04 Jan 2012 17:28:56 PST. Retry the command at a later time. To complete this operation run the following command locally on host localhost from the GlassFish install location /export/home/j2eetest/3.1.2/glassfish3: bin/asadmin start-local-instance --node localhost-domain1 --sync normal cll2 clt1: Could not start instance clt1 on node tuppy (tuppy). Command failed on node tuppy (tuppy): Previous synchronization failed at Jan 4, 2012 6:04:59 PM Will perform full synchroni .... msg.seeServerLog

I'will attach screenshot and server.log. I am not certain what is causing this issue but I was testing Download Logs in Admin Console before hitting this issue. I'm going to leave the machine intact, in case anyone wants to poke around. This issue may not be easily reproducible.



 Comments   
Comment by lidiam [ 05/Jan/12 ]

I had another cluster with two instances on a DCOM node. After removing this cluster, I could start the solaris/ssh cluster without problems. This seems to have been caused by issue http://java.net/jira/browse/GLASSFISH-18123 - trying to download log files for a server instance on a DCOM node.

Comment by Anissa Lam [ 05/Jan/12 ]

Console is showing whatever backend gives back.
Transfer to backend for evaluation.

Comment by lidiam [ 05/Jan/12 ]

I tried several times, but cannot reproduce it. I see messages like:

[#|2012-01-04T21:34:23.865-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=286;_ThreadName=Thread-2;|Warning: Synchronization with DAS failed, continuing startup...

but cluster starts fine after that.

Comment by Byron Nevins [ 09/Jan/12 ]

This bug seems to be related to domain-locking and synchronization.

Comment by Tom Mueller [ 09/Jan/12 ]

The submitter of this issue reported that this issue cannot be reproduced.
Marking this as "Cannot Reproduce".

If this issue shows up again, please reopen the issue.





[GLASSFISH-18123] Cannot download logs for a server instance on DCOM node Created: 05/Jan/12  Updated: 09/Feb/12  Resolved: 09/Feb/12

Status: Closed
Project: glassfish
Component/s: logging
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b21

Type: Bug Priority: Major
Reporter: lidiam Assignee: naman_mehta
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File server.log.txt    
Tags: 312_gui_new, 312_qa, 312_verified

 Description   

This may be related to issue 18055. Steps to reproduce:

1. Create a standalone instance or a cluster with one instance and start it.
2. Go to Domain Logs page and select the newly created instance/cluster.
3. Click Download Logs - Admin Console waits for DAS over 10 minutes and then nothing happens on the screen but the following exception is printed in DAS server.log:

[#|2012-01-04T17:08:38.649-0800|WARNING|glassfish3.1.2|javax.enterprise.system.c
ontainer.web.com.sun.enterprise.web|_ThreadID=165;_ThreadName=Thread-2;|Standard
WrapperValve[DownloadServlet]: PWC1406: Servlet.service() for servlet DownloadSe
rvlet threw exception
java.lang.RuntimeException: com.sun.jersey.api.client.ClientHandlerException: ja
va.io.InterruptedIOException: Operation interrupted
at org.glassfish.admingui.common.servlet.LogFilesContentSource.getInputS
tream(LogFilesContentSource.java:110)
at org.glassfish.admingui.common.servlet.DownloadServlet.writeContent(Do
wnloadServlet.java:277)
at org.glassfish.admingui.common.servlet.DownloadServlet.doPost(Download
Servlet.java:174)
at org.glassfish.admingui.common.servlet.DownloadServlet.doGet(DownloadS
ervlet.java:155)

I'm attaching server.log.



 Comments   
Comment by Anissa Lam [ 05/Jan/12 ]

Transfer to logging for initial evaluation.

Comment by naman_mehta [ 05/Jan/12 ]

This issue has dependency on 18055. After fixing 18055, it's not reproducible.

It just prints 'No items found, No record found' in GUI. It won't hang now. Also it will print exception in server.log but it's intended to get exact error for debug in future.

Comment by lidiam [ 09/Feb/12 ]

Reopening to close as fixed. This issue is certainly reproducible on 3.1.2_b16.

Comment by lidiam [ 09/Feb/12 ]

This issue is fixed.

Comment by lidiam [ 09/Feb/12 ]

Verified in build ogs-3.1.2-b21.zip.





[GLASSFISH-18122] SSH/DCOM connection credentials stored in clear text in domain.xml Created: 05/Jan/12  Updated: 05/Jan/12  Resolved: 05/Jan/12

Status: Resolved
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: Tom Mueller
Resolution: Works as designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Tags: 312_qa, 3_1_2-review

 Description   

Currently when we create an SSH or DCOM node, connection credentials are stored in domain.xml as follows:

<node node-host="localhost" name="localhost-domain1" type="CONFIG" install-dir="$

{com.sun.aas.productRoot}

"></node>
<node node-host="jed-asqe-43" name="jedy" windows-domain="jed-asqe-43" type="DCOM" install-dir="C:\as\dcomtest\glassfish3">
<ssh-connector ssh-port="135">
<ssh-auth user-name="usernameincleartext" password="passwordincleartext"></ssh-auth>
</ssh-connector>
</node>

While on unix systems domain.xml is protected by file permissions, it is not so on windows. We should not be storing machine connection credentials in clear text.



 Comments   
Comment by Tom Mueller [ 05/Jan/12 ]

SSH and DCOM passwords in domain.xml can use the password alias mechanism to hide passwords.
So this concern is already dealt with.

Marking this as "works as designed".





[GLASSFISH-18121] Intermittent: cannot start instance on a remote node: server requires a password Created: 05/Jan/12  Updated: 10/Jan/12

Status: Open
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Major
Reporter: lidiam Assignee: lidiam
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip, DAS on solaris, node on WinXP


Attachments: Text File server.log     Text File server.log.clusteredinstance.txt    
Tags: 312_gui_new, 312_qa, 3_1_2-exclude

 Description   

This is an intermittent issue but it happened twice already. I create a standalone instance on a DCOM node and cannot start it with the following in the server.log of the instance:

[#|2012-01-04T15:44:32.265-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.to
ols.admin.com.sun.enterprise.container.common|_ThreadID=10;_ThreadName=Thread-2;

The server requires a valid admin password to be set before it can start. Pleas
e set a password using the change-admin-password command.
#]

I have a cluster with two instances on the same node running fine. I also created another standalone instance and it started fine. I'm accessing Admin Console on solaris from the windows box, so secure admin and domain password are both set. I'll attach server.log.



 Comments   
Comment by lidiam [ 05/Jan/12 ]

I just got this error for the 3rd time but this time with an SSH node. I had a cluster with two instances, one on localhost one on an ssh node (solaris). The cluster was running fine. I stopped it and added another instance on the ssh node. When I tried to start cluster the newly added server instance failed to start with the following error in Admin Console:

Command succeeded with Warning
clt2: Could not start instance clt2 on node tuppy (tuppy). Command failed on node tuppy (tuppy): Warning: Synchronization with DAS failed, continuing startup... Waiting for clt2 to start .....................................Command start-local-instance failed. Error starting instance clt2. The server exited prematurely with exit code 0. Before it died, it produced the following output: Launching GlassFish on Felix platform Jan 4, 2012 10:31:54 PM com.sun.common.util.logging.LoggingConfigImpl copyLoggingPropertiesFile WARNING: Logging.properties file not found, creating new file using DAS logging.properties [#|2012-01-04T22:31:55.037-0800|INFO|glassfish3.1.2|com.sun.enterprise.server.logging.GFFileHandler|_ThreadID=1;_ThreadName=main;|Running GlassFish Version: GlassFish Server Open Source Edition 3.1.2-b16 (build 16)|#] [#|2012-01-04T22:31:57.647-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=10;_ThreadName=main;|JTS5014: Reco .... msg.seeServerLog

Instance's server.log contained the same error and exception:

[#|2012-01-04T22:31:59.029-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.to
ols.admin.com.sun.enterprise.container.common|_ThreadID=10;_ThreadName=Thread-2;

The server requires a valid admin password to be set before it can start. Pleas
e set a password using the change-admin-password command.
#]

[#|2012-01-04T22:31:59.031-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.co
re.com.sun.enterprise.v3.services.impl|_ThreadID=10;_ThreadName=Thread-2;|Unable
to start v3. Closing all ports
org.jvnet.hk2.component.ComponentException: injection failed on com.sun.enterpri
se.v3.admin.AdminAdapter.authenticator with interface org.glassfish.internal.api
.AdminAccessController
at org.jvnet.hk2.component.InjectionManager.error_injectionException(Inj
ectionManager.java:284)
at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java
:165)
at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java
:93)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.
java:126)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreato
r.java:91)

Comment by lidiam [ 05/Jan/12 ]

Attaching log file for clustered instance on an ssh node that fails to start.

Comment by Byron Nevins [ 09/Jan/12 ]

This will take more time to hunt-down than is available before HCF for 3.1.2.

Comment by Byron Nevins [ 10/Jan/12 ]

I can't reproduce this. Please do the following. We need to make sure it's a Dcom issue.

When it happens again simply run start-local-instance directly on the remote machine. Use the --verbose option

What does it say?





[GLASSFISH-18119] Tree highliting never removed from Domain after accessing Domain Logs Created: 04/Jan/12  Updated: 10/Jan/12

Status: Open
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: sumasri
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip, DAS on solaris


Attachments: JPEG File highlight.JPG    
Tags: 312_gui_new, 312_qa

 Description   

Steps to reproduce:

1. Go to Admin Console and click on Domain
2. Click on any other node - tree highlight moves to the new node.
3. Click again on Domain and then on Domain Logs tab, and WAIT till page finishes loading completely (no hour glass at all).
4. Now click on any other tree node - the highlight is now in two places: on the newly selected node and still on Domain (will attach screenshot).

This can be "reset" by clicking on Domain node again, e.g. click on Domain node and then other node - now highlight from Domain is removed.



 Comments   
Comment by Anissa Lam [ 04/Jan/12 ]

request help from Suma.

Comment by sumasri [ 05/Jan/12 ]

Not able to reproduce the issue with 3.1.2 workspace.
Lidia, I would like you to test it with the fresh installation and please let us know the result.

Comment by lidiam [ 05/Jan/12 ]

I have just reinstalled b16 on solaris (ogs zip) and see the same issue.

Comment by Anissa Lam [ 10/Jan/12 ]

This is browser specific and timing related also.

I see the issue consistently on
Chrome 14.0.835.202 on Mac.
Safari 5 on Mac

intermittently on
Chrome 16 on Window XP
Safari 4 on Window XP

However, the following works fine, cannot reproduce the problem.
Firefox 7 on my Mac
Mozilla Firefox 7 on Solaris
IE 8 on Windows XP
Firefox 6 on Windows XP

So, it seems this happens only on Chrome and Safari more often.

Lidia what browser are you using ?

Given the fact that

  • this isn't causing any functionality issue,
  • can easily be fixed, just go to Common Task page, or click Home Button, or refresh browser,
  • no data lost
  • only appear in some browser

Plus the fact that

  • HCF is tonight and this involves the ajax tree code which is of high risk when changed,

I am lowering this to a P4.
We may look into this if there is followup v3 release after 3.1.2.

Comment by lidiam [ 10/Jan/12 ]

I'm testing on Chrome on Windows XP. It happens every time I go to Domain Logs.





[GLASSFISH-18114] Collecting log files for a cluster also collects DAS files Created: 04/Jan/12  Updated: 21/Jan/12  Resolved: 21/Jan/12

Status: Closed
Project: glassfish
Component/s: docs
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17

Type: Bug Priority: Minor
Reporter: lidiam Assignee: Paul Davies
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Tags: 312_gui_new, 312_qa

 Description   

Steps to reproduce:

1. Create a cluster with one instance and start it.
2. Go to Domain Logs page, select cluster and click on Collect Logs.
3. Open up the zip file and you will see all the DAS logs under "server" directory.

Only logs for cluster instances should be present in the zip file.



 Comments   
Comment by Anissa Lam [ 04/Jan/12 ]

I tried this using CLI, and get the DAS log file as well. Transfer to logging for evaluation.

Comment by naman_mehta [ 04/Jan/12 ]

It's not an issue. It's feature. User wants DAS log with cluster log also. Is there any issue, let me know else I will close this bug.

Comment by Anissa Lam [ 04/Jan/12 ]

So, regardless what the target is, the DAS server.log will be included ?
Not an issue for me. If thats how it suppose to be, you can close that as work as designed.

Comment by naman_mehta [ 04/Jan/12 ]

It's feature not an issue. server.log is always present when you collect cluster logs. It's designed in this way.

Comment by lidiam [ 05/Jan/12 ]

This is inconsistent, since downloading standalone instance server logs does not download DAS logs. Anyhow, it would be good to put this information in Admin Console on the Download Logs page that DAS logs will be downloaded when downloading cluster logs. It is not intuitive.

Comment by lidiam [ 05/Jan/12 ]

We should include information that DAS logs are downloaded with cluster logs on the Admin Console Download Logs page.

Comment by Paul Davies [ 10/Jan/12 ]

Fix committed in revision 51972

Comment by lidiam [ 20/Jan/12 ]

Tested in build ogs-3.1.2-b18.zip and the following is printed in online help:

-------
Note:

The ZIP archive also includes the log file for the domain administration server (DAS), regardless of whether you specify an instance or a cluster.
-------

The above is incorrect. DAS log files are only downloaded with cluster logs, but not standalone instance.

Comment by Paul Davies [ 21/Jan/12 ]

This issue is already captured in GLASSFISH-18181. I am closing this issue as a duplicate.





[GLASSFISH-18113] IOException in server.log when selecting server instance for collecting logs Created: 04/Jan/12  Updated: 20/Mar/13  Resolved: 20/Mar/13

Status: Resolved
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: lidiam Assignee: andriy.zhdanov
Resolution: Won't Fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Attachments: Text File server.log.txt    
Tags: 312_gui_new, 312_qa, 3_1_2-exclude

 Description   

Steps to reproduce:

1. Create a standalone instance and start it.
2. Go to Domain -> Domain Logs page and as soon as it displays select the newly created instance in the drop down box. The following exception is printed to the DAS server.log when the hour glass on the page disappears:

[#|2012-01-03T15:50:08.681-0800|INFO|glassfish3.1.2|javax.enterprise.resource.webcontainer.jsf.context|_ThreadID=80;_ThreadName=Thread-2;|Exception when handling error trying to reset the response.
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:430)
at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:356)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:455)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:442)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:160)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
at com.sun.faces.renderkit.html_basic.HtmlResponseWriter.flushAttributes(HtmlResponseWriter.java:1093)
at com.sun.faces.renderkit.html_basic.HtmlResponseWriter.closeStartIfNecessary(HtmlResponseWriter.java:1043)
at com.sun.faces.renderkit.html_basic.HtmlResponseWriter.startElement(HtmlResponseWriter.java:599)
at com.sun.webui.jsf.renderkit.html.ImageRenderer.renderStart(ImageRenderer.java:86)
at com.sun.webui.jsf.renderkit.html.AbstractRenderer.encodeBegin(AbstractRenderer.java:138)
at javax.faces.component.UIComponentBase.encodeBegin(UIComponentBase.java:820)
at com.sun.webui.jsf.util.RenderingUtilities.renderComponent(RenderingUtilities.java:89)
at com.sun.webui.jsf.renderkit.html.ImageHyperlinkRenderer.finishRenderAttributes(ImageHyperlinkRenderer.java:85)
at com.sun.webui.jsf.renderkit.html.HyperlinkRenderer.renderLink(HyperlinkRenderer.java:320)
at com.sun.webui.jsf.renderkit.html.HyperlinkRenderer.renderEnd(HyperlinkRenderer.java:184)
at com.sun.webui.jsf.renderkit.html.AbstractRenderer.encodeEnd(AbstractRenderer.java:225)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at com.sun.webui.jsf.util.RenderingUtilities.renderComponent(RenderingUtilities.java:99)
at com.sun.webui.jsf.renderkit.html.TreeNodeRenderer.renderTreeRow(TreeNodeRenderer.java:262)
at com.sun.webui.jsf.renderkit.html.TreeNodeRenderer.encodeEnd(TreeNodeRenderer.java:177)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at com.sun.webui.jsf.util.RenderingUtilities.renderComponent(RenderingUtilities.java:99)
at com.sun.webui.jsf.renderkit.html.TreeNodeRenderer.encodeEnd(TreeNodeRenderer.java:206)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at com.sun.webui.jsf.util.RenderingUtilities.renderComponent(RenderingUtilities.java:99)
at com.sun.webui.jsf.renderkit.html.TreeNodeRenderer.encodeEnd(TreeNodeRenderer.java:206)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at com.sun.webui.jsf.util.RenderingUtilities.renderComponent(RenderingUtilities.java:99)
at com.sun.webui.jsf.renderkit.html.TreeRenderer.encodeEnd(TreeRenderer.java:196)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encodeChild(LayoutElementBase.java:558)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encodeChild(LayoutElementBase.java:555)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encodeChild(LayoutElementBase.java:555)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encodeChild(LayoutElementBase.java:555)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encodeChild(LayoutElementBase.java:555)
at com.sun.jsftemplating.layout.descriptors.LayoutComponent.encode(LayoutComponent.java:243)
at com.sun.jsftemplating.layout.descriptors.LayoutComposition.encodeThis(LayoutComposition.java:161)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encode(LayoutElementBase.java:330)
at com.sun.jsftemplating.layout.descriptors.LayoutComposition.encodeThis(LayoutComposition.java:161)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encode(LayoutElementBase.java:330)
at com.sun.jsftemplating.layout.descriptors.LayoutComposition.encodeThis(LayoutComposition.java:161)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encode(LayoutElementBase.java:330)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.encode(LayoutElementBase.java:348)
at com.sun.jsftemplating.layout.descriptors.LayoutDefinition.encode(LayoutDefinition.java:246)
at com.sun.jsftemplating.layout.LayoutViewHandler.renderView(LayoutViewHandler.java:683)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:121)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:594)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:232)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
at com.sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:102)
at com.sun.grizzly.ssl.SSLOutputBuffer.flushChannel(SSLOutputBuffer.java:138)
at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:489)
at com.sun.grizzly.http.SocketChannelOutputBuffer.realWriteBytes(SocketChannelOutputBuffer.java:373)
at com.sun.grizzly.tcp.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:894)
at com.sun.grizzly.tcp.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:167)
at com.sun.grizzly.tcp.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:661)
at com.sun.grizzly.tcp.Response.doWrite(Response.java:685)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:425)
... 80 more

#]

This exception is due to some timing issue on this page. If I go to Domain Logs page and wait for the hour glass to disappear and THEN select the new server instance, the exception is not printed. However, if I select the instance while the hour glass is still displayed, the exception appears in the server.log. Therefore this issue may not be noticeable on a fast machine.



 Comments   
Comment by andriy.zhdanov [ 04/Jan/12 ]

Can not reproduce: when starting an instance, I can't get to Domain Logs page, I click Domain and then Domain Logs tab, but "Long running process" pop up appears and as soon as it's over, Stanadlone Instances page is displayed, and it's too late to go to Domain Logs tab again.

Comment by lidiam [ 04/Jan/12 ]

This will always happen on a slower system (happens all the time for me). I'll send you machine details so you can see it/test.

Comment by lidiam [ 06/Jan/12 ]

This issue should not be assigned to me.

There is another side effect of this screen still loading after it is displayed: the next action does not take place. Here is how to reproduce:

1. Go to Domain Logs page and then click on server node BEFORE the hour glass disappears (while the page is still doing something).
2. Click on Rotate Log button. A "long process" message is displayed but once it is gone, DAS server.log is not rotated and the above mentioned exception is written to the log file.

If I click on Rotate Log button again, it is rotated then. Perhaps we should have a "long process is running" message displayed on Domain Logs page as well till it's done loading?

Comment by Anissa Lam [ 13/Jan/12 ]

This bug is hard to reproduce and is not a frequent operation.
Downgrade to P4 and exclude from 3.1.2 release as we have passed HCF and only show stopper bug will be addressed.

Comment by Anissa Lam [ 20/Mar/13 ]

From previous comment, this is a hard to reproduce bug, and i cannot reproduce this even on the slowest machine i can find.
This isn't a frequent operation either.
Marking as won't fix.





[GLASSFISH-18108] Remove exception from user message Created: 31/Dec/11  Updated: 11/Jan/12  Resolved: 03/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b17, 4.0

Type: Bug Priority: Major
Reporter: lidiam Assignee: andriy.zhdanov
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Attachments: File working.diff    
Tags: 312_gui_new, 312_qa, 312_verified, 3_1_2_approved

 Description   

Create a cluster with no instances. Go to Domain Logs page. Select cluster and click Collect Logs. The popup message that shows up starts with java.lang.RuntimeException string. This string should be removed from the message.



 Comments   
Comment by andriy.zhdanov [ 03/Jan/12 ]
  • What is the impact on the customer of the bug?

Minimal.

  • What is the cost/risk of fixing the bug?

No risk.

  • Is there an impact on documentation or message strings?

No

  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

Collect Domain Logs tests

  • Which is the targeted build of 3.1.2 for this fix?

3_1_2_b17

Comment by Anissa Lam [ 03/Jan/12 ]

changes look good.

Comment by andriy.zhdanov [ 03/Jan/12 ]

Committed revision 51858
Committed revision 51861

Comment by lidiam [ 11/Jan/12 ]

Verified in build ogs-3.1.2-b17.zip.





[GLASSFISH-18098] Display message that instance does not exist when collecting logs Created: 30/Dec/11  Updated: 09/Feb/12  Resolved: 09/Feb/12

Status: Closed
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b16
Fix Version/s: 3.1.2_b18

Type: Bug Priority: Major
Reporter: lidiam Assignee: naman_mehta
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b16.zip


Attachments: JPEG File instance-startup-failure.JPG     File server.log.das     File server.log.instance.startupfailure    
Tags: 312_gui_new, 312_qa, 312_verified

 Description   

Create a standalone instance and do not start it. Go to Domain -> Domain Logs tab and select the newly created instance. Click Download button. The browser will be spinning for a very long time waiting for DAS (e.g. more than 10 minutes) and eventually come back with nothing. Instead, we should check if the instance directory exists and display a message to the user if it does not.



 Comments   
Comment by lidiam [ 30/Dec/11 ]

The following exception is printed to the DAS server.log:

[#|2011-12-29T16:27:05.931-0800|WARNING|glassfish3.1.2|com.sun.grizzly.config.Gr
izzlyServiceListener|_ThreadID=13;_ThreadName=Thread-2;|GRIZZLY0023: Interruptin
g idle Thread: admin-thread-pool-4848(12).|#]

[#|2011-12-29T16:27:05.935-0800|WARNING|glassfish3.1.2|javax.enterprise.system.c
ontainer.web.com.sun.enterprise.web|_ThreadID=29;_ThreadName=Thread-2;|StandardW
rapperValve[DownloadServlet]: PWC1406: Servlet.service() for servlet DownloadSer
vlet threw exception
java.lang.RuntimeException: com.sun.jersey.api.client.ClientHandlerException: ja
va.io.InterruptedIOException: Operation interrupted
at org.glassfish.admingui.common.servlet.LogFilesContentSource.getInputS
tream(LogFilesContentSource.java:110)
at org.glassfish.admingui.common.servlet.DownloadServlet.writeContent(Do
wnloadServlet.java:277)

Will attach server.log for details.

Comment by lidiam [ 30/Dec/11 ]

I'm getting an exception trying to attach server.log, so here is the full exception:

[#|2011-12-29T16:27:05.935-0800|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=29;_ThreadName=Thread-2;|StandardWrapperValve[DownloadServlet]: PWC1406: Servlet.service() for servlet DownloadServlet threw exception
java.lang.RuntimeException: com.sun.jersey.api.client.ClientHandlerException: java.io.InterruptedIOException: Operation interrupted
at org.glassfish.admingui.common.servlet.LogFilesContentSource.getInputStream(LogFilesContentSource.java:110)
at org.glassfish.admingui.common.servlet.DownloadServlet.writeContent(DownloadServlet.java:277)
at org.glassfish.admingui.common.servlet.DownloadServlet.doPost(DownloadServlet.java:174)
at org.glassfish.admingui.common.servlet.DownloadServlet.doGet(DownloadServlet.java:155)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:232)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.io.InterruptedIOException: Operation interrupted
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.CsrfProtectionFilter.handle(CsrfProtectionFilter.java:97)
at com.sun.jersey.api.client.filter.CsrfProtectionFilter.handle(CsrfProtectionFilter.java:97)
at com.sun.jersey.api.client.filter.CsrfProtectionFilter.handle(CsrfProtectionFilter.java:97)
at com.sun.jersey.api.client.Client.handle(Client.java:648)
at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:563)
at org.glassfish.admingui.common.util.RestUtil.postRestRequestFromServlet(RestUtil.java:712)
at org.glassfish.admingui.common.servlet.LogFilesContentSource.getInputStream(LogFilesContentSource.java:106)
... 28 more
Caused by: java.io.InterruptedIOException: Operation interrupted
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:695)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:640)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:660)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:318)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
... 37 more

#]
Comment by lidiam [ 31/Dec/11 ]

Consequently if I try to start the created server instance, it fails with the following in server.log:

[#|2011-12-30T17:13:53.734-0800|SEVERE|glassfish3.1.2|javax.enterprise.system.to
ols.admin.com.sun.enterprise.container.common|_ThreadID=10;_ThreadName=Thread-2;|The server requires a valid admin password to be set before it can start. Pleas
e set a password using the change-admin-password command.|#]

I created another instance on the same DCOM node and started it fine from Admin Console. However, if I create an instance, try to collect log files and then try to start it, startup fails with the above error (server.log attached).

Comment by lidiam [ 31/Dec/11 ]

Attaching DAS server.log.

Comment by andriy.zhdanov [ 02/Jan/12 ]

Sorry, can't reproduce any of the two problems on glassfish-3.1.2-b16, and can't understand how this could happen, or how second problem can relate to collecting log files.

Would it be possible to get access to an instance where the problem reproduces?

Comment by Anissa Lam [ 02/Jan/12 ]

If the instance is created using "localhost-domain1", and try to do collect-logs right after it is created, before starting it or restarting DAS, then I see a popup saying error occurs during download. I think thats correct and acceptable.
"java.lang.RuntimeException: Error while downloading log files from instance1"

If the instance is created using SSH node, the same experience as using localhost-domain1. see the popup error.

If the instance is created using DCOM node, then i see 'waiting for localhost' for a long time and nothing happens. Like Lidia described.
If i use CLI to do it,
%asadmin collect-log-files --target dcom-instance1

the command hangs and nothing comes back. Thats probably what is causing the behavior in GUI. Console is waiting for the command to finish.
Since this is specific to DCOM instance, transfer to Byron to take a look.

Andriy, I will send you the machine name and password to create a DCOM node so you can reproduce that too.

Comment by naman_mehta [ 09/Jan/12 ]

It's depends on 18055 so once 18055 is closed it's not now reproducible.

Comment by lidiam [ 09/Feb/12 ]

Reopening to mark as fixed/verified. Issue was certainly reproducible on earlier builds.

Comment by lidiam [ 09/Feb/12 ]

Fixed as part of issue 18055.

Comment by lidiam [ 09/Feb/12 ]

Verified in build ogs-3.1.2-b21.zip





[GLASSFISH-18079] RuntimeException when switching from DCOM to SSH node Created: 23/Dec/11  Updated: 13/Jan/12  Resolved: 05/Jan/12

Status: Closed
Project: glassfish
Component/s: admin_gui
Affects Version/s: 3.1.2_b15
Fix Version/s: 3.1.2_b17, 4.0

Type: Bug Priority: Major
Reporter: lidiam Assignee: Anissa Lam
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b15.zip on solaris sparc, node on windows xp


Attachments: JPEG File runtimeException.JPG     File server.log.exception     File server.log.exception2.with.password    
Tags: 312_gui_new, 312_qa, 312_verified, 3_1_2-approved

 Description   

I created a DCOM node and tried to switch it to SSH. After several minutes of waiting got RuntimeException on the screen (screenshot attached). The following exception is in server.log (also attached):

[#|2011-12-22T19:46:05.012-0800|WARNING|glassfish3.1.2|javax.enterprise.resource
.webcontainer.jsf.lifecycle|_ThreadID=212;_ThreadName=Thread-2;|java.lang.reflec
t.InvocationTargetException while attempting to process a 'command' event for 's
aveButton'.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException while at
tempting to process a 'command' event for 'saveButton'.
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHa
ndlers(LayoutElementBase.java:422)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHa
ndlers(LayoutElementBase.java:394)
at com.sun.jsftemplating.layout.event.CommandActionListener.invokeComman
dHandlers(CommandActionListener.java:150)
at com.sun.jsftemplating.layout.event.CommandActionListener.processActio
n(CommandActionListener.java:98)
at javax.faces.event.ActionEvent.processListener(ActionEvent.java:88)
at javax.faces.component.UIComponentBase.broadcast(UIComponentBase.java:
769)
at javax.faces.component.UICommand.broadcast(UICommand.java:300)
at com.sun.webui.jsf.component.WebuiCommand.broadcast(WebuiCommand.java:
166)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:794)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1
259)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicat
ionPhase.java:81)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:593)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java
:1542)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:217)
at com.sun.webui.jsf.util.UploadFilter.doFilter(UploadFilter.java:233)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:256)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:217)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:279)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.j
ava:655)
at org.apache.catalina.core.StandardPipeline.doChainInvoke(StandardPipel
ine.java:600)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:96)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESess
ionLockingStandardPipeline.java:91)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:162)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.j
ava:331)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(Container
Mapper.java:232)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:8
49)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFil
ter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultPro
tocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.jav
a:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.jav
a:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java
:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextT
ask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.
java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadP
ool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool
.java:513)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jsftemplating.layout.descriptors.handler.Handler.invoke(Handl
er.java:442)
at com.sun.jsftemplating.layout.descriptors.LayoutElementBase.dispatchHa
ndlers(LayoutElementBase.java:420)
... 43 more
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.io.Interrupted
IOException: Operation interrupted
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle
(URLConnectionClientHandler.java:149)
at com.sun.jersey.api.client.filter.CsrfProtectionFilter.handle(CsrfProt
ectionFilter.java:97)
...

I cannot create an SSH node on this windows system, so did not expect the conversion to go through. However, we should not be printing exceptions on the screen in such cases.



 Comments   
Comment by lidiam [ 23/Dec/11 ]

I managed to create an SSH node on the same windows system using password. Hence, I tried this scenario again: first created a DCOM node and then tried to convert it to an SSH node using password (as opposed to key file). Again got runtime exception on the screen after a long time of what looked like a hang. The server.log is attached as server.log.exception2.with.password.

Comment by Anissa Lam [ 02/Jan/12 ]

The stack trace shows IOException,
Caused by: com.sun.jersey.api.client.ClientHandlerException: java.io.Interrupted
IOException: Operation interrupted

How long have you waited to see this ?

For me, i cannot get to this state even though i waited for over 10 min. I only see the "Long Process is detected" box. Waited for a long time, and click the "x" to remove that box, and i can go back to other screens without problem.

If you do the conversion using CLI, using update-node-ssh , what did you get ?

Comment by lidiam [ 04/Jan/12 ]

I tried CLI and it worked fine, no hang, node got converted to SSH and I can ping it successfully from Admin Console after conversion. The command I used was:

asadmin update-node-ssh --nodehost <host name> --sshport 22 --sshuser <user name> jedy

The command passed by Admin Console may be passing many more parameters, such as the Glassfish installation directory, since it's filled out. In my case the Glassfish install dir is C:\as\dcomtest... meaning, it contains backslashes, not sure if that causes issues.

Comment by Anissa Lam [ 05/Jan/12 ]

What is the impact on the customer of the bug?
User is seeing Runtime exception on screen, which should never happen.

What is the cost/risk of fixing the bug?
Minimal. Catch the exception instead of letting it propagated onward.

Is there an impact on documentation or message strings?
No

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Run this same test and see that the exception is logged in server.log, and the error displayed to user nicely.

Which is the targeted build of 3.1.2 for this fix?
3_1_2_b17

Note that this fix is to ensure no runtime exception is seen on screen. If there is issue of converting SSH to DCOM or vise versa, another bug should be open for that.

Index: src/main/java/org/glassfish/admingui/common/handlers/RestApiHandlers.java
===================================================================
— src/main/java/org/glassfish/admingui/common/handlers/RestApiHandlers.java (revision 51836)
+++ src/main/java/org/glassfish/admingui/common/handlers/RestApiHandlers.java (working copy)
@@ -40,6 +40,7 @@

package org.glassfish.admingui.common.handlers;

+import java.util.logging.Logger;
import com.sun.jsftemplating.annotation.Handler;
import com.sun.jsftemplating.annotation.HandlerInput;
import com.sun.jsftemplating.annotation.HandlerOutput;
@@ -218,7 +219,23 @@
if (GuiUtil.isEmpty(endpoint))

{ handlerCtx.setOutputValue("result", new HashMap()); }

else{

  • handlerCtx.setOutputValue("result", RestUtil.restRequest(endpoint, attrs, method, handlerCtx, quiet, throwException));
    + try { + Map result = RestUtil.restRequest(endpoint, attrs, method, handlerCtx, quiet, throwException); + handlerCtx.setOutputValue("result", result ); + }

    catch(Exception ex)

    Unknown macro: {+ Logger logger = GuiUtil.getLogger();+ if (logger.isLoggable(Level.FINE)) { + ex.printStackTrace(); + }+ Map maskedAttr = RestUtil.maskOffPassword(attrs);+ GuiUtil.getLogger().log(+ Level.SEVERE,+ ex.getMessage() + ";n" ++ ex.getCause() + ";n" + + GuiUtil.getCommonMessage("LOG_REST_REQUEST_INFO", new Object[]{endpoint, maskedAttr, method}));+ GuiUtil.handleError(handlerCtx, GuiUtil.getMessage("msg.error.checkLog"));+ return;+ }

    }
    }

Index: src/main/java/org/glassfish/admingui/common/util/RestUtil.java
===================================================================
— src/main/java/org/glassfish/admingui/common/util/RestUtil.java (revision 51859)
+++ src/main/java/org/glassfish/admingui/common/util/RestUtil.java (working copy)
@@ -218,7 +218,7 @@
return parseResponse(restResponse, handlerCtx, endpoint, (useData && "post".equals(method))? data: attrs, quiet, throwException);
}

  • private static Map maskOffPassword(Map<String, Object> attrs){
    + public static Map maskOffPassword(Map<String, Object> attrs){
    Map masked = new HashMap();
    if (attrs == null){
    return masked;
Comment by scatari [ 05/Jan/12 ]

Should the Log Level be SEVERE for this not so common use case?

Comment by Anissa Lam [ 05/Jan/12 ]

I believe this needs to be logged at SEVERE, whether it is common or not.
Whenever there is exception thrown from the REST, the request is not able to perform successfully. Although it really shouldn't happen. This maybe post, get, delete. Any action that user wants to perform has failed. We should log that.
Note that the exception stack trace is only logged in FINE level.

Comment by Anissa Lam [ 05/Jan/12 ]

Changes checked into both trunk and 3.1.2 branch.

GLASSFISH-18079. Catch runtime exception in restRequest handler for the rare case that REST is throwing exception.

Revisions:
----------
51906

Modified Paths:
---------------
branches/3.1.2/admingui/common/src/main/java/org/glassfish/admingui/common/util/RestUtil.java
branches/3.1.2/admingui/common/src/main/java/org/glassfish/admingui/common/handlers/RestApiHandlers.java

===============

Revisions:
----------
51907

Modified Paths:
---------------
trunk/main/appserver/admingui/common/src/main/java/org/glassfish/admingui/common/util/RestUtil.java
trunk/main/appserver/admingui/common/src/main/java/org/glassfish/admingui/common/handlers/RestApiHandlers.java

Comment by lidiam [ 13/Jan/12 ]

Verified in build ogs-3.1.2-b17.zip. Exception no longer printed, but switching from DCOM to SSH is still not possible in Admin Console. I'll log a separate issue for that.





[GLASSFISH-18075] Solaris 11. Remote commands - failed: /home/hudson/workspace/Cluster/glassfish3/glassfish/lib/nadmin: No such file or directory Created: 22/Dec/11  Updated: 22/Dec/11  Resolved: 22/Dec/11

Status: Resolved
Project: glassfish
Component/s: admin
Affects Version/s: 3.1.2_b15
Fix Version/s: None

Type: Bug Priority: Major
Reporter: easarina Assignee: Tom Mueller
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 312_admin, 312_blocking, 312_qa, 312_regression

 Description   

Two Solaris 11 machines, jdk 1.6.0_30, 3.1.2 build 15.

Installed build on both machines. Then executed the follow commands:
start-domain
change-admin-password
enable-secure-admin
stop-domain
start-domain

Then from DAS machine asqe-sblade-1 I've executed such commands:
asadmin --user admin create-cluster c1
Enter admin password for user "admin">
Command create-cluster executed successfully.
asadmin --user admin create-node-ssh --nodehost asqe-sblade-2 node2
Enter admin password for user "admin">
Command create-node-ssh executed successfully.
asadmin --user admin create-instance --node node2 --cluster c1 in1
Enter admin password for user "admin">
Command failed on node node2 (asqe-sblade-2): bash: /home/hudson/workspace/Cluster/glassfish3/glassfish/lib/nadmin: No such file or directory
Command create-instance completed with warnings.

hudson@asqe-sblade-1:~/workspace/Cluster/glassfish3/glassfish/bin$ asadmin --user admin create-cluster c2
Enter admin password for user "admin">
Command create-cluster executed successfully.

Then from asqe-sblade-2:

asadmin --user admin --host asqe-sblade-1 create-local-instance --node node2 --cluster c2 in3
Enter admin password for user "admin">
Rendezvoused with DAS on asqe-sblade-1:4848.
Port Assignments for server instance in3:
JMX_SYSTEM_CONNECTOR_PORT=28687
JMS_PROVIDER_PORT=27677
HTTP_LISTENER_PORT=28081
ASADMIN_LISTENER_PORT=24849
JAVA_DEBUGGER_PORT=29010
IIOP_SSL_LISTENER_PORT=23821
IIOP_LISTENER_PORT=23701
OSGI_SHELL_TELNET_PORT=26667
HTTP_SSL_LISTENER_PORT=28182
IIOP_SSL_MUTUALAUTH_PORT=23921
Command create-local-instance executed successfully.

Again from DAS on asqe-sblade-1:

asadmin --user admin start-cluster c2
Enter admin password for user "admin">
remote failure: in3: Could not start instance in3 on node node2 (asqe-sblade-2).

Command failed on node node2 (asqe-sblade-2): bash: /home/hudson/workspace/Cluster/glassfish3/glassfish/lib/nadmin: No such file or directory

To complete this operation run the following command locally on host asqe-sblade-2 from the GlassFish install location /home/hudson/workspace/Cluster/glassfish3:

lib/nadmin start-local-instance --node node2 --sync normal in3

The command start-instance failed for: in3
Command start-cluster failed.

asadmin --user admin start-instance in3
Enter admin password for user "admin">
remote failure: Could not start instance in3 on node node2 (asqe-sblade-2).

Command failed on node node2 (asqe-sblade-2): bash: /home/hudson/workspace/Cluster/glassfish3/glassfish/lib/nadmin: No such file or directory

To complete this operation run the following command locally on host asqe-sblade-2 from the GlassFish install location /home/hudson/workspace/Cluster/glassfish3:

lib/nadmin start-local-instance --node node2 --sync normal in3
Command start-instance failed.

==========================================================================

So all remote commands: start-instance, start-cluster, create-instance - failed.



 Comments   
Comment by Tom Mueller [ 22/Dec/11 ]

There was no --installdir option provided to the create-node-ssh command. Was glassfish actually installed in the /home/hudson/workspace/Cluster/glassfish3 directory on asqe-sblade-2?

If not, then this is the expected behavior.

Comment by easarina [ 22/Dec/11 ]

Yes, GF on asqe-sblade-2 and on asqe-sblade-1 was installed at the same location: /home/hudson/workspace/Cluster/glassfish3.

I saw in /home/hudson/workspace/Cluster/glassfish3/lib/ only asadmin, not nadmin.

Also I've executed on that machines the regular cli automated test, that I've executed successfully on othe machines against 3.1.2, but here it totally failed, because the remote commands failed.

Comment by Tom Mueller [ 22/Dec/11 ]

How was the GlassFish software installed on asqe-sblade-2? Via unzip or an installer? Or via "asadmin install-node"?

Comment by Tom Mueller [ 22/Dec/11 ]

Hold the phone. "nadmin" only is referenced by the GlassFish trunk code (4.0). It is not referenced by 3.1.2.
Is the DAS install on asqe-sblade-1 really 3.1.2?

Comment by easarina [ 22/Dec/11 ]

I've unzipped bits on both machines. DAS was on asqe=sblade-1

hudson@asqe-sblade-1:~/workspace/Cluster/glassfish3/glassfish/bin$ ./asadmin version --verbose
Version = GlassFish Server Open Source Edition 3.1.2-b15 (build 15), JRE version 1.6.0_30
Command version executed successfully.
hudson@asqe-sblade-1:~/workspace/Cluster/glassfish3/glassfish/bin$

hudson@asqe-sblade-2:~/workspace/Cluster/glassfish3/glassfish/bin$ ./asadmin version --verbose
Version string could not be obtained from Server [localhost:4848] for some reason.
(Turn debugging on e.g. by setting AS_DEBUG=true in your environment, to see the details).
Using locally retrieved version string from version class.
Version = GlassFish Server Open Source Edition 3.1.2-b15 (build 15)
asadmin Java Runtime Environment version: 1.6.0_30
Command version executed successfully.

This is my first execution against b15, I will run the tests against b15 on other platforms.

Comment by Tom Mueller [ 22/Dec/11 ]

Is it possible that the domain was started using just "asadmin" expecting to find the one that you installed, but the PATH was actually pointing to a trunk build?

Comment by easarina [ 22/Dec/11 ]

For the automated test I'm using full paths and the test puts my GF bin dir in a path. But for manual executions I did not have anything in a path and I've used ./asadmin from my bin directory.

And when it can not find a path (bash: /home/hudson/workspace/Cluster/glassfish3/glassfish/lib/nadmin: No such file or directory) it refers to my installation path.

Comment by Tom Mueller [ 22/Dec/11 ]

The only way to get a reference to "lib/nadmin" is if you are running trunk code in the DAS.

Can you please try to recreate this issue, making sure that you use only 3.1.2 software?

I'm going to resolve this as "Cannot Reproduce" until you are able to reproduce this issue using only 3.1.2 software. Please include the complete sequence of commands, including those that are used to install the software, and where the installation files are downloaded from as well as the command that is used to start the domain.

If you are able to reproduce it, please reopen the issue.





[GLASSFISH-18067] Failure to delete files not reported to user in Admin Console Created: 21/Dec/11  Updated: 12/Jan/12  Resolved: 27/Dec/11

Status: Closed
Project: glassfish
Component/s: distributed management
Affects Version/s: 3.1.2_b15
Fix Version/s: 3.1.2_b15, 4.0_b16

Type: Bug Priority: Major
Reporter: lidiam Assignee: Byron Nevins
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b15.zip on solaris


Attachments: JPEG File delete-cluster1.JPG     JPEG File delete-cluster2.JPG     Text File server.log    
Tags: 312_gui_new, 312_qa, 312_verified, 3_1_2-approved

 Description   

Steps to reproduce:

In Admin Console:

1. Create a remote DCOM node.
2. Create a cluster with one instance on the new node.
3. Start and stop cluster. (optional)
4. Log in to the remote windows machine, where cluster resides and go to the following directory in DOS window:

<install directory>/glassfish/nodes/<DCOM node name>/<clustered instance name>

This is so that removing of this directory fails (since it is in use).

5. On Clusters page select the cluster and click on Delete. No warning will be seen, however server.log contains the following:

[#|2011-12-20T18:21:51.250-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=18;_ThreadName=Thread-2;|Executed this remote command:
C:\as\dcomtest\glassfish3\glassfish\bin\asadmin.bat --_auxinput C:\as\dcomtest\glassfish3\glassfish\bin\DELETE_ME_31217372646810002375 --interactive=false _delete-instance-filesystem --node jedy43 cljed1
Received this output:
UTIL6046: Attempt to rename C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\cljed1 to C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\oldinst3712054486246994744.tmp failed after 6 retries
Command _delete-instance-filesystem failed.

#]

[#|2011-12-20T18:21:51.254-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=18;_ThreadName=Thread-2;|UTIL6046: Attempt to rename C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\cljed1 to C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\oldinst3712054486246994744.tmp failed after 6 retries
Command _delete-instance-filesystem failed.|#]

These messages should not be printed at INFO level, which is probably why Admin Console does not detect that something is wrong.



 Comments   
Comment by Anissa Lam [ 21/Dec/11 ]

Have you tried using CLI ? Do you see the error reported if using CLI ?
Assign to Byron for initial evaluation.

Comment by Byron Nevins [ 22/Dec/11 ]

The message told you precisely what it was unable to do. On Windows if you yourself can't delete/rename a file than neither can we. So in a case like this never file a bug without trying to do what it said it couldn't do. In this case you would simply do this:

rd /s C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\cljed1

Almost certainly you have a file or directory locked.

Go get this application from the internet and run it like so:
handle.exe

handle cljed1

That will tell you what process is using the file.

Comment by lidiam [ 22/Dec/11 ]

The point of this bug is not that files are not deleted. That is not the expectation. In fact, I purposely had the instance directory in use on the windows machine to see what happens when I issue uninstall from Admin Console. The problem is that the cause is not reported to the user in Admin Console. This is inconsistent with other commands, when they fail, and a usability issue. When a failure happens, the cause of the failure needs to be displayed in Admin Console, if possible. In this case, not only that the cause of the failure is not displayed - there is no warning at all so user may have no idea that directories were left behind on the windows system. In short here are the issues:

1. No warning in Admin Console
2. Cause of failure to delete directories not printed in Admin Console
3. Failure to delete directories printed in server.log at INFO level.

All of the above should be fixed.

Comment by Anissa Lam [ 22/Dec/11 ]

Lidia, why spell out "Admin Console" ? do you see the warning given out if you use CLI to perform the "delete-cluster" command ?

1. No warning in Admin Console
2. Cause of failure to delete directories not printed in Admin Console

Comment by lidiam [ 22/Dec/11 ]

My concern is with Admin Console behaviour, and that's why I listed issues as seen in Admin Console. However, I tried this scenario in CLI and the problem with deleting directories is reported to the user there:

lancerl(j2eetest):/export/home/j2eetest/3.1.2/glassfish3/glassfish/domains/domain1/logs# asadmin delete-instance clj1
Enter admin password for user "admin">
UTIL6046: Attempt to rename C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\clj1 to C:\as\dcomtest\glassfish3\glassfish\nodes\jedy43\oldinst4407860992800066086.tmp failed after 6 retries
Command _delete-instance-filesystem failed.
The instance, clj1, was deleted from host jed-asqe-43
Command delete-instance executed successfully.

I think the above may be confusing to new users as well:

1. "instance, clj1, was deleted from host jed-asqe-43" - well, it was not; it was only deleted from configuration
2. command executed successfully even though subcommand failed

However, Admin Console does not report the above. There is no warning image/message displayed at all (I'll add screenshots). That happens for both deleting instance from a cluster as well as deleting the entire cluster with instances (which is allowed from Admin Console but not allowed in CLI).

Comment by lidiam [ 22/Dec/11 ]

Screenshots for deleting cluster in Admin Console, when directory on host windows machine is in use and therefore not deleted.

Comment by Anissa Lam [ 23/Dec/11 ]

Looked at this with Byron.
I confirmed that

  • if the instance is on localhost-domain1, then even if the nodes directory is set RO, that it cannot be deleted, the delete command returns WARNING, and console can show that nicely to user.
  • if the instance is on a DCOM node, then the delete command returns SUCCESS, even though the nodes directory cannot be deleted.
    So, this is indeed a DCOM node specific error.
Comment by Byron Nevins [ 23/Dec/11 ]

The problem is that we don't know how to get the return status of a remote script run by DCOM.
I added a hack that looks for the magic string "failed" in the returned output.

Comment by Byron Nevins [ 23/Dec/11 ]

fixed in 4.0

d:\gf\trunk\main\nucleus\cluster>svn commit
Sending cluster\ssh\src\main\java\org\glassfish\cluster\ssh\connect\NodeRunner.java
Transmitting file data .
Committed revision 51744.

Comment by Byron Nevins [ 23/Dec/11 ]

What is the impact on the customer of the bug?
False positive. If the files were NOT deleted on Windows they won't see a Warning in the GUI.

How likely is it that a customer will see the bug and how serious is the bug?
A bit likely and serious simply because it is broken.

Is it a regression? Does it meet other bug fix criteria (security, performance, etc.)?
Not a regression – new feature

What is the cost/risk of fixing the bug?
Fixing it perfectly – unended. I don't know how to get the integer return value via COM.
A reasonable but brittle fix is to look for a "magic string" to know that a warning occured.

How risky is the fix? How much work is the fix? Is the fix complicated?
Not risky.
1 day.
String Parsing.

Is there an impact on documentation or message strings?
No - it just does what the analogous commands do already.

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
QA should re-run manual/auto tests to verify

Which is the targeted build of 3.1.2 for this fix?
B16

Comment by Byron Nevins [ 23/Dec/11 ]

The fix is ready to go whenever I get approval

Comment by Anissa Lam [ 23/Dec/11 ]

I don't quite understand the following :

What is the impact on the customer of the bug?
_delete-service will always fail on Linux

The problem is for DCOM node instance, my DAS os on my Mac. Where and why Linux comes into the picture ?

Comment by Byron Nevins [ 27/Dec/11 ]

My fix that went into 4.0 needs to get rolled back.

problem – we don't know how to get status back from a remote Windows command.

Solution in this case – simply check if the directory still exists after running the command.

Comment by Byron Nevins [ 27/Dec/11 ]

Simple? Not so simple after all!

Solution is to check and see if the directory that was supposed to be deleted still exists using jcifs (SAMBA).

Fixed in 3.1.2 and in 4.0

Sending D:\gf\branches\3.1.2\cluster\ssh\src\main\java\org\glassfish\cluster\ssh\connect\NodeRunnerDcom.java
Sending D:\gf\trunk\main\nucleus\cluster\ssh\src\main\java\org\glassfish\cluster\ssh\connect\NodeRunnerDcom.java
Transmitting file data ..
Committed revision 51792.

Comment by lidiam [ 12/Jan/12 ]

verified in build ogs-3.1.2-b17.zip





[GLASSFISH-18066] Instance state not updated after startup Created: 21/Dec/11  Updated: 12/Jan/12  Resolved: 27/Dec/11

Status: Closed
Project: glassfish
Component/s: admin
Affects Version/s: 4.0_b16
Fix Version/s: 3.1.2_b15, 4.0_b16

Type: Bug Priority: Major
Reporter: lidiam Assignee: Byron Nevins
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

ogs-3.1.2-b15.zip on solaris sparc, Chrome on winxp


Attachments: JPEG File instance-started-wrong-state.JPG     File server.log.cltup1    
Issue Links:
Duplicate
duplicates GLASSFISH-17116 list-instances lets asadmin timeout w... Resolved
Tags: 312_qa, 312_regression, 312_verified

 Description   

I created a cluster with one instance on a remote machine, on an SSH node. I started cluster but after Console comes back, the instance is still showing as stopped (screenshot attached). I can see from server.log that instance started fine. After clicking on Nodes, I can see in Admin Console that the instance is running as well, it's only the initial state is not refreshed after a clustered instance started. This happened with the second cluster I created. The first one happened to be on a DCOM node. I tried third cluster also on SSH node and again, the status did not get updated once Admin Console is back from performing the startup action.



 Comments   
Comment by lidiam [ 21/Dec/11 ]

I removed all clusters and created just one, with an instance on a remote SSH node. After starting cluster the status was not updated on the Clusters page, so it looks like a regression.

Comment by Anissa Lam [ 21/Dec/11 ]

I can reproduce this if the node is SSH. But not if the node is localhost-domain1
There is no difference on refreshing this page regardless what kind of node the instance is created on. I am suspecting backend is not returning the correct info in a timely manner.

After the page comes back and show the 'stalled' info, if you click on the clusters tree node again, which means refresh this page, the correct info is displayed. The same code path is executed and yet it is fine this time.

Comment by Anissa Lam [ 21/Dec/11 ]

in fact, if you create a cluster with 2 instance, one on SSH node, and another one on localhost-domain1
after the cluster started, the local one says running, but the SSH one still says stopped.

Now, refresh by clicking the clusters treenode, now the SSH one also refreshed correctly.

Comment by Anissa Lam [ 21/Dec/11 ]

I believe this is a backend issue.
I change GUI log level to FINEST, and also printed out what list-instances returned.
After cluster is started, It clearly shows that the status of the instance created with SSH node is returned as not running, but the localhost instance status is running.

The log for starting the cluster and whats returned is attached. There is lots of noise since GUI logger is set to FINEST.
But here is relevant one:

[#|2011-12-21T15:24:12.272-0800|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=130;_ThreadName=stdout;|Command start-local-instance executed successfully.|#]

[#|2011-12-21T15:24:12.622-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=131;_ThreadName=pool-101-thread-2;|Waiting for local-instance to start ...........
Successfully started the instance: local-instance
instance Location: /Users/anilam/Awork/V3/3.1.2/3.1.2/dist-gf/glassfish/nodes/localhost-domain1/local-instance
Log File: /Users/anilam/Awork/V3/3.1.2/3.1.2/dist-gf/glassfish/nodes/localhost-domain1/local-instance/logs/server.log
Admin Port: 24848
Command start-local-instance executed successfully.|#]

[#|2011-12-21T15:24:20.932-0800|INFO|glassfish3.1.2|javax.enterprise.system.tools.admin.com.sun.enterprise.v3.admin.cluster|_ThreadID=132;_ThreadName=pool-101-thread-1;|Waiting for SSH-instance to start .............
Successfully started the instance: SSH-instance
instance Location: /tmp/testing/glassfish/nodes/BT/SSH-instance
Log File: /tmp/testing/glassfish/nodes/BT/SSH-instance/logs/server.log
Admin Port: 24848
Command start-local-instance executed successfully.|#]

[#|2011-12-21T15:24:21.016-0800|INFO|glassfish3.1.2|javax.org.glassfish.gms.org.glassfish.gms.admin|_ThreadID=23;_ThreadName=admin-thread-pool-4848(2);|GMSAD3001: GMSAnnounceAfterStartClusterCommand: exitCode:SUCCESS members null clusterMembers:[SSH-instance, local-instance]|#]

[#|2011-12-21T15:24:21.016-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=23;_ThreadName=admin-thread-pool-4848(2);|GMS1098: GMS:Announcing GroupStartup[COMPLETED_SUCCESS] for group: clusterABC members: SSH-instance,local-instance,|#]

[#|2011-12-21T15:24:21.016-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=23;_ThreadName=admin-thread-pool-4848(2);|GMS1062: GroupStart for group: clusterABC State: COMPLETED_SUCCESS Started Members: SSH-instance,local-instance,|#]

[#|2011-12-21T15:24:21.102-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/clusters/cluster
attrs={}
method=get|#]

[#|2011-12-21T15:24:21.106-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/clusters/cluster/clusterABC
attrs={}
method=get|#]

[#|2011-12-21T15:24:21.120-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/_get-restart-required
attrs={}
method=get|#]

[#|2011-12-21T15:24:21.127-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/anonymous-user-enabled
attrs={}
method=get|#]

[#|2011-12-21T15:24:21.252-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/clusters/cluster/clusterABC/server-ref
attrs={}
method=get|#]

[#|2011-12-21T15:24:21.300-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/list-instances
attrs=

{id=SSH-instance, long=true}

method=get|#]

[#|2011-12-21T15:24:23.536-0800|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);|=============== requestScope.statusMap =

{SSH-instance=NOT_RUNNING}

|#]

[#|2011-12-21T15:24:23.538-0800|FINEST|glassfish3.1.2|org.glassfish.admingui|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);ClassName=org.glassfish.admingui.common.util.RestUtil;MethodName=restRequest;|restRequest: endpoint=http://localhost:4848/management/domain/list-instances
attrs=

{id=local-instance, long=true}

method=get|#]

[#|2011-12-21T15:24:23.809-0800|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=17;_ThreadName=admin-thread-pool-4848(1);|=============== requestScope.statusMap =

{local-instance=RUNNING}

|#]

Comment by Tom Mueller [ 22/Dec/11 ]

Byron, I suspect this might be the InstanceInfo timeout issue in another form.
Can you please take a look.

Comment by Byron Nevins [ 27/Dec/11 ]

We have a hodge-podge of techniques for determining the answer to this question:

is the server running?

The problem here is:

start-local-instance (which is called by start-instance which is called by start-cluster) says that an instance is running if the PID file exists. This happens fairly early in the server startup.

list-instances – which the GUI calls frequently defines a running instance as a server that returns output from the _locations command. The ability to do that happens later in the startup sequence. Obviously it takes longer than the "new" timeout (2 seconds) to go from PID file exists to _locations returning.

RECOMMENDED FIX:

for 3.1.1 -> Make the list-instances timeout longer. 2 seconds>60 seconds

for 4+ --> Define the answer to "is this server running" one and only one way with, preferably, one and only one utility method.

Comment by Byron Nevins [ 27/Dec/11 ]

17116

Comment by Byron Nevins [ 27/Dec/11 ]

The 60 second timeout in list-instance should resolve the problem in almost all cases.

Comment by Byron Nevins [ 27/Dec/11 ]

Reopening against 4.0 +

We need a general solution for determining exactly what constitutes "is running" for a server.

Comment by Byron Nevins [ 27/Dec/11 ]

I changed my mind. This is too confusing to re-use this issue. I'll just create another issue.

Comment by lidiam [ 12/Jan/12 ]

verified in build ogs-3.1.2-b17.zip





[GLASSFISH-18065] Warnings printed to server.log on successfull startup Created: 21/Dec/11  Updated: 12/Jan/12  Resolved: 22/Dec/11

Status: Closed
Project: glassfish
Component/s: jts
Affects Version/s: 3.1.2_b15
Fix Version/s: 3.1.2_b16

<
Type: Bug Priority: Major
Reporter: lidiam Assignee: marina vatkina
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified