Issue Details (XML | Word | Printable)

Key: GLASSFISH-20712
Type: Bug Bug
Status: Open Open
Priority: Blocker Blocker
Assignee: Rajiv Mordani
Reporter: alev50
Votes: 1
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
glassfish

Session Replication (Possibly CNF error in web container?)

Created: 19/Jul/13 07:56 AM   Updated: 13/Aug/13 12:15 AM
Component/s: web_container
Affects Version/s: 3.1.2.2
Fix Version/s: None

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive logs_180713_1612.zip (3.64 MB) 19/Jul/13 05:21 PM - Ed Bratt

Environment:

1 node, 1 cluster, 2 instances


Tags:
Participants: alev50, Ed Bratt, Rajiv Mordani, Sanjeeb Sahoo and Shing Wai Chan


 Description  « Hide

Here is the scenario I played :

1 - Start cluster
2 - Deploy web app
3 - Connect through load balancer -> instance 2
4 - Log on
5 - Stop instance 2
6 - Refresh browser -> instance 1
7 - Session lost + StreamCorruptedException
8 - Stop cluster

Logs are attached.

Thanks for help.

Regards,

Anthony

-----------------------------------------------------------------------------------------------------------------
Fialli Joe joe.fialli@oracle.com via shoal.java.net

Thanks for providing a full set of server logs just containing the test scenario that is failing for you.

I just want to make an overall statement that we are trying to track down class loading issues in the web container.
So you might want to post this issue to glassfish alias instead of the shoal alias.
So it is overkill to turn on the shoal logging to FINEST since that subsystem could not even be responsible
for corrupted stream (grizzly is the transport for messages between cluster members and the web container passes a
byte of serialized content into Shoal messaging system and when trying to reconstitute replicated session,
the byte array is taken out of shoal messaging subsystem and deserialized should be deserialized using web container class path.
While the failure is stating corrupted stream, the failure is always exactly the same type code of "00". If it truely was a corrupted stream,
we would see different values all the time. The best bet for Type Code 00 in stream is a class not being found.
The ClassNotFoundException is getting consumed and not reported.

Possible reason for the ClassNotFoundException is the following.

My observations when looking at the server log you sent yesterday was that it was incorrect
for there not to be a Web Container classloader context to deserialize the session context
(after all, the class is the app loaded in the container, the default ObjectInputStream is not going
to be able to deserialize that.)

So this failure is significant and needs interpretation by someone who works on web container. It happens quite a bit in the server logs that you sent in.
I am not aware why this would happen. The following might explain trying to deserialize replicated content in web container without using a web container
class path.

[#|2013-07-18T16:00:02.639+0200|FINEST|glassfish3.1.2|org.apache.catalina.loader.WebappLoader|_ThreadID=10;_ThreadName=Thread-2;ClassName=null;MethodName=null;|getClasspath
java.lang.NoSuchMethodException: com.sun.enterprise.v3.server.APIClassLoaderServiceImpl$APIClassLoader.getClasspath()
at java.lang.Class.getMethod(Class.java:1607)
at org.apache.catalina.loader.WebappLoader.getClasspath(WebappLoader.java:1196)
at org.apache.catalina.loader.WebappLoader.setClassPath(WebappLoader.java:1145)
at org.apache.catalina.loader.WebappLoader.start(WebappLoader.java:692)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:5298)
at com.sun.enterprise.web.WebModule.start(WebModule.java:498)
at org.apache.catalina.core.ContainerBase.startChildren(ContainerBase.java:1518)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1184)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:995)
at org.apache.catalina.core.ContainerBase.startChildren(ContainerBase.java:1518)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1184)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:529)
at org.apache.catalina.startup.Embedded.start(Embedded.java:942)
at com.sun.enterprise.web.WebContainer.postConstruct(WebContainer.java:604)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:93)
at com.sun.enterprise.v3.services.impl.WebContainerStarter.startWebContainer(WebContainerStarter.java:202)
at com.sun.enterprise.v3.services.impl.WebContainerStarter.postConstruct(WebContainerStarter.java:134)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)

#]

*************

Here is time of first stream corrupted warning log message.
#|2013-07-18T16:06:35.588+0200|WARNING|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.ha.session.management|_ThreadID=75;_ThreadName=Thread-2;|Exception occurred in getSession

java.io.StreamCorruptedException: invalid type code: 00

Here is ClassNotFoundException that was probably related to that failure. (Note that this is a nested anonymous class. Typically difficult to serialize correctly.)
So the class below is probably one to look at.

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| --> Passing on ClassNotFoundException|#]

Here are a large number classes that are being looked for just before this ClassNotFoundException log message.
[#|2013-07-18T16:06:34.749+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| --> Passing on ClassNotFoundException|#]

Note that the classes are mostly UI classes related to ajax, java faces.

[#|2013-07-18T16:06:37.493+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.web.messages)|#]
[#|2013-07-18T16:06:37.508+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.web.messages_fr)|#]
[#|2013-07-18T16:06:37.514+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.web.messages_fr_FR)|#]

[#|2013-07-18T16:06:38.123+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities)|#]
[#|2013-07-18T16:06:38.129+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities_fr)|#]

[#|2013-07-18T16:06:38.137+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.xml.serializer.XMLEntities_fr_FR)|#]

[#|2013-07-18T16:06:35.921+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.component.AjaxViewRootBeanInfo)|#]

[#|2013-07-18T16:06:35.925+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIViewRootBeanInfo)|#]

[#|2013-07-18T16:06:35.926+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIComponentBaseBeanInfo)|#]

[#|2013-07-18T16:06:36.012+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.messages_fr)|#]

[#|2013-07-18T16:06:36.019+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.ajax4jsf.messages_fr_FR)|#]

[#|2013-07-18T16:06:37.178+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.sun.facelets.compiler.UIInstructionsBeanInfo)|#]

[#|2013-07-18T16:06:37.183+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.sun.facelets.compiler.UILeafBeanInfo)|#]

[#|2013-07-18T16:06:37.190+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(login)|#]
[#|2013-07-18T16:06:37.198+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(login_fr_FR)|#]

[#|2013-07-18T16:06:37.205+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.html.HtmlFormBeanInfo)|#]
[#|2013-07-18T16:06:37.207+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIFormBeanInfo)|#]

[#|2013-07-18T16:06:37.253+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.html.HtmlMessagesBeanInfo)|#]
[#|2013-07-18T16:06:37.255+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIMessagesBeanInfo)|#]
[#|2013-07-18T16:06:37.265+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.richfaces.component.html.HtmlInputTextBeanInfo)|#]
[#|2013-07-18T16:06:37.273+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIInputBeanInfo)|#]

[#|2013-07-18T16:06:37.277+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UIOutputBeanInfo)|#]
[#|2013-07-18T16:06:37.326+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.richfaces.component.html.HtmlInputSecretBeanInfo)|#]

[#|2013-07-18T16:06:37.328+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(org.richfaces.component.html.HtmlInputSecretBeanInfo)|#]

[#|2013-07-18T16:06:37.341+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.html.HtmlCommandButtonBeanInfo)|#]
[#|2013-07-18T16:06:37.344+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.UICommandBeanInfo)|#]
[#|2013-07-18T16:06:37.392+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(javax.faces.component.html.HtmlSelectBooleanCheckboxBeanInfo)|#]

[#|2013-07-18T16:06:37.195+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=76;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(login_fr)|#]

[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|loadClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;| Delegating to classloader1 org.glassfish.internal.api.DelegatingClassLoader@7a8f9805|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;| Searching local repositories|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| --> Passing on ClassNotFoundException|#]

[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;|loadClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.582+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;| Delegating to classloader1 org.glassfish.internal.api.DelegatingClassLoader@7a8f9805|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=loadClass;| Searching local repositories|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClass(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]

[#|2013-07-18T16:06:35.584+0200|FINER|glassfish3.1.2|javax.enterprise.system.container.web.org.glassfish.web.loader|_ThreadID=75;_ThreadName=Thread-2;ClassName=org.glassfish.web.loader.WebappClassLoader;MethodName=findClass;| findClassInternal(com.transat.ga2010.service.BanqueServiceImpl$$EnhancerByCGLIB$$fedb0661)|#]



Ed Bratt added a comment - 19/Jul/13 05:21 PM

Posting for Reporter. Clear virus scan.


Sanjeeb Sahoo added a comment - 22/Jul/13 07:45 PM

The following exception happens because WebappLoader is reflectively calling getClasspath method in all class loaders in the class loader chain and APIClassLoader does not have such a method. From what I see WebappLoader does not consider this to be an issue, as it logs the exception in FINEST level. So, I am assigning the issue to web container folks to figure out why session replication does not happen.

"[#|2013-07-18T16:00:02.639+0200|FINEST|glassfish3.1.2|org.apache.catalina.loader.WebappLoader|_ThreadID=10;_ThreadName=Thread-2;ClassName=null;MethodName=null;|getClasspath
java.lang.NoSuchMethodException: com.sun.enterprise.v3.server.APIClassLoaderServiceImpl$APIClassLoader.getClasspath()

Thanks,
Sahoo


Shing Wai Chan added a comment - 06/Aug/13 07:16 PM

Can you attach a test case for this?


alev50 added a comment - 12/Aug/13 08:47 PM

Not really, I deploy my production web app on a cluster, I stop the current instance and replication fails on a StreamCorruptedException. How could I write a simple test case for this ?

Thanks,

Anthony


Shing Wai Chan added a comment - 12/Aug/13 10:17 PM

We have a unit test for session replication. It is working fine. So, we need more information to debug this. Can you try to construct a unit test case?


Shing Wai Chan added a comment - 13/Aug/13 12:15 AM

Assign to Rajiv for session replication issue.