Issue Details (XML | Word | Printable)

Key: GLASSFISH-13881
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Sivakumar Thyagarajan
Reporter: shreedhar_ganapathy
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
glassfish

[Blocking] On Failover, Conversation-based app in cluster shows deserialization issue due to CNFE

Created: 08/Oct/10 09:06 AM   Updated: 26/Nov/10 12:16 AM   Resolved: 19/Nov/10 07:47 AM
Component/s: cdi
Affects Version/s: 3.1
Fix Version/s: 3.1_ms07

Time Tracking:
Not Specified

File Attachments: 1. Text File server.log (102 kB) 08/Nov/10 04:21 AM - Sreekanth
2. Text File server.log (53 kB) 08/Oct/10 09:08 AM - shreedhar_ganapathy
3. File ShoppingCart.war (28 kB) 08/Oct/10 09:08 AM - shreedhar_ganapathy
4. Text File ShoppingCartTables.sql (0.6 kB) 08/Oct/10 09:11 AM - shreedhar_ganapathy

Environment:

Operating System: All
Platform: Macintosh

Issue Links:
Dependency
 

Issuezilla Id: 13,881
Tags:
Participants: shreedhar_ganapathy, Sivakumar Thyagarajan, sonymanuel and Sreekanth


 Description  « Hide

Steps to reproduce:
1. create tables in Derby in the APP Database - see attached sql flle
2. create a cluster of 2 or more instances.
3. start cluster
4. create jdbc resource jdbc/sample using DerbyPool as the connection pool and create resource ref
under the target cluster
5. Deploy the attached ShoppingCart.war to the target cluster with availabilityenabled set to true.
6. Access one of the instance's urls say :http://localhost:28080/ShoppingCart representing inst2
7. Add some item to cart.
8. Simulate failover by hopping over to another instance's port say,
http://localhost:38080/ShoppingCart/faces/shop.xhtml?cid=1
representing inst3
9. Page does not load. Look in inst3's logs for the stack trace.

The trace I see is pasted below :

[#|2010-10-08T08:49:45.896-
0700|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=15;_ThreadName=T
hread-1;|PWC3989: An exception or error occurred in the container during the request processing
org.jboss.weld.exceptions.WeldException: WELD-001500 Failed to deserialize proxy object
at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at org.apache.catalina.session.StandardSession.readRemainingObject(StandardSession.java:1947)
at org.apache.catalina.session.StandardSession.readObject(StandardSession.java:1855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:527)
at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:482)
at
org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:
392)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:376)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:371)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
at org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2832)
at org.apache.catalina.connector.Request.getSession(Request.java:2559)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:618)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:824)

at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:721)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1014)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:220)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:530)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:511)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:120)
... 62 more
Caused by: java.lang.ClassNotFoundException:
org.jboss.weld.conversation.ConversationImpl_$$_WeldProxy
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1499)
at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1427)
at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:723)
at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at com.sun.ejb.base.io.EJBObjectInputStream.resolveClass(EJBObjectInputStream.java:165)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.jboss.weld.conversation.ConversationImpl_$$WeldProxy.deserializeProxy(ConversationImpl$$_Wel
dProxy.java)
... 67 more

#]


shreedhar_ganapathy added a comment - 08/Oct/10 09:08 AM

Created an attachment (id=5100)
ShoppingCart App


shreedhar_ganapathy added a comment - 08/Oct/10 09:08 AM

Created an attachment (id=5101)
Inst3 server log


shreedhar_ganapathy added a comment - 08/Oct/10 09:11 AM

Created an attachment (id=5102)
SQL file to create tables in APP schema


shreedhar_ganapathy added a comment - 08/Oct/10 09:16 AM

Sources shared with Siva. Issue filed for tracking purposes.


Sivakumar Thyagarajan added a comment - 13/Oct/10 08:38 AM

Need to investigate this in light of the new proxy serialization SPIs.
Targetting this for MS7.


shreedhar_ganapathy added a comment - 15/Oct/10 10:02 AM

This scenario is being added to HA functional testing and hence needs a fix by MS06.
Siva, could you change the target milestone to MS6 and prioritize on this issue?
This issue may be masking other issues that we would like to ferret out early.


shreedhar_ganapathy added a comment - 15/Oct/10 10:10 AM

added cc


shreedhar_ganapathy added a comment - 15/Oct/10 10:11 AM

cc Sudipa


sonymanuel added a comment - 24/Oct/10 11:22 PM

Unable to try this scenario due to issue 14179.


Sreekanth added a comment - 08/Nov/10 04:20 AM

I tried executing the same test with the patch jars provided by Siva.Now I get
the below exception.

Exception
==========

[#|2010-11-
08T15:38:23.684+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;|java.lang.RuntimeException: java.lang.NullPointerException|#]

[#|2010-11-
08T15:38:23.685+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:211)|#]

[#|2010-11-
08T15:38:23.685+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.blc.EmbeddedBrokerRunner.start(EmbeddedBrokerRunner.java:3
31)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:454)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:370)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJm
sResourceAdapter.java:360)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at java.security.AccessController.doPrivileged(Native Method)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:353)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundR
esourceAdapter.java:129)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(Acti
veInboundResourceAdapterImpl.java:90)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(A
ctiveRAFactory.java:135)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(Active
RAFactory.java:106)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:210)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:344)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(Conne
ctorRuntime.java:350)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(J
msProviderLifecycle.java:110)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsP
roviderLifecycle.java:93)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:128)|#]

[#|2010-11-
08T15:38:23.690+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:88)|#]

[#|2010-11-
08T15:38:23.698+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:79)|#]

[#|2010-11-
08T15:38:23.698+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:64)|#]

[#|2010-11-
08T15:38:23.699+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.ja
va:136)|#]

[#|2010-11-
08T15:38:23.700+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:73)

#]

[#|2010-11-
08T15:38:23.700+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)|#]

[#|2010-11-
08T15:38:23.701+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)|#
]

[#|2010-11-
08T15:38:23.701+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:78
)|#]

[#|2010-11-
08T15:38:23.702+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMa
in.java:119)|#]

[#|2010-11-
08T15:38:23.702+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)|#]

[#|2010-11-
08T15:38:23.703+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)|#]

[#|2010-11-
08T15:38:23.703+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)|#]

[#|2010-11-
08T15:38:23.704+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at java.lang.reflect.Method.invoke(Method.java:597)|#]

[#|2010-11-
08T15:38:23.709+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:99)

#]

[#|2010-11-
08T15:38:23.709+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)|#]

[#|2010-11-
08T15:38:23.710+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;|Caused by: java.lang.NullPointerException|#]

[#|2010-11-
08T15:38:23.711+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.data.TransactionList.<init>
(TransactionList.java:174)|#]

[#|2010-11-
08T15:38:23.711+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:1166)|#]

[#|2010-11-
08T15:38:23.712+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:453)|#]

[#|2010-11-
08T15:38:23.712+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsserver.BrokerProcess.start(BrokerProcess.java:164)|#]

[#|2010-11-
08T15:38:23.713+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsserver.DirectBrokerProcess.start(DirectBrokerProcess.ja
va:92)|#]

[#|2010-11-
08T15:38:23.713+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:206)|#]

[#|2010-11-
08T15:38:23.714+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| ... 31 more|#]

[#|2010-11-
08T15:38:23.714+0530|SEVERE|glassfish3.1|javax.enterprise.resource.resourceadapt
er.com.sun.enterprise.connectors|_ThreadID=16;_ThreadName=Thread-1;|RAR6035 :
Resource adapter start failed : {0}
javax.resource.spi.ResourceAdapterInternalException:
java.security.PrivilegedActionException:
javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:369)
at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundR
esourceAdapter.java:129)
at
com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(Acti
veInboundResourceAdapterImpl.java:90)
at
com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(A
ctiveRAFactory.java:135)
at
com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(Active
RAFactory.java:106)
at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:210)
at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:344)
at
com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(Conne
ctorRuntime.java:350)
at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(J
msProviderLifecycle.java:110)
at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsP
roviderLifecycle.java:93)
at
com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:128)
at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:88)
at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:79)
at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:64)
at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.ja
va:136)
at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:73)
at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)
at
com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:78
)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMa
in.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:99)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.security.PrivilegedActionException:
javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at java.security.AccessController.doPrivileged(Native Method)
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:353)
... 25 more
Caused by: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:458)
at
com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:370)
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJm
sResourceAdapter.java:360)
... 27 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:211)
at
com.sun.messaging.jms.blc.EmbeddedBrokerRunner.start(EmbeddedBrokerRunner.java:3
31)
at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:454)
... 29 more
Caused by: java.lang.NullPointerException
at com.sun.messaging.jmq.jmsserver.data.TransactionList.<init>
(TransactionList.java:174)
at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:1166)
at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:453)
at
com.sun.messaging.jmq.jmsserver.BrokerProcess.start(BrokerProcess.java:164)
at
com.sun.messaging.jmq.jmsserver.DirectBrokerProcess.start(DirectBrokerProcess.ja
va:92)
at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:206)
... 31 more


Sreekanth added a comment - 08/Nov/10 04:21 AM

Created an attachment (id=5365)
Server log after patching with jars provided by Siva


sonymanuel added a comment - 08/Nov/10 06:26 PM

Marking as blocking. This blocks validating CDI failover cases.

The core issue is deserialization. Please ignore the NPE from MQ startup. See
log Sreekanth attached for more details.

[#|2010-11-08T15:38:14.264+0530|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=16;_ThreadName=Thread-1;|PWC3989:
An exception or error occurred in the container during the request processing
org.jboss.weld.exceptions.WeldException: WELD-001500 Failed to deserialize proxy
object
at
org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.apache.catalina.session.StandardSession.readRemainingObject(StandardSession.java:1947)
at
org.apache.catalina.session.StandardSession.readObject(StandardSession.java:1855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
at
org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:524)
at
org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:476)
at
org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:394)
at
org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:377)
at
org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:372)
at
org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
at
org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
at
org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
at
org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2833)
at org.apache.catalina.connector.Request.getSession(Request.java:2560)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:623)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:119)
... 62 more
Caused by: java.lang.ClassNotFoundException:
server.org$jboss$weld$bean-$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$-ManagedBean-class_server$ShoppingCart_$$_WeldProxy
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1551)
at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1473)
at
org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:735)
at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:72)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1765)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at
com.sun.ejb.base.io.EJBObjectInputStream.resolveClass(EJBObjectInputStream.java:163)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
server.org$jboss$weld$bean-$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$ManagedBean-class_server$ShoppingCart_$$WeldProxy.deserializeProxy(org$jboss$weld$bean$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$-ManagedBean-class_server$ShoppingCart$$_WeldProxy.java)
... 67 more

#]

Sivakumar Thyagarajan added a comment - 10/Nov/10 03:48 AM

Analysis (thanks to Sahoo for his assistance) is complete and a workaround fix
is in progress.

Analysis:

  • Weld proxies are now loaded through a GlassFish "proxy classloader". This
    classloader search path includes the Weld bundle and the application
    classloader. The Weld bundle is included because Weld generated proxies
    reference some internal(read: not exported by the Weld osgi bundle) Weld
    implementation classes.
  • When the session is deserialized and a bean proxy is deserialized, the proxy
    Class is initially loaded through the GlassFish proxy classloader (through the
    ProxyServices SPI). During initialization of the proxy instance (readObject),
    EJBObjectInputStream while reading the class descriptor tries to resolve the
    bean proxy through the EJBObjectInputStream.resolveClass. The resolveClass
    implementation only checks the application classloader.

A temproary fix would be for resolveClass to check the proxy classloader or fix
using a fragment bundle approach. I am coming up with the fix with the latter
approach


Sivakumar Thyagarajan added a comment - 15/Nov/10 03:45 AM

A first stab at the fix is available. Requesting SQE to run the test again and
let me know if the issue is still seen


Sivakumar Thyagarajan added a comment - 15/Nov/10 08:33 AM
      • Issue 14179 has been marked as a duplicate of this issue. ***

Sivakumar Thyagarajan added a comment - 16/Nov/10 08:51 AM

Fix confirmed to be solve this issue by SQE team. Getting fix reviewed and will
commit.


Sivakumar Thyagarajan added a comment - 19/Nov/10 07:47 AM

Fixed as part of revision 42968.

Two new weld-integration related fragments are introduced:
1. weld-integration-fragment to fix issue 13881 [1]
2. weld-integration-test-fragment to handle the use of weld internal packages by
Weld arquillian tests (issue 13713).

The weld-integration-fragment is always bundled with the glassfish-jcdi package,
as we want those packages to be available for Weld generated proxies at runtime
as described at [1]. The weld-integration-test-fragment is not added to jcdi
package as that would be used only the SQE team and other developers who want to
run the arquillian tests against GlassFish.

[1]

  • detailed description of fix for 13881 -
    To fix issue 13882, the new implementation of ProxyServices SPI uses the thread
    context classloader (the application classloader) as the classloader for
    loading Weld-generated bean proxies. The classloader that loaded the Bean must
    be used to load and define the bean proxy to handle Beans with package-private
    constructor as discussed in WELD-737.

Weld proxies today have references to some internal weld implementation classes
such as javassist and org.jboss.weld.proxy.* packages. These classes are
temporarily re-exported through the weld-integration-fragment bundle so that
when the bean proxies when loaded using the application classloader will have
visibility to these internal implementation classes.

As a fix for WELD-737, Weld may use the Bean's classloader rather than asking
the ProxyServices service implementation. Weld also plans to remove the
dependencies of the bean proxy on internal implementation classes. When that
happens we can remove the weld-integration-fragment workaround and the
ProxyServices implementation.