[GLASSFISH-13881] [Blocking] On Failover, Conversation-based app in cluster shows deserialization issue due to CNFE Created: 08/Oct/10  Updated: 26/Nov/10  Resolved: 19/Nov/10

Status: Resolved
Project: glassfish
Component/s: cdi
Affects Version/s: 3.1
Fix Version/s: 3.1_ms07

Type: Bug Priority: Critical
Reporter: shreedhar_ganapathy Assignee: Sivakumar Thyagarajan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: Macintosh


Attachments: Text File server.log     Text File server.log     File ShoppingCart.war     Text File ShoppingCartTables.sql    
Issue Links:
Dependency
depends on GLASSFISH-14179 NPE in org.glassfish.web.ha.session.m... Resolved
Issuezilla Id: 13,881

 Description   

Steps to reproduce:
1. create tables in Derby in the APP Database - see attached sql flle
2. create a cluster of 2 or more instances.
3. start cluster
4. create jdbc resource jdbc/sample using DerbyPool as the connection pool and create resource ref
under the target cluster
5. Deploy the attached ShoppingCart.war to the target cluster with availabilityenabled set to true.
6. Access one of the instance's urls say :http://localhost:28080/ShoppingCart representing inst2
7. Add some item to cart.
8. Simulate failover by hopping over to another instance's port say,
http://localhost:38080/ShoppingCart/faces/shop.xhtml?cid=1
representing inst3
9. Page does not load. Look in inst3's logs for the stack trace.

The trace I see is pasted below :

[#|2010-10-08T08:49:45.896-
0700|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=15;_ThreadName=T
hread-1;|PWC3989: An exception or error occurred in the container during the request processing
org.jboss.weld.exceptions.WeldException: WELD-001500 Failed to deserialize proxy object
at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at org.apache.catalina.session.StandardSession.readRemainingObject(StandardSession.java:1947)
at org.apache.catalina.session.StandardSession.readObject(StandardSession.java:1855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:527)
at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:482)
at
org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:
392)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:376)
at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:371)
at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
at org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2832)
at org.apache.catalina.connector.Request.getSession(Request.java:2559)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:618)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:824)

at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:721)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1014)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:220)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:530)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:511)
at java.lang.Thread.run(Thread.java:637)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:120)
... 62 more
Caused by: java.lang.ClassNotFoundException:
org.jboss.weld.conversation.ConversationImpl_$$_WeldProxy
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1499)
at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1427)
at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:723)
at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at com.sun.ejb.base.io.EJBObjectInputStream.resolveClass(EJBObjectInputStream.java:165)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.jboss.weld.conversation.ConversationImpl_$$WeldProxy.deserializeProxy(ConversationImpl$$_Wel
dProxy.java)
... 67 more

#]


 Comments   
Comment by shreedhar_ganapathy [ 08/Oct/10 ]

Created an attachment (id=5100)
ShoppingCart App

Comment by shreedhar_ganapathy [ 08/Oct/10 ]

Created an attachment (id=5101)
Inst3 server log

Comment by shreedhar_ganapathy [ 08/Oct/10 ]

Created an attachment (id=5102)
SQL file to create tables in APP schema

Comment by shreedhar_ganapathy [ 08/Oct/10 ]

Sources shared with Siva. Issue filed for tracking purposes.

Comment by Sivakumar Thyagarajan [ 13/Oct/10 ]

Need to investigate this in light of the new proxy serialization SPIs.
Targetting this for MS7.

Comment by shreedhar_ganapathy [ 15/Oct/10 ]

This scenario is being added to HA functional testing and hence needs a fix by MS06.
Siva, could you change the target milestone to MS6 and prioritize on this issue?
This issue may be masking other issues that we would like to ferret out early.

Comment by shreedhar_ganapathy [ 15/Oct/10 ]

added cc

Comment by shreedhar_ganapathy [ 15/Oct/10 ]

cc Sudipa

Comment by sonymanuel [ 24/Oct/10 ]

Unable to try this scenario due to issue 14179.

Comment by Sreekanth [ 08/Nov/10 ]

I tried executing the same test with the patch jars provided by Siva.Now I get
the below exception.

Exception
==========

[#|2010-11-
08T15:38:23.684+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;|java.lang.RuntimeException: java.lang.NullPointerException|#]

[#|2010-11-
08T15:38:23.685+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:211)|#]

[#|2010-11-
08T15:38:23.685+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.blc.EmbeddedBrokerRunner.start(EmbeddedBrokerRunner.java:3
31)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:454)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:370)|#]

[#|2010-11-
08T15:38:23.686+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJm
sResourceAdapter.java:360)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at java.security.AccessController.doPrivileged(Native Method)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:353)|#]

[#|2010-11-
08T15:38:23.687+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundR
esourceAdapter.java:129)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(Acti
veInboundResourceAdapterImpl.java:90)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(A
ctiveRAFactory.java:135)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(Active
RAFactory.java:106)|#]

[#|2010-11-
08T15:38:23.688+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:210)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:344)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(Conne
ctorRuntime.java:350)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(J
msProviderLifecycle.java:110)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsP
roviderLifecycle.java:93)|#]

[#|2010-11-
08T15:38:23.689+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:128)|#]

[#|2010-11-
08T15:38:23.690+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:88)|#]

[#|2010-11-
08T15:38:23.698+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:79)|#]

[#|2010-11-
08T15:38:23.698+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:64)|#]

[#|2010-11-
08T15:38:23.699+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.ja
va:136)|#]

[#|2010-11-
08T15:38:23.700+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:73)

#]

[#|2010-11-
08T15:38:23.700+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)|#]

[#|2010-11-
08T15:38:23.701+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)|#
]

[#|2010-11-
08T15:38:23.701+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:78
)|#]

[#|2010-11-
08T15:38:23.702+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMa
in.java:119)|#]

[#|2010-11-
08T15:38:23.702+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)|#]

[#|2010-11-
08T15:38:23.703+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)|#]

[#|2010-11-
08T15:38:23.703+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)|#]

[#|2010-11-
08T15:38:23.704+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at java.lang.reflect.Method.invoke(Method.java:597)|#]

[#|2010-11-
08T15:38:23.709+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:99)

#]

[#|2010-11-
08T15:38:23.709+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)|#]

[#|2010-11-
08T15:38:23.710+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;|Caused by: java.lang.NullPointerException|#]

[#|2010-11-
08T15:38:23.711+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.data.TransactionList.<init>
(TransactionList.java:174)|#]

[#|2010-11-
08T15:38:23.711+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:1166)|#]

[#|2010-11-
08T15:38:23.712+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:453)|#]

[#|2010-11-
08T15:38:23.712+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsserver.BrokerProcess.start(BrokerProcess.java:164)|#]

[#|2010-11-
08T15:38:23.713+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsserver.DirectBrokerProcess.start(DirectBrokerProcess.ja
va:92)|#]

[#|2010-11-
08T15:38:23.713+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:206)|#]

[#|2010-11-
08T15:38:23.714+0530|SEVERE|glassfish3.1|null|_ThreadID=16;_ThreadName=Thread-
1;| ... 31 more|#]

[#|2010-11-
08T15:38:23.714+0530|SEVERE|glassfish3.1|javax.enterprise.resource.resourceadapt
er.com.sun.enterprise.connectors|_ThreadID=16;_ThreadName=Thread-1;|RAR6035 :
Resource adapter start failed :

{0}

javax.resource.spi.ResourceAdapterInternalException:
java.security.PrivilegedActionException:
javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:369)
at
com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundR
esourceAdapter.java:129)
at
com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(Acti
veInboundResourceAdapterImpl.java:90)
at
com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(A
ctiveRAFactory.java:135)
at
com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(Active
RAFactory.java:106)
at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:210)
at
com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActi
veResourceAdapter(ResourceAdapterAdminServiceImpl.java:344)
at
com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(Conne
ctorRuntime.java:350)
at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(J
msProviderLifecycle.java:110)
at
com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsP
roviderLifecycle.java:93)
at
com.sun.hk2.component.AbstractWombImpl.inject(AbstractWombImpl.java:128)
at
com.sun.hk2.component.ConstructorWomb.initialize(ConstructorWomb.java:88)
at com.sun.hk2.component.AbstractWombImpl.get(AbstractWombImpl.java:79)
at
com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:64)
at
com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.ja
va:136)
at
com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:73)
at
com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)
at
com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:78
)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMa
in.java:119)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:99)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.security.PrivilegedActionException:
javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at java.security.AccessController.doPrivileged(Native Method)
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceA
dapter(ActiveJmsResourceAdapter.java:353)
... 25 more
Caused by: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001:
start:Aborting:Exception starting EMBEDDED broker=java.lang.NullPointerException
at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:458)
at
com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:370)
at
com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJm
sResourceAdapter.java:360)
... 27 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:211)
at
com.sun.messaging.jms.blc.EmbeddedBrokerRunner.start(EmbeddedBrokerRunner.java:3
31)
at
com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.ja
va:454)
... 29 more
Caused by: java.lang.NullPointerException
at com.sun.messaging.jmq.jmsserver.data.TransactionList.<init>
(TransactionList.java:174)
at com.sun.messaging.jmq.jmsserver.Broker._start(Broker.java:1166)
at com.sun.messaging.jmq.jmsserver.Broker.start(Broker.java:453)
at
com.sun.messaging.jmq.jmsserver.BrokerProcess.start(BrokerProcess.java:164)
at
com.sun.messaging.jmq.jmsserver.DirectBrokerProcess.start(DirectBrokerProcess.ja
va:92)
at
com.sun.messaging.jmq.jmsclient.runtime.impl.BrokerInstanceImpl.start(BrokerInst
anceImpl.java:206)
... 31 more

Comment by Sreekanth [ 08/Nov/10 ]

Created an attachment (id=5365)
Server log after patching with jars provided by Siva

Comment by sonymanuel [ 08/Nov/10 ]

Marking as blocking. This blocks validating CDI failover cases.

The core issue is deserialization. Please ignore the NPE from MQ startup. See
log Sreekanth attached for more details.

[#|2010-11-08T15:38:14.264+0530|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=16;_ThreadName=Thread-1;|PWC3989:
An exception or error occurred in the container during the request processing
org.jboss.weld.exceptions.WeldException: WELD-001500 Failed to deserialize proxy
object
at
org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.apache.catalina.session.StandardSession.readRemainingObject(StandardSession.java:1947)
at
org.apache.catalina.session.StandardSession.readObject(StandardSession.java:1855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
at
org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:524)
at
org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:476)
at
org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:394)
at
org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:377)
at
org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:372)
at
org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
at
org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
at
org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
at
org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
at org.apache.catalina.connector.Request.doGetSession(Request.java:2833)
at org.apache.catalina.connector.Request.getSession(Request.java:2560)
at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:623)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
at
com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
at
com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at
com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at
com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at
com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:119)
... 62 more
Caused by: java.lang.ClassNotFoundException:
server.org$jboss$weld$bean-$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$-ManagedBean-class_server$ShoppingCart_$$_WeldProxy
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at
org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1551)
at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1473)
at
org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:735)
at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:72)
at
org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1765)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at
com.sun.ejb.base.io.EJBObjectInputStream.resolveClass(EJBObjectInputStream.java:163)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at
server.org$jboss$weld$bean-$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$ManagedBean-class_server$ShoppingCart_$$WeldProxy.deserializeProxy(org$jboss$weld$bean$space$Sreekanth$servers$glassfish3$glassfish$nodes$agent1$instance101$applications$ShoppingCart$-ManagedBean-class_server$ShoppingCart$$_WeldProxy.java)
... 67 more

#]
Comment by Sivakumar Thyagarajan [ 10/Nov/10 ]

Analysis (thanks to Sahoo for his assistance) is complete and a workaround fix
is in progress.

Analysis:

  • Weld proxies are now loaded through a GlassFish "proxy classloader". This
    classloader search path includes the Weld bundle and the application
    classloader. The Weld bundle is included because Weld generated proxies
    reference some internal(read: not exported by the Weld osgi bundle) Weld
    implementation classes.
  • When the session is deserialized and a bean proxy is deserialized, the proxy
    Class is initially loaded through the GlassFish proxy classloader (through the
    ProxyServices SPI). During initialization of the proxy instance (readObject),
    EJBObjectInputStream while reading the class descriptor tries to resolve the
    bean proxy through the EJBObjectInputStream.resolveClass. The resolveClass
    implementation only checks the application classloader.

A temproary fix would be for resolveClass to check the proxy classloader or fix
using a fragment bundle approach. I am coming up with the fix with the latter
approach

Comment by Sivakumar Thyagarajan [ 15/Nov/10 ]

A first stab at the fix is available. Requesting SQE to run the test again and
let me know if the issue is still seen

Comment by Sivakumar Thyagarajan [ 15/Nov/10 ]
      • Issue 14179 has been marked as a duplicate of this issue. ***
Comment by Sivakumar Thyagarajan [ 16/Nov/10 ]

Fix confirmed to be solve this issue by SQE team. Getting fix reviewed and will
commit.

Comment by Sivakumar Thyagarajan [ 19/Nov/10 ]

Fixed as part of revision 42968.

Two new weld-integration related fragments are introduced:
1. weld-integration-fragment to fix issue 13881 [1]
2. weld-integration-test-fragment to handle the use of weld internal packages by
Weld arquillian tests (issue 13713).

The weld-integration-fragment is always bundled with the glassfish-jcdi package,
as we want those packages to be available for Weld generated proxies at runtime
as described at [1]. The weld-integration-test-fragment is not added to jcdi
package as that would be used only the SQE team and other developers who want to
run the arquillian tests against GlassFish.

[1]

  • detailed description of fix for 13881 -
    To fix issue 13882, the new implementation of ProxyServices SPI uses the thread
    context classloader (the application classloader) as the classloader for
    loading Weld-generated bean proxies. The classloader that loaded the Bean must
    be used to load and define the bean proxy to handle Beans with package-private
    constructor as discussed in WELD-737.

Weld proxies today have references to some internal weld implementation classes
such as javassist and org.jboss.weld.proxy.* packages. These classes are
temporarily re-exported through the weld-integration-fragment bundle so that
when the bean proxies when loaded using the application classloader will have
visibility to these internal implementation classes.

As a fix for WELD-737, Weld may use the Bean's classloader rather than asking
the ProxyServices service implementation. Weld also plans to remove the
dependencies of the bean proxy on internal implementation classes. When that
happens we can remove the weld-integration-fragment workaround and the
ProxyServices implementation.

Generated at Mon Jun 29 23:35:55 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.