glassfish
  1. glassfish
  2. GLASSFISH-13881

[Blocking] On Failover, Conversation-based app in cluster shows deserialization issue due to CNFE

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.1_ms07
    • Component/s: cdi
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: Macintosh

    • Issuezilla Id:
      13,881

      Description

      Steps to reproduce:
      1. create tables in Derby in the APP Database - see attached sql flle
      2. create a cluster of 2 or more instances.
      3. start cluster
      4. create jdbc resource jdbc/sample using DerbyPool as the connection pool and create resource ref
      under the target cluster
      5. Deploy the attached ShoppingCart.war to the target cluster with availabilityenabled set to true.
      6. Access one of the instance's urls say :http://localhost:28080/ShoppingCart representing inst2
      7. Add some item to cart.
      8. Simulate failover by hopping over to another instance's port say,
      http://localhost:38080/ShoppingCart/faces/shop.xhtml?cid=1
      representing inst3
      9. Page does not load. Look in inst3's logs for the stack trace.

      The trace I see is pasted below :

      [#|2010-10-08T08:49:45.896-
      0700|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=15;_ThreadName=T
      hread-1;|PWC3989: An exception or error occurred in the container during the request processing
      org.jboss.weld.exceptions.WeldException: WELD-001500 Failed to deserialize proxy object
      at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:124)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at org.apache.catalina.session.StandardSession.readRemainingObject(StandardSession.java:1947)
      at org.apache.catalina.session.StandardSession.readObject(StandardSession.java:1855)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
      at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
      at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:527)
      at org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:482)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:
      392)
      at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:376)
      at org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:371)
      at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
      at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
      at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
      at org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
      at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
      at org.apache.catalina.connector.Request.doGetSession(Request.java:2832)
      at org.apache.catalina.connector.Request.getSession(Request.java:2559)
      at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
      at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:618)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:824)

      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:721)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1014)
      at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:220)
      at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:135)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:102)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:88)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:76)
      at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:53)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:57)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:69)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:530)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:511)
      at java.lang.Thread.run(Thread.java:637)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.weld.bean.proxy.util.SerializableProxy.readObject(SerializableProxy.java:120)
      ... 62 more
      Caused by: java.lang.ClassNotFoundException:
      org.jboss.weld.conversation.ConversationImpl_$$_WeldProxy
      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      at org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1499)
      at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1427)
      at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:723)
      at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73)
      at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:247)
      at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
      at com.sun.ejb.base.io.EJBObjectInputStream.resolveClass(EJBObjectInputStream.java:165)
      at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
      at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at
      org.jboss.weld.conversation.ConversationImpl_$$WeldProxy.deserializeProxy(ConversationImpl$$_Wel
      dProxy.java)
      ... 67 more

      #]

        Issue Links

          Activity

          Hide
          Sivakumar Thyagarajan added a comment -

          Analysis (thanks to Sahoo for his assistance) is complete and a workaround fix
          is in progress.

          Analysis:

          • Weld proxies are now loaded through a GlassFish "proxy classloader". This
            classloader search path includes the Weld bundle and the application
            classloader. The Weld bundle is included because Weld generated proxies
            reference some internal(read: not exported by the Weld osgi bundle) Weld
            implementation classes.
          • When the session is deserialized and a bean proxy is deserialized, the proxy
            Class is initially loaded through the GlassFish proxy classloader (through the
            ProxyServices SPI). During initialization of the proxy instance (readObject),
            EJBObjectInputStream while reading the class descriptor tries to resolve the
            bean proxy through the EJBObjectInputStream.resolveClass. The resolveClass
            implementation only checks the application classloader.

          A temproary fix would be for resolveClass to check the proxy classloader or fix
          using a fragment bundle approach. I am coming up with the fix with the latter
          approach

          Show
          Sivakumar Thyagarajan added a comment - Analysis (thanks to Sahoo for his assistance) is complete and a workaround fix is in progress. Analysis: Weld proxies are now loaded through a GlassFish "proxy classloader". This classloader search path includes the Weld bundle and the application classloader. The Weld bundle is included because Weld generated proxies reference some internal(read: not exported by the Weld osgi bundle) Weld implementation classes. When the session is deserialized and a bean proxy is deserialized, the proxy Class is initially loaded through the GlassFish proxy classloader (through the ProxyServices SPI). During initialization of the proxy instance (readObject), EJBObjectInputStream while reading the class descriptor tries to resolve the bean proxy through the EJBObjectInputStream.resolveClass. The resolveClass implementation only checks the application classloader. A temproary fix would be for resolveClass to check the proxy classloader or fix using a fragment bundle approach. I am coming up with the fix with the latter approach
          Hide
          Sivakumar Thyagarajan added a comment -

          A first stab at the fix is available. Requesting SQE to run the test again and
          let me know if the issue is still seen

          Show
          Sivakumar Thyagarajan added a comment - A first stab at the fix is available. Requesting SQE to run the test again and let me know if the issue is still seen
          Hide
          Sivakumar Thyagarajan added a comment -
              • Issue 14179 has been marked as a duplicate of this issue. ***
          Show
          Sivakumar Thyagarajan added a comment - Issue 14179 has been marked as a duplicate of this issue. ***
          Hide
          Sivakumar Thyagarajan added a comment -

          Fix confirmed to be solve this issue by SQE team. Getting fix reviewed and will
          commit.

          Show
          Sivakumar Thyagarajan added a comment - Fix confirmed to be solve this issue by SQE team. Getting fix reviewed and will commit.
          Hide
          Sivakumar Thyagarajan added a comment -

          Fixed as part of revision 42968.

          Two new weld-integration related fragments are introduced:
          1. weld-integration-fragment to fix issue 13881 [1]
          2. weld-integration-test-fragment to handle the use of weld internal packages by
          Weld arquillian tests (issue 13713).

          The weld-integration-fragment is always bundled with the glassfish-jcdi package,
          as we want those packages to be available for Weld generated proxies at runtime
          as described at [1]. The weld-integration-test-fragment is not added to jcdi
          package as that would be used only the SQE team and other developers who want to
          run the arquillian tests against GlassFish.

          [1]

          • detailed description of fix for 13881 -
            To fix issue 13882, the new implementation of ProxyServices SPI uses the thread
            context classloader (the application classloader) as the classloader for
            loading Weld-generated bean proxies. The classloader that loaded the Bean must
            be used to load and define the bean proxy to handle Beans with package-private
            constructor as discussed in WELD-737.

          Weld proxies today have references to some internal weld implementation classes
          such as javassist and org.jboss.weld.proxy.* packages. These classes are
          temporarily re-exported through the weld-integration-fragment bundle so that
          when the bean proxies when loaded using the application classloader will have
          visibility to these internal implementation classes.

          As a fix for WELD-737, Weld may use the Bean's classloader rather than asking
          the ProxyServices service implementation. Weld also plans to remove the
          dependencies of the bean proxy on internal implementation classes. When that
          happens we can remove the weld-integration-fragment workaround and the
          ProxyServices implementation.

          Show
          Sivakumar Thyagarajan added a comment - Fixed as part of revision 42968. Two new weld-integration related fragments are introduced: 1. weld-integration-fragment to fix issue 13881 [1] 2. weld-integration-test-fragment to handle the use of weld internal packages by Weld arquillian tests (issue 13713). The weld-integration-fragment is always bundled with the glassfish-jcdi package, as we want those packages to be available for Weld generated proxies at runtime as described at [1] . The weld-integration-test-fragment is not added to jcdi package as that would be used only the SQE team and other developers who want to run the arquillian tests against GlassFish. [1] detailed description of fix for 13881 - To fix issue 13882, the new implementation of ProxyServices SPI uses the thread context classloader (the application classloader) as the classloader for loading Weld-generated bean proxies. The classloader that loaded the Bean must be used to load and define the bean proxy to handle Beans with package-private constructor as discussed in WELD-737. Weld proxies today have references to some internal weld implementation classes such as javassist and org.jboss.weld.proxy.* packages. These classes are temporarily re-exported through the weld-integration-fragment bundle so that when the bean proxies when loaded using the application classloader will have visibility to these internal implementation classes. As a fix for WELD-737, Weld may use the Bean's classloader rather than asking the ProxyServices service implementation. Weld also plans to remove the dependencies of the bean proxy on internal implementation classes. When that happens we can remove the weld-integration-fragment workaround and the ProxyServices implementation.

            People

            • Assignee:
              Sivakumar Thyagarajan
              Reporter:
              shreedhar_ganapathy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: