glassfish
  1. glassfish
  2. GLASSFISH-14372

[BLOCKING] java.io.StreamCorruptedException when deserializing session on another instance.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.1_ms07
    • Component/s: web_container
    • Labels:
      None
    • Environment:

      Operating System: other
      Platform: Linux

    • Issuezilla Id:
      14,372

      Description

      When running SFSB tests lot test scenarios fail with
      java.io.StreamCorruptedException when loading/deserializing the session from
      backing store. This error seems to occur on the QE setup where as on the DEV
      setup it works. Analysing this a bit I think the error on the QE setup is
      probably a combination of platform + JDK.

      The basic scenario is

      1. Send a few http requests to instance1
      2. Fail instance1
      3. Send another http request to instance2
      4. Now instance2 will load the session from replica.
      5. In the process of deserializing the session we get the StreamCorruptedException.

      See the attached logs (testTXCPDriver.zip). This has shoal cache log level set
      to FINE and shows steps 1 - 4 goes fine. The session is properly replicated. On
      failover the replica data is correctly loaded. At the point of deserializing
      this fails.
      [#|2010-11-03T07:27:32.340+0000|FINE|glassfish3.1|org.shoal.ha.cache.command.load_request|_ThreadID=15;_ThreadName=Thread-1;ClassName=org.shoal.adapter.store.commands.LoadRequestCommand;MethodName=writeObject;|instance102LoadRequestCommand:35
      sending load_request command for 0a1c66818654ae4b30870e1aa70bto instance108|#]

      [#|2010-11-03T07:27:32.665+0000|FINE|glassfish3.1|org.shoal.ha.cache.command.load_response|_ThreadID=15;_ThreadName=Thread-1;ClassName=org.shoal.adapter.store.commands.LoadResponseCommand;MethodName=execute;|instance102
      received load_response key=0a1c66818654ae4b30870e1aa70b;
      value=SimpleMetadata

      {version=4, lastAccessTime=1288769230862, maxInactiveInterval=1800, state.length=1470, state=SimpleMetadata->state-84_-19_0_5_115_114_0_53_111_114_103_46_103_108_97_115_115_102_105_115_104_46_119_101_98_46_104_97_46_115_101_115_115_105_111_110_46_109_97_110_97_103_101_109_101_110_116_46_70_117_108_108_72_65_83_101_115_115_105_111_110_72_76_71_-125_-73_120_111_24_2_0_0_119_8_0_0_0_0_0_0_0_-93_120_114_0_53_111_114_103_46_103_108_97_115_115_102_105_115_104_46_119_101_98_46_104_97_46_115_101_115_115_105_111_110_46_109_97_110_97_103_101_109_101_110_116_46_66_97_115_101_72_65_83_101_115_115_105_111_110_-22_32_75_-60_-102_-8_-61_-40_3_0_2_90_0_14_112_101_114_115_105_115_116_101_110_116_70_108_97_103_76_0_8_117_115_101_114_78_97_109_101_116_0_18_76_106_97_118_97_47_108_97_110_103_47_83_116_114_105_110_103_59_119_8_0_0_0_0_0_0_0_-93_120_114_0_43_111_114_103_46_97_112_97_99_104_101_46_99_97_116_97_108_105_110_97_46_115_101_115_115_105_111_110_46_83_116_97_110_100_97_114_100_83_101_115_115_105_111_110_126_62_89_94_-109_-97_117_102_3_0_12_74_0_12_99_114_101_97_116_105_111_110_84_105_109_101_90_0_5_105_115_78_101_119_90_0_7_105_115_86_97_108_105_100_74_0_16_108_97_115_116_65_99_99_101_115_115_101_100_84_105_109_101_73_0_19_109_97_120_73_110_97_99_116_105_118_101_73_110_116_101_114_118_97_108_74_0_16_116_104_105_115_65_99_99_101_115_115_101_100_84_105_109_101_76_0_10_97_116_116_114_105_98_117_116_101_115_116_0_15_76_106_97_118_97_47_117_116_105_108_47_77_97_112_59_76_0_5_98_101_75_101_121_113_0_126_0_2_76_0_2_105_100_113_0_126_0_2_76_0_15_115_105_112_65_112_112_83_101_115_115_105_111_110_73_100_113_0_126_0_2_76_0_5_115_115_111_73_100_113_0_126_0_2_76_0_7_118_101_114_115_105_111_110_116_0_40_76_106_97_118_97_47_117_116_105_108_47_99_111_110_99_117_114_114_101_110_116_47_97_116_111_109_105_99_47_65_116_111_109_105_99_76_111_110_103_59_119_8_0_0_0_0_0_0_0_-29_120_112_115_114_0_15_106_97_118_97_46_108_97_110_103_46_83_104_111_114_116_104_77_55_19_52_96_-38_82_2_0_1_83_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_114_0_16_106_97_118_97_46_108_97_110_103_46_78_117_109_98_101_114_-122_-84_-107_29_11_-108_-32_-117_2_0_0_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_112_0_1_115_114_0_14_106_97_118_97_46_108_97_110_103_46_76_111_110_103_59_-117_-28_-112_-52_-113_35_-33_2_0_1_74_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_113_0_126_0_8_0_0_1_44_16_-95_-58_104_115_113_0_126_0_10_0_0_1_44_16_-95_-40_14_115_114_0_17_106_97_118_97_46_108_97_110_103_46_73_110_116_101_103_101_114_18_-30_-96_-92_-9_-127_-121_56_2_0_1_73_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_113_0_126_0_8_0_0_7_8_115_114_0_17_106_97_118_97_46_108_97_110_103_46_66_111_111_108_101_97_110_-51_32_114_-128_-43_-100_-6_-18_2_0_1_90_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_112_0_115_113_0_126_0_15_1_115_113_0_126_0_10_0_0_1_44_16_-95_-36_19_116_0_28_48_97_49_99_54_54_56_49_56_54_53_52_97_101_52_98_51_48_56_55_48_101_49_97_97_55_48_98_115_113_0_126_0_13_0_0_0_3_116_0_6_83_70_83_66_95_49_115_114_0_94_99_111_109_46_115_117_110_46_101_106_98_46_99_111_110_116_97_105_110_101_114_115_46_69_74_66_76_111_99_97_108_79_98_106_101_99_116_73_110_118_111_99_97_116_105_111_110_72_97_110_100_108_101_114_68_101_108_101_103_97_116_101_36_83_101_114_105_97_108_105_122_97_98_108_101_76_111_99_97_108_79_98_106_101_99_116_68_101_108_101_103_97_116_101_-122_-89_-78_26_-97_-128_-117_34_2_0_5_74_0_11_99_111_110_116_97_105_110_101_114_73_100_90_0_27_105_115_79_112_116_105_111_110_97_108_76_111_99_97_108_66_117_115_105_110_101_115_115_86_105_101_119_74_0_7_118_101_114_115_105_111_110_76_0_13_105_110_116_102_67_108_97_115_115_78_97_109_101_113_0_126_0_2_76_0_10_112_114_105_109_97_114_121_75_101_121_116_0_18_76_106_97_118_97_47_108_97_110_103_47_79_98_106_101_99_116_59_119_8_0_0_0_0_0_0_0_34_120_112_1_44_16_-112_-76_-10_0_1_1_0_0_0_0_0_0_0_11_116_0_67_99_111_109_46_115_117_110_46_103_108_97_115_115_102_105_115_104_46_115_102_115_98_46_95_95_69_74_66_51_49_95_71_101_110_101_114_97_116_101_100_95_95_83_105_109_112_108_101_83_101_115_115_105_111_110_66_101_97_110_95_95_73_110_116_102_95_95_115_114_0_62_99_111_109_46_115_117_110_46_101_106_98_46_98_97_115_101_46_115_102_115_98_46_117_116_105_108_46_83_105_109_112_108_101_75_101_121_71_101_110_101_114_97_116_111_114_36_83_105_109_112_108_101_83_101_115_115_105_111_110_75_101_121_15_37_-8_-113_-64_94_31_37_2_0_3_73_0_2_105_100_74_0_6_112_114_101_102_105_120_74_0_6_115_117_102_102_105_120_119_8_0_0_0_0_0_0_0_34_120_112_0_0_0_1_31_0_-112_10_16_-95_-69_45_-18_0_5_-36_92_71_107_57_116_0_6_83_70_83_66_95_50_115_113_0_126_0_22_1_44_16_-112_-76_-10_0_0_1_0_0_0_0_0_0_0_4_116_0_66_99_111_109_46_115_117_110_46_103_108_97_115_115_102_105_115_104_46_115_102_115_98_46_95_95_69_74_66_51_49_95_71_101_110_101_114_97_116_101_100_95_95_67_104_101_99_107_112_111_105_110_116_101_100_66_101_97_110_95_95_73_110_116_102_95_95_115_113_0_126_0_26_0_0_0_1_31_0_-112_10_16_-95_-69_10_-18_0_5_-36_13_109_122_-92_116_0_7_105_110_116_65_116_116_114_115_113_0_126_0_13_0_0_0_5_112_112_120_116_0_0_120_}

      ;
      from instance108|#]

      [#|2010-11-03T07:27:32.695+0000|FINE|glassfish3.1|org.shoal.ha.cache.command.load_request|_ThreadID=15;_ThreadName=Thread-1;ClassName=org.shoal.ha.cache.impl.store.ReplicatedDataStore;MethodName=get;|/SFSBDriver:
      load(0a1c66818654ae4b30870e1aa70b) Final result: SimpleMetadata

      {version=4, lastAccessTime=1288769230862, maxInactiveInterval=1800, state.length=1470, state=SimpleMetadata->state-84_-19_0_5_115_114_0_53_111_114_103_46_103_108_97_115_115_102_105_115_104_46_119_101_98_46_104_97_46_115_101_115_115_105_111_110_46_109_97_110_97_103_101_109_101_110_116_46_70_117_108_108_72_65_83_101_115_115_105_111_110_72_76_71_-125_-73_120_111_24_2_0_0_119_8_0_0_0_0_0_0_0_-93_120_114_0_53_111_114_103_46_103_108_97_115_115_102_105_115_104_46_119_101_98_46_104_97_46_115_101_115_115_105_111_110_46_109_97_110_97_103_101_109_101_110_116_46_66_97_115_101_72_65_83_101_115_115_105_111_110_-22_32_75_-60_-102_-8_-61_-40_3_0_2_90_0_14_112_101_114_115_105_115_116_101_110_116_70_108_97_103_76_0_8_117_115_101_114_78_97_109_101_116_0_18_76_106_97_118_97_47_108_97_110_103_47_83_116_114_105_110_103_59_119_8_0_0_0_0_0_0_0_-93_120_114_0_43_111_114_103_46_97_112_97_99_104_101_46_99_97_116_97_108_105_110_97_46_115_101_115_115_105_111_110_46_83_116_97_110_100_97_114_100_83_101_115_115_105_111_110_126_62_89_94_-109_-97_117_102_3_0_12_74_0_12_99_114_101_97_116_105_111_110_84_105_109_101_90_0_5_105_115_78_101_119_90_0_7_105_115_86_97_108_105_100_74_0_16_108_97_115_116_65_99_99_101_115_115_101_100_84_105_109_101_73_0_19_109_97_120_73_110_97_99_116_105_118_101_73_110_116_101_114_118_97_108_74_0_16_116_104_105_115_65_99_99_101_115_115_101_100_84_105_109_101_76_0_10_97_116_116_114_105_98_117_116_101_115_116_0_15_76_106_97_118_97_47_117_116_105_108_47_77_97_112_59_76_0_5_98_101_75_101_121_113_0_126_0_2_76_0_2_105_100_113_0_126_0_2_76_0_15_115_105_112_65_112_112_83_101_115_115_105_111_110_73_100_113_0_126_0_2_76_0_5_115_115_111_73_100_113_0_126_0_2_76_0_7_118_101_114_115_105_111_110_116_0_40_76_106_97_118_97_47_117_116_105_108_47_99_111_110_99_117_114_114_101_110_116_47_97_116_111_109_105_99_47_65_116_111_109_105_99_76_111_110_103_59_119_8_0_0_0_0_0_0_0_-29_120_112_115_114_0_15_106_97_118_97_46_108_97_110_103_46_83_104_111_114_116_104_77_55_19_52_96_-38_82_2_0_1_83_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_114_0_16_106_97_118_97_46_108_97_110_103_46_78_117_109_98_101_114_-122_-84_-107_29_11_-108_-32_-117_2_0_0_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_112_0_1_115_114_0_14_106_97_118_97_46_108_97_110_103_46_76_111_110_103_59_-117_-28_-112_-52_-113_35_-33_2_0_1_74_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_113_0_126_0_8_0_0_1_44_16_-95_-58_104_115_113_0_126_0_10_0_0_1_44_16_-95_-40_14_115_114_0_17_106_97_118_97_46_108_97_110_103_46_73_110_116_101_103_101_114_18_-30_-96_-92_-9_-127_-121_56_2_0_1_73_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_113_0_126_0_8_0_0_7_8_115_114_0_17_106_97_118_97_46_108_97_110_103_46_66_111_111_108_101_97_110_-51_32_114_-128_-43_-100_-6_-18_2_0_1_90_0_5_118_97_108_117_101_119_8_-1_-1_-1_-1_-1_-1_-1_-1_120_112_0_115_113_0_126_0_15_1_115_113_0_126_0_10_0_0_1_44_16_-95_-36_19_116_0_28_48_97_49_99_54_54_56_49_56_54_53_52_97_101_52_98_51_48_56_55_48_101_49_97_97_55_48_98_115_113_0_126_0_13_0_0_0_3_116_0_6_83_70_83_66_95_49_115_114_0_94_99_111_109_46_115_117_110_46_101_106_98_46_99_111_110_116_97_105_110_101_114_115_46_69_74_66_76_111_99_97_108_79_98_106_101_99_116_73_110_118_111_99_97_116_105_111_110_72_97_110_100_108_101_114_68_101_108_101_103_97_116_101_36_83_101_114_105_97_108_105_122_97_98_108_101_76_111_99_97_108_79_98_106_101_99_116_68_101_108_101_103_97_116_101_-122_-89_-78_26_-97_-128_-117_34_2_0_5_74_0_11_99_111_110_116_97_105_110_101_114_73_100_90_0_27_105_115_79_112_116_105_111_110_97_108_76_111_99_97_108_66_117_115_105_110_101_115_115_86_105_101_119_74_0_7_118_101_114_115_105_111_110_76_0_13_105_110_116_102_67_108_97_115_115_78_97_109_101_113_0_126_0_2_76_0_10_112_114_105_109_97_114_121_75_101_121_116_0_18_76_106_97_118_97_47_108_97_110_103_47_79_98_106_101_99_116_59_119_8_0_0_0_0_0_0_0_34_120_112_1_44_16_-112_-76_-10_0_1_1_0_0_0_0_0_0_0_11_116_0_67_99_111_109_46_115_117_110_46_103_108_97_115_115_102_105_115_104_46_115_102_115_98_46_95_95_69_74_66_51_49_95_71_101_110_101_114_97_116_101_100_95_95_83_105_109_112_108_101_83_101_115_115_105_111_110_66_101_97_110_95_95_73_110_116_102_95_95_115_114_0_62_99_111_109_46_115_117_110_46_101_106_98_46_98_97_115_101_46_115_102_115_98_46_117_116_105_108_46_83_105_109_112_108_101_75_101_121_71_101_110_101_114_97_116_111_114_36_83_105_109_112_108_101_83_101_115_115_105_111_110_75_101_121_15_37_-8_-113_-64_94_31_37_2_0_3_73_0_2_105_100_74_0_6_112_114_101_102_105_120_74_0_6_115_117_102_102_105_120_119_8_0_0_0_0_0_0_0_34_120_112_0_0_0_1_31_0_-112_10_16_-95_-69_45_-18_0_5_-36_92_71_107_57_116_0_6_83_70_83_66_95_50_115_113_0_126_0_22_1_44_16_-112_-76_-10_0_0_1_0_0_0_0_0_0_0_4_116_0_66_99_111_109_46_115_117_110_46_103_108_97_115_115_102_105_115_104_46_115_102_115_98_46_95_95_69_74_66_51_49_95_71_101_110_101_114_97_116_101_100_95_95_67_104_101_99_107_112_111_105_110_116_101_100_66_101_97_110_95_95_73_110_116_102_95_95_115_113_0_126_0_26_0_0_0_1_31_0_-112_10_16_-95_-69_10_-18_0_5_-36_13_109_122_-92_116_0_7_105_110_116_65_116_116_114_115_113_0_126_0_13_0_0_0_5_112_112_120_116_0_0_120_}

      |#]

      [#|2010-11-03T07:27:32.698+0000|WARNING|glassfish3.1|javax.enterprise.system.container.web.org.glassfish.web.ha.session.management|_ThreadID=15;_ThreadName=Thread-1;|Exception
      occurred in getSession
      java.io.StreamCorruptedException: invalid type code: 00
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1356)
      at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at
      org.apache.catalina.session.StandardSession.deserialize(StandardSession.java:1144)
      at org.apache.catalina.session.StoreBase.readSession(StoreBase.java:288)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:524)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.getSession(ReplicationStore.java:476)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.loadFromBackingStore(ReplicationStore.java:394)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:377)
      at
      org.glassfish.web.ha.session.management.ReplicationStore.load(ReplicationStore.java:372)
      at
      org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1055)
      at
      org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1016)
      at
      org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:982)
      at
      org.apache.catalina.session.PersistentManagerBase.findSession(PersistentManagerBase.java:738)
      at org.apache.catalina.session.ManagerBase.findSession(ManagerBase.java:874)
      at org.apache.catalina.connector.Request.doGetSession(Request.java:2832)
      at org.apache.catalina.connector.Request.getSession(Request.java:2559)
      at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:920)
      at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:623)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:154)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at
      com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
      at
      com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
      at
      com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      at
      com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      at
      com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      at java.lang.Thread.run(Thread.java:619)

      #]

      To verify that the replication / saving to backing store is correct I changed
      the scenario slightly.

      1. Send a few http requests to instance1
      2. Fail instance1
      3. Send another http request to the instance specified in JREPLICA.
      4. Session is loaded from local cache.
      5. Test passes.

      See the attached logs (testTXCPDriver-pass.zip) for details. I think this proves
      that replication and de-serialization works correctly on the same VM. Loading
      the session remotely and de-serializing it seems to fail.

      Please work with Ming to reproduce this error.

      1. SessionData.zip
        4 kB
        mzh777
      2. testTXCPDriver.zip
        185 kB
        sonymanuel
      3. testTXCPDriver.zip
        185 kB
        sonymanuel

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Rajiv Mordani
              Reporter:
              sonymanuel
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: