glassfish
  1. glassfish
  2. GLASSFISH-15588

[blocking][regression][shoal] ClassCastException when loading or saving data.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1_b38
    • Fix Version/s: None
    • Component/s: failover
    • Labels:
      None

      Description

      Build Used:
      GlassFish Build 38 Nightly from 16th Jan

      The below ClassCastException appears in the instance logs of instance's that are attempting to load or save. This causes data to be lost on instance failure.

      ********
      [#|2011-01-17T20:18:00.534+0530|WARNING|glassfish3.1|org.shoal.ha.cache.command|_ThreadID=25;_ThreadName=Thread-1;|Error[2] during parsing command: opcode: -84
      java.lang.ClassCastException: java.util.concurrent.ConcurrentLinkedQueue cannot be cast to java.util.List
      at org.shoal.ha.cache.impl.interceptor.ReplicationFramePayloadCommand.readObject(ReplicationFramePayloadCommand.java:132)
      at sun.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
      at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
      at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
      at org.shoal.ha.cache.impl.command.CommandManager.handleMessage(CommandManager.java:142)
      at org.shoal.ha.cache.impl.util.MessageReceiver.processNotification(MessageReceiver.java:70)
      at com.sun.enterprise.ee.cms.impl.client.MessageActionImpl.processMessage(MessageActionImpl.java:89)
      at com.sun.enterprise.ee.cms.impl.client.MessageActionImpl.consumeSignal(MessageActionImpl.java:72)
      at com.sun.enterprise.ee.cms.impl.common.Router$CallableAction.call(Router.java:661)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

      #]

      **********

      Attached logs show the Exception in instance102 and instance103 when instance101 is attempting to load/save to instance102/103.

        Activity

        Hide
        varunrupela added a comment -

        This issue was not observed with promoted build 37.

        Show
        varunrupela added a comment - This issue was not observed with promoted build 37.
        Hide
        Joe Fialli added a comment -

        Classcast issue is in org.shoal.ha.cache.impl.interceptor.ReplicationFramePayloadCommand.
        So reassigned Component to failover and assignee to Mahesh.
        I did look in class and do not see where ConcurrentLinkedQueue is coming from.
        Will continue investigation and also discuss this with Mahesh.

        Show
        Joe Fialli added a comment - Classcast issue is in org.shoal.ha.cache.impl.interceptor.ReplicationFramePayloadCommand. So reassigned Component to failover and assignee to Mahesh. I did look in class and do not see where ConcurrentLinkedQueue is coming from. Will continue investigation and also discuss this with Mahesh.
        Hide
        Joe Fialli added a comment -

        One line fix required. Will require integration of shoal-cache.jar into glassfish 3.1.
        Waiting for review.

        Show
        Joe Fialli added a comment - One line fix required. Will require integration of shoal-cache.jar into glassfish 3.1. Waiting for review.
        Hide
        Nazrul added a comment -

        How bad is its impact? (Severity)
        Identify why the fix needs to occur now:
        (all apply)

        • Is a regression of functionality or performance available in a prior release
        • Introduces an incompatibility
        • Likely to generate a customer support call
        • Significantly impacts the operation of a primary release driver feature
        • An in-your-face issue that will touch the majority of users

        How often does it happen? Often

        How much effort is required to fix it? Fix is known. One line change. Requires integration of shoal into glassfish 3.1 workspace.

        What is the risk of fixing it? Riskier not to fix due to bad class cast in a java serialization readObject method.

        Does a work around for the issue exist? no Can the workaround be reasonably employed by the end user? no

        If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes?
        no workaround.

        Here is diff:

        Index:
        cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java

        ===================================================================


        cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java
        (revision 1452)

        +++
        cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java
        (working copy)

        @@ -129,7 +129,7 @@
        commands = (List<Command<K, V>>) ris.readObject();
        boolean ktAbsent = ris.readBoolean();
        if (ktAbsent)

        { - removedKeys = (List<K>) ris.readObject(); + removedKeys = (Collection<K>)ris.readObject(); }

        else

        { rawRemovedKeys = (List<byte[]>)ris.readObject(); }

        Note that the type of instance variable removedKeys was recently changed from List<K> to Collection<K>.
        Many occurances of removeKeys are still a List, but one usage is a ConcurrentQueueLinkedQueue.
        (See line 297 in ReplicationCommandTransmitterWithMap) That line is causing the reported issue.

        Show
        Nazrul added a comment - How bad is its impact? (Severity) Identify why the fix needs to occur now: (all apply) Is a regression of functionality or performance available in a prior release Introduces an incompatibility Likely to generate a customer support call Significantly impacts the operation of a primary release driver feature An in-your-face issue that will touch the majority of users How often does it happen? Often How much effort is required to fix it? Fix is known. One line change. Requires integration of shoal into glassfish 3.1 workspace. What is the risk of fixing it? Riskier not to fix due to bad class cast in a java serialization readObject method. Does a work around for the issue exist? no Can the workaround be reasonably employed by the end user? no If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes? no workaround. Here is diff: Index: cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java =================================================================== — cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (revision 1452) +++ cache/src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (working copy) @@ -129,7 +129,7 @@ commands = (List<Command<K, V>>) ris.readObject(); boolean ktAbsent = ris.readBoolean(); if (ktAbsent) { - removedKeys = (List<K>) ris.readObject(); + removedKeys = (Collection<K>)ris.readObject(); } else { rawRemovedKeys = (List<byte[]>)ris.readObject(); } Note that the type of instance variable removedKeys was recently changed from List<K> to Collection<K>. Many occurances of removeKeys are still a List, but one usage is a ConcurrentQueueLinkedQueue. (See line 297 in ReplicationCommandTransmitterWithMap) That line is causing the reported issue.
        Hide
        Mahesh Kannan added a comment -

        svn commit -m "Fix for 15588. QL Passed"
        Sending packager/resources/pkg_conf.py
        Sending pom.xml
        Transmitting file data ..
        Committed revision 44563.

        Show
        Mahesh Kannan added a comment - svn commit -m "Fix for 15588. QL Passed" Sending packager/resources/pkg_conf.py Sending pom.xml Transmitting file data .. Committed revision 44563.

          People

          • Assignee:
            Mahesh Kannan
            Reporter:
            varunrupela
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: