glassfish
  1. glassfish
  2. GLASSFISH-15376

[STRESS] java.lang.ArrayIndexOutOfBoundsException in replication layer when executing richAccess 24x7

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.1_b37
    • Component/s: failover
    • Labels:
      None

      Description

      GF build : 21-Dec-2010 nightly build.

      When running richaccess 24x7, we see ~50 instances of this exception in the cluster over a 24x7 period.This does not seem to cause any client side issues.

      [#|2010-12-22T09:25:34.496+0530|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=16;_ThreadName=Thread-1;|PWC3989: An exception or error occurred in the container during the request processing

      java.lang.ArrayIndexOutOfBoundsException: 19
      at java.util.ArrayList.add(ArrayList.java:352)
      at org.shoal.ha.cache.impl.interceptor.ReplicationCommandTransmitterWithMap$BatchedCommandMapDataFrame.doAddOrRemove(ReplicationCommandTransmitterWithMap.java:218)
      at org.shoal.ha.cache.impl.interceptor.ReplicationCommandTransmitterWithMap.removeCommand(ReplicationCommandTransmitterWithMap.java:162)
      at org.shoal.ha.cache.impl.interceptor.ReplicationCommandTransmitterManager.onTransmit(ReplicationCommandTransmitterManager.java:104)
      at org.shoal.ha.cache.api.AbstractCommandInterceptor.onTransmit(AbstractCommandInterceptor.java:98)
      at org.shoal.ha.cache.impl.interceptor.CommandHandlerInterceptor.onTransmit(CommandHandlerInterceptor.java:74)
      at org.shoal.ha.cache.impl.command.CommandManager.executeCommand(CommandManager.java:122)
      at org.shoal.ha.cache.impl.command.CommandManager.execute(CommandManager.java:114)
      at org.shoal.ha.cache.impl.store.ReplicatedDataStore.remove(ReplicatedDataStore.java:459)
      at org.shoal.adapter.store.ReplicatedBackingStore.remove(ReplicatedBackingStore.java:115)
      at org.glassfish.web.ha.session.management.ReplicationManagerBase.doRemove(ReplicationManagerBase.java:95)
      at org.glassfish.web.ha.session.management.HAStoreBase.remove(HAStoreBase.java:270)
      at org.apache.catalina.session.PersistentManagerBase.removeSession(PersistentManagerBase.java:898)
      at org.apache.catalina.session.PersistentManagerBase.remove(PersistentManagerBase.java:876)
      at org.apache.catalina.session.PersistentManagerBase.remove(PersistentManagerBase.java:861)
      at org.apache.catalina.session.StandardSession.setId(StandardSession.java:401)
      at org.glassfish.web.ha.session.management.BaseHASession.setId(BaseHASession.java:85)
      at org.apache.catalina.session.ManagerBase.changeSessionId(ManagerBase.java:941)
      at org.apache.catalina.authenticator.AuthenticatorBase.register(AuthenticatorBase.java:832)
      at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:209)
      at com.sun.web.security.RealmAdapter.invokeAuthenticateDelegate(RealmAdapter.java:1193)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:551)
      at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:623)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
      at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:326)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)

        Issue Links

          Activity

          Hide
          sonymanuel added a comment -

          Fix category. Attach server.log

          Show
          sonymanuel added a comment - Fix category. Attach server.log
          Hide
          Mahesh Kannan added a comment -

          Most likely this is due to the use of an ArrayList in the doAddOrRemove method. I am testing a patch that uses ConcurrentLinkedQueue

          Show
          Mahesh Kannan added a comment - Most likely this is due to the use of an ArrayList in the doAddOrRemove method. I am testing a patch that uses ConcurrentLinkedQueue
          Hide
          Mahesh Kannan added a comment -

          How bad is its impact? (Severity)
          Exception seen during Longivity runs.

          How often does it happen? Will many users see this problem? (Frequency)
          Intermittent but can cause remove calls to fail

          How much effort is required to fix it? (Cost)
          Fix is reasonably small.

          What is the risk of fixing it and how will the risk be mitigated? (Risk)
          Small. We are also testing the patch with Trade2 to ensure that this doesnt introduce any regression in performance

          Index: src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java
          ===================================================================
          — src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java (revision 1412)
          +++ src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java (working copy)
          @@ -190,8 +190,10 @@

          private volatile ConcurrentHashMap map = new ConcurrentHashMap();

          • private volatile List removedKeys = new ArrayList();
            + private AtomicInteger removedKeysSize = new AtomicInteger(0);

          + private volatile ConcurrentLinkedQueue removedKeys = new ConcurrentLinkedQueue();
          +
          private volatile long lastTS = System.currentTimeMillis();

          BatchedCommandMapDataFrame()

          { @@ -216,8 +218,9 @@ }

          else {
          map.remove(cmd.getKey());
          removedKeys.add(cmd.getKey());
          + int removedSz = removedKeysSize.incrementAndGet();
          result = true;

          • if (removedKeys.size() >= (2 * MAX_BATCH_SIZE))
            Unknown macro: {+ if (removedSz >= (2 * MAX_BATCH_SIZE)) { batchThresholdReached.compareAndSet(false, true); } }

            @@ -247,13 +250,13 @@
            //Called by periodic task
            void flushAndTransmit() {
            dsc.getDataStoreMBean().incrementFlushThreadWakeupCount();

          • if ((!alreadySent.get()) && ((map.size() > 0) || (removedKeys.size() > 0))) {
            + if ((!alreadySent.get()) && ((map.size() > 0) || (removedKeysSize.get() > 0))) {
            if (lastTS == timeStamp) {
            if (_statsLogger.isLoggable(Level.FINE)) { _statsLogger.log(Level.FINE, "flushAndTransmit will flush data because lastTS = " + lastTS + "; timeStamp = " + timeStamp + "; lastTS = " + lastTS + "; map.size() = " + map.size() - + "; removedKeys.size() = " +removedKeys.size()); + + "; removedKeys.size() = " +removedKeysSize.get()); }

            boolean completed = false;
            int index = 0;
            @@ -267,7 +270,7 @@
            _statsLogger.log(Level.FINER, "flushAndTransmit will NOT flush data because lastTS = " + lastTS
            + "; timeStamp = " + timeStamp + "; lastTS = " + lastTS
            + "; map.size() = " + map.size()

          • + "; removedKeys.size() = " +removedKeys.size());
            + + "; removedKeys.size() = " +removedKeysSize.get());
            }
            timeStamp = lastTS;
            }
            Index: src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java
            ===================================================================
              • src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (revision 1412)
                +++ src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (working copy)
                @@ -54,6 +54,7 @@
                import java.io.ObjectInputStream;
                import java.io.ObjectOutputStream;
                import java.util.ArrayList;
                +import java.util.Collection;
                import java.util.List;
                import java.util.logging.Level;
                import java.util.logging.Logger;
                @@ -72,7 +73,7 @@

          private List<Command<K, V>> commands = new ArrayList<Command<K, V>>();

          • private List<K> removedKeys = new ArrayList<K>();
            + private Collection<K> removedKeys = new ArrayList<K>();

          private List<byte[]> rawRemovedKeys = new ArrayList<byte[]>();

          @@ -89,7 +90,7 @@
          targetInstanceName = target;
          }

          • void setRemovedKeys(List<K> removedKeys) {
            + void setRemovedKeys(Collection<K> removedKeys) { this.removedKeys = removedKeys; }
          Show
          Mahesh Kannan added a comment - How bad is its impact? (Severity) Exception seen during Longivity runs. How often does it happen? Will many users see this problem? (Frequency) Intermittent but can cause remove calls to fail How much effort is required to fix it? (Cost) Fix is reasonably small. What is the risk of fixing it and how will the risk be mitigated? (Risk) Small. We are also testing the patch with Trade2 to ensure that this doesnt introduce any regression in performance Index: src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java =================================================================== — src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java (revision 1412) +++ src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java (working copy) @@ -190,8 +190,10 @@ private volatile ConcurrentHashMap map = new ConcurrentHashMap(); private volatile List removedKeys = new ArrayList(); + private AtomicInteger removedKeysSize = new AtomicInteger(0); + private volatile ConcurrentLinkedQueue removedKeys = new ConcurrentLinkedQueue(); + private volatile long lastTS = System.currentTimeMillis(); BatchedCommandMapDataFrame() { @@ -216,8 +218,9 @@ } else { map.remove(cmd.getKey()); removedKeys.add(cmd.getKey()); + int removedSz = removedKeysSize.incrementAndGet(); result = true; if (removedKeys.size() >= (2 * MAX_BATCH_SIZE)) Unknown macro: {+ if (removedSz >= (2 * MAX_BATCH_SIZE)) { batchThresholdReached.compareAndSet(false, true); } } @@ -247,13 +250,13 @@ //Called by periodic task void flushAndTransmit() { dsc.getDataStoreMBean().incrementFlushThreadWakeupCount(); if ((!alreadySent.get()) && ((map.size() > 0) || (removedKeys.size() > 0))) { + if ((!alreadySent.get()) && ((map.size() > 0) || (removedKeysSize.get() > 0))) { if (lastTS == timeStamp) { if (_statsLogger.isLoggable(Level.FINE)) { _statsLogger.log(Level.FINE, "flushAndTransmit will flush data because lastTS = " + lastTS + "; timeStamp = " + timeStamp + "; lastTS = " + lastTS + "; map.size() = " + map.size() - + "; removedKeys.size() = " +removedKeys.size()); + + "; removedKeys.size() = " +removedKeysSize.get()); } boolean completed = false; int index = 0; @@ -267,7 +270,7 @@ _statsLogger.log(Level.FINER, "flushAndTransmit will NOT flush data because lastTS = " + lastTS + "; timeStamp = " + timeStamp + "; lastTS = " + lastTS + "; map.size() = " + map.size() + "; removedKeys.size() = " +removedKeys.size()); + + "; removedKeys.size() = " +removedKeysSize.get()); } timeStamp = lastTS; } Index: src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java =================================================================== src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (revision 1412) +++ src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java (working copy) @@ -54,6 +54,7 @@ import java.io.ObjectInputStream; import java.io.ObjectOutputStream; import java.util.ArrayList; +import java.util.Collection; import java.util.List; import java.util.logging.Level; import java.util.logging.Logger; @@ -72,7 +73,7 @@ private List<Command<K, V>> commands = new ArrayList<Command<K, V>>(); private List<K> removedKeys = new ArrayList<K>(); + private Collection<K> removedKeys = new ArrayList<K>(); private List<byte[]> rawRemovedKeys = new ArrayList<byte[]>(); @@ -89,7 +90,7 @@ targetInstanceName = target; } void setRemovedKeys(List<K> removedKeys) { + void setRemovedKeys(Collection<K> removedKeys) { this.removedKeys = removedKeys; }
          Hide
          Mahesh Kannan added a comment -

          svn commit -m "Fix for 15376. Ran trade2 on Steve's setup with no regression" src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java
          Sending src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java
          Sending src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java
          Transmitting file data ..
          Committed revision 1417.

          Show
          Mahesh Kannan added a comment - svn commit -m "Fix for 15376. Ran trade2 on Steve's setup with no regression" src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java Sending src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationCommandTransmitterWithMap.java Sending src/main/java/org/shoal/ha/cache/impl/interceptor/ReplicationFramePayloadCommand.java Transmitting file data .. Committed revision 1417.
          Hide
          Mahesh Kannan added a comment -

          svn diff
          Index: pom.xml
          ===================================================================
          — pom.xml (revision 44457)
          +++ pom.xml (working copy)
          @@ -159,7 +159,7 @@
          <v3-docs.version>3.1-b10</v3-docs.version>
          <l10n.version>3.1-b34</l10n.version>
          <asm.version>3.1</asm.version>

          • <shoal.version>1.5.25</shoal.version>
            + <shoal.version>1.5.26</shoal.version>
            <ha-api.version>3.1.8</ha-api.version>
            </properties>

          Index: packager/resources/pkg_conf.py
          ===================================================================
          — packager/resources/pkg_conf.py (revision 44457)
          +++ packager/resources/pkg_conf.py (working copy)
          @@ -68,7 +68,7 @@
          grizzly_version="1.9.28,0-1"
          metro_version="2.1,0-25"
          javahelp_version="2.0.2,0-1"
          -shoal_version="1.5.25,0-0"
          +shoal_version="1.5.26,0-0"

          #description
          glassfish_description="GlassFish Application Server"

          =============================================================

          svn commit -m "New shoal integration to fix 15376. QL Passed"
          Sending packager/resources/pkg_conf.py
          Sending pom.xml
          Transmitting file data ..
          Committed revision 44458.

          Show
          Mahesh Kannan added a comment - svn diff Index: pom.xml =================================================================== — pom.xml (revision 44457) +++ pom.xml (working copy) @@ -159,7 +159,7 @@ <v3-docs.version>3.1-b10</v3-docs.version> <l10n.version>3.1-b34</l10n.version> <asm.version>3.1</asm.version> <shoal.version>1.5.25</shoal.version> + <shoal.version>1.5.26</shoal.version> <ha-api.version>3.1.8</ha-api.version> </properties> Index: packager/resources/pkg_conf.py =================================================================== — packager/resources/pkg_conf.py (revision 44457) +++ packager/resources/pkg_conf.py (working copy) @@ -68,7 +68,7 @@ grizzly_version="1.9.28,0-1" metro_version="2.1,0-25" javahelp_version="2.0.2,0-1" -shoal_version="1.5.25,0-0" +shoal_version="1.5.26,0-0" #description glassfish_description="GlassFish Application Server" ============================================================= svn commit -m "New shoal integration to fix 15376. QL Passed" Sending packager/resources/pkg_conf.py Sending pom.xml Transmitting file data .. Committed revision 44458.

            People

            • Assignee:
              Mahesh Kannan
              Reporter:
              sonymanuel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: