glassfish
  1. glassfish
  2. GLASSFISH-16707

[Blocking] AIX. RichAccess stress test. Multiple ArrayIndexOutOfBoundsException and NPE in server.log files

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 3.1.1_b05
    • Fix Version/s: 3.1.2
    • Component/s: web_container
    • Labels:
      None

      Description

      Glassfish 3.1.1 b05 nightly 05/19. Installed this build on three AIX 6.1 machines. Created a cluster with three instances, one instance per a machine. Configured richAccess and started a stress test (100 virtual users per an instance). The test is running, but in the instances server.log files were created multiple error messages. Please see bellow the first error message in each server.log and I've attached full server.log files.

      Instance 1:
      ===============================================
      [#|2011-05-21T03:11:17.537-0700|SEVERE|glassfish3.1|com.s1as.e2e.richAccess.servlet.sendorder|_ThreadID=15;_ThreadName=Thread
      -8;|Exception e java.lang.NullPointerException|#]
      =====================================================

      Instance 2
      ==============================================
      [#|2011-05-20T20:44:11.002-0700|SEVERE|glassfish3.1|com.s1as.e2e.richAccess.servlet.sendorder|_ThreadID=15;_ThreadName=Thread
      -8;|Exception e java.lang.NullPointerException|#]
      ====================================================

      Instance 3
      ====================================================
      [#|2011-05-20T15:17:31.746-0700|SEVERE|glassfish3.1|org.apache.catalina.core.ContainerBase|_ThreadID=16;_ThreadName=Thread-8;

      Exception invoking periodic operation:
      java.lang.ArrayIndexOutOfBoundsException
      at java.util.AbstractCollection.toArray(AbstractCollection.java:353)
      at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:907)
      at org.apache.catalina.session.PersistentManagerBase.processExpires(PersistentManagerBase.java:602)
      at org.apache.catalina.session.PersistentManagerBase.backgroundProcess(PersistentManagerBase.java:255)
      at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5783)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1791)
      at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1780)
      at java.lang.Thread.run(Thread.java:736)
      =======================================================
      1. CHMTest.java
        2 kB
        Shing Wai Chan
      2. server.log.in1
        217 kB
        easarina
      3. server.log.in2
        222 kB
        easarina
      4. server.log.in3
        36 kB
        easarina

        Activity

        Hide
        Dhiru Pandey added a comment -

        This is NOT really a blocking issue.

        Based on my understanding of the code and discussions with Shing Wai, this issue will only occur when there is session reaping going on during runtime. The session reaping may fail occasionally because of this issue on IBM JDK but it should not cause any user application issue. What is likely to happen over time is a memory leak because some of the old sessions are not reaped whenever this exception occurs.

        The fact that the Stress test ran thru its desired course of time even though there were failures in the log file for this issue supports the above findings.

        Show
        Dhiru Pandey added a comment - This is NOT really a blocking issue. Based on my understanding of the code and discussions with Shing Wai, this issue will only occur when there is session reaping going on during runtime. The session reaping may fail occasionally because of this issue on IBM JDK but it should not cause any user application issue. What is likely to happen over time is a memory leak because some of the old sessions are not reaped whenever this exception occurs. The fact that the Stress test ran thru its desired course of time even though there were failures in the log file for this issue supports the above findings.
        Hide
        easarina added a comment -

        There are not only error messages in server.log files, but also some sessions failed.

        Show
        easarina added a comment - There are not only error messages in server.log files, but also some sessions failed.
        Hide
        Rebecca Parks added a comment -

        So what should the Release Notes say? "On AIX some sessions that are no longer used are not properly garbage collected, which causes a memory leak that in turn interferes with the creation of new sessions"?

        And is there a workaround?

        Show
        Rebecca Parks added a comment - So what should the Release Notes say? "On AIX some sessions that are no longer used are not properly garbage collected, which causes a memory leak that in turn interferes with the creation of new sessions"? And is there a workaround?
        Hide
        scatari added a comment -

        Reducing the priority to P3 as this is not a release stopper given what it takes to reproduce it and the auto-recovery/cleanup.

        Show
        scatari added a comment - Reducing the priority to P3 as this is not a release stopper given what it takes to reproduce it and the auto-recovery/cleanup.
        Hide
        Rebecca Parks added a comment - - edited

        Based on info received via email from Shing Wai and Alex, I added the following to the 3.1.1 Release Notes:

        AIX. RichAccess stress test. Multiple ArrayIndexOutOfBoundsException and NPE in server.log files (16707)

        Description

        The following exception may occur occasionally when the web container tries to clean up expired sessions. This is due to a bug in the IBM JDK.

        java.lang.ArrayIndexOutOfBoundsException
        at java.util.AbstractCollection.toArray(AbstractCollection.java:353)
        at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:907)
        at org.apache.catalina.session.PersistentManagerBase.processExpires(PersistentManagerBase.java:602)

        A support ticket has been filed with IBM (PMR# 79545,756,000) but no fix is yet available.

        Workaround

        None. If the above exception occurs, the expired sessions are cleaned up when the method PersistentManagerBase.processExpires is called again later. The application continues to work.

        Show
        Rebecca Parks added a comment - - edited Based on info received via email from Shing Wai and Alex, I added the following to the 3.1.1 Release Notes: AIX. RichAccess stress test. Multiple ArrayIndexOutOfBoundsException and NPE in server.log files (16707) Description The following exception may occur occasionally when the web container tries to clean up expired sessions. This is due to a bug in the IBM JDK. java.lang.ArrayIndexOutOfBoundsException at java.util.AbstractCollection.toArray(AbstractCollection.java:353) at org.apache.catalina.session.ManagerBase.findSessions(ManagerBase.java:907) at org.apache.catalina.session.PersistentManagerBase.processExpires(PersistentManagerBase.java:602) A support ticket has been filed with IBM (PMR# 79545,756,000) but no fix is yet available. Workaround None. If the above exception occurs, the expired sessions are cleaned up when the method PersistentManagerBase.processExpires is called again later. The application continues to work.

          People

          • Assignee:
            Rajiv Mordani
            Reporter:
            easarina
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: