glassfish
  1. glassfish
  2. GLASSFISH-15639

[Stress] richAcces + SSL stress test run on Win 2008, jvm crashed after 4 days of running.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1_b37
    • Fix Version/s: None
    • Component/s: other
    • Labels:
      None

      Description

      Windows 2008 machines. Nightly nuild 38 01/14 was installed on three amchines:
      asqe-oblade-15 DAS + in3
      bigapp-oblade-1 in1
      bigapp-oblade-2 in2

      Was executed richAcces stress test, SSL was enbaled. Was used adefault EMBEDDED MQ mode.
      After about 4 days of running without any issues, in3 jvm crashed on one asqe-oblade-15.
      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      The JVM crash on asqe-oblade-15 happened on 1/18/11 at 1:36 pm. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node3\in3\config (see this log bellow)
      ================================================================

      --------------- T H R E A D ---------------

      Current thread (0x00000000090bf000): JavaThread "TransactionReaper" [_thread_in_Java, id=2716, stack(0x000000000d140000,0x000000000d160000)]

      siginfo: ExceptionCode=0xc0000005, reading address 0x000000000066006e

      Registers:
      RAX=0x00000000c7a1e3e8, RBX=0x00000000d3274840, RCX=0x00000000d4209be0, RDX=0x00000000f170afd8
      RSP=0x000000000d15f4c0, RBP=0x00000000f170afd8, RSI=0x0000000000000000, RDI=0x00000000f1769cc8
      R8=0x00000000f1709698, R9=0x0000000000660036, R10=0x0000000000000000, R11=0x00000000f1709698
      R12=0x0000000000000000, R13=0x0000000000000000, R14=0x000000000d15f558, R15=0x00000000090bf000
      RIP=0x0000000002af57c2, EFLAGS=0x0000000000010202

      Register to memory mapping:

      RAX=0x00000000c7a1e3e8

      {instance class}
      - klass: {other class}

      RBX=0x00000000d3274840
      java.util.ArrayList
      - klass: 'java/util/ArrayList'

      RCX=0x00000000d4209be0
      [Ljava.lang.Object;
      - klass: 'java/lang/Object'[]
      - length: 38

      RDX=0x00000000f170afd8

      [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xe0000000]

      Stack: [0x000000000d140000,0x000000000d160000], sp=0x000000000d15f4c0, free space=125k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      J com.sun.messaging.jmq.util.UID.equals(Ljava/lang/Object;)Z

      ====================================
      On in2 bigapp-oblade-2 : There is a JVM crash on 1/19/11 1:22 am. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node1\in1\config

      --------------- T H R E A D ---------------

      Current thread (0x0000000006ac4800): JavaThread "TransactionReaper" daemon [_thread_in_Java, id=3928, stack(0x000000000ca00000,0x000000000ca20000)]

      siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000650085

      Registers:
      RAX=0x00000000c40c4b20, RBX=0x0000000000000003, RCX=0x00000000f0953400, RDX=0x00000000c76fe1f8
      RSP=0x000000000ca1f640, RBP=0x00000000d2dbd440, RSI=0x00000000c7a1d728, RDI=0x59d92d9d95057f07
      R8=0x0000000000000001, R9=0x00000000d4249428, R10=0x0000000000000004, R11=0x000000000065004d
      R12=0x0000000000000000, R13=0x00000000c78f3488, R14=0x00000000c40121b0, R15=0x0000000006ac4800
      RIP=0x0000000001d782c6, EFLAGS=0x0000000000010286

      Register to memory mapping:

      RAX=0x00000000c40c4b20{instance class}
      • klass: {other class}

        RBX=0x0000000000000003
        0x0000000000000003 is pointing to unknown location

        RCX=0x00000000f0953400

        [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xe0000000]

        Stack: [0x000000000ca00000,0x000000000ca20000], sp=0x000000000ca1f640, free space=125k
        Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
        J com.sun.messaging.jmq.jmsserver.data.TransactionReaper.run()V

        ====================================
        On in1 - bigapp-oblade-1 : There is a JVM crash on 1/19/11 10:08 AM. Here is the reason for the crash. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node1\in1\config

        --------------- T H R E A D ---------------

        Current thread (0x0000000008aae000): JavaThread "Grizzly-kernel-thread(1)" daemon [_thread_in_vm, id=3572, stack(0x0000000008360000,0x0000000008380000)]

        siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000

        Registers:
        RAX=0x0000000000000000, RBX=0x0000000008aae000, RCX=0x0000000000000e00, RDX=0x0000000000000000
        RSP=0x000000000837f2b0, RBP=0x000000000a85bc50, RSI=0x0000000000000000, RDI=0x0000000000000000
        R8=0x0000000000000032, R9=0x000000000a85bc50, R10=0x0000000000000000, R11=0x0000000000000246
        R12=0x0000000000000032, R13=0x00000000c53922d8, R14=0x000000000837f448, R15=0x0000000008aae000
        RIP=0x000000006dadf79e, EFLAGS=0x0000000000010202

        Register to memory mapping:

        RAX=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RBX=0x0000000008aae000
        "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
        java.lang.Thread.State: RUNNABLE

        RCX=0x0000000000000e00
        0x0000000000000e00 is pointing to unknown location

        RDX=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RSP=0x000000000837f2b0
        0x000000000837f2b0 is pointing into the stack for thread: 0x0000000008aae000
        "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
        java.lang.Thread.State: RUNNABLE

        RBP=0x000000000a85bc50
        0x000000000a85bc50 is pointing to unknown location

        RSI=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RDI=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        R8 =0x0000000000000032
        0x0000000000000032 is pointing to unknown location

        R9 =0x000000000a85bc50
        0x000000000a85bc50 is pointing to unknown location

        R10=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        R11=0x0000000000000246
        0x0000000000000246 is pointing to unknown location

        R12=0x0000000000000032
        0x0000000000000032 is pointing to unknown location

        R13=0x00000000c53922d8 {constMethod}
        - klass: {other class}
      • method: 0x00000000c53922e0 {method}

        'accept0' '(Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;[Ljava/net/InetSocketAddress;)I' in 'sun/nio/ch/ServerSocketChannelImpl'

      • exceptions: 0x00000000c4001ef8

      R14=0x000000000837f448
      0x000000000837f448 is pointing into the stack for thread: 0x0000000008aae000
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
      java.lang.Thread.State: RUNNABLE

      R15=0x0000000008aae000
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
      java.lang.Thread.State: RUNNABLE

      Top of Stack: (sp=0x000000000837f2b0)
      0x000000000837f2b0: 0000000008aae000 0000000000002048
      0x000000000837f2c0: 000000000837f438 000000000837f438
      0x000000000837f2d0: 00000000000009a0 000000000837f430
      0x000000000837f2e0: 0000000000000000 000000000837f438
      0x000000000837f2f0: 0000000008aae1c8 00000000c4832e18
      0x000000000837f300: 0000000000000000 000000006d6c3215
      0x000000000837f310: 000000006dee6520 000000000837f430
      0x000000000837f320: 0000000008aae1c8 00000000c4149600
      0x000000000837f330: 0000000000000001 0000000000000001
      0x000000000837f340: 0000000000000010 50bc850a8fa60002
      0x000000000837f350: 0000000000000000 0000b74226c117d6
      0x000000000837f360: 0000000000000001 00000000012258fa
      0x000000000837f370: 00000000c4149600 000000000837f410
      0x000000000837f380: 00000000c53922e0 00000000012312a0
      0x000000000837f390: 0000000000000001 00000000c53c1f98
      0x000000000837f3a0: 0000000008aae000 000000006da91dbd

      Instructions: (pc=0x000000006dadf79e)
      0x000000006dadf78e: 83 38 02 00 00 06 00 00 00 80 3d 2a ca 41 00 00
      0x000000006dadf79e: 48 8b 16 74 17 44 8b 4a 08 0f b6 0d 0a b0 40 00

      Stack: [0x0000000008360000,0x0000000008380000], sp=0x000000000837f2b0, free space=124k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [jvm.dll+0x24f79e]

      [error occurred during error reporting (printing native stack), id 0xc0000005]

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      j sun.nio.ch.ServerSocketChannelImpl.accept0(Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;[Ljava/net/InetSocketAddress;)I+0
      j sun.nio.ch.ServerSocketChannelImpl.accept()Ljava/nio/channels/SocketChannel;+94
      j com.sun.grizzly.TCPSelectorHandler.acceptWithoutRegistration(Ljava/nio/channels/SelectionKey;)Ljava/nio/channels/SelectableChannel;+11
      j com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectorHandler.acceptWithoutRegistration(Ljava/nio/channels/SelectionKey;)Ljava/nio/channels/SelectableChannel;+2
      j com.sun.grizzly.http.SelectorThreadHandler.onAcceptInterest(Ljava/nio/channels/SelectionKey;Lcom/sun/grizzly/Context;)Z+2
      J com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(Ljava/nio/channels/SelectionKey;Lcom/sun/grizzly/SelectorHandler;Lcom/sun/grizzly/NIOContext;)Z
      J com.sun.grizzly.SelectorHandlerRunner.doSelect(Lcom/sun/grizzly/SelectorHandler;Lcom/sun/grizzly/NIOContext;)Z
      J com.sun.grizzly.SelectorHandlerRunner.run()V
      j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+59
      j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
      j java.lang.Thread.run()V+11
      v ~StubRoutines::call_stub

      1. domain.xml
        42 kB
        sonymanuel
      2. hs_err_pid4216.log
        38 kB
        sonymanuel

        Activity

        Hide
        sb110099 added a comment -

        Update : jvm bug 7014061 filed by Amy .

        --Sudipa

        Show
        sb110099 added a comment - Update : jvm bug 7014061 filed by Amy . --Sudipa
        Hide
        Nazrul added a comment -

        Tracking bug. Excluding from un-scrubbed list

        Show
        Nazrul added a comment - Tracking bug. Excluding from un-scrubbed list
        Hide
        Nazrul added a comment -

        I am closing this tracking bug. Please re-open the issue if we see issues with RichAccess run.

        Show
        Nazrul added a comment - I am closing this tracking bug. Please re-open the issue if we see issues with RichAccess run.
        Hide
        sb110099 added a comment -

        Reopening this bug as we are still seeing the issue with b40.

        This is the 3rd time we are able to reproduce the issue on same setup (at SCA) with either SSL or without SSL . (Last update from India: they also saw a crash on Windows setup with b40 and Richaccess+SSL , yet to get more details).

        With Abhijit's help, we have a jdk Engineer (Zhengyu Gu) monitoring the last run. He has updated bug http://monaco.sfbay.sun.com/detail.jsf?cr=7014061 with his findings so far .

        We are going to restart the run with flag XX:+ShowMessageBoxOnError as advised by Zhengyu for him to be able to attach a debugger.

        Thanks,
        Sudipa

        Show
        sb110099 added a comment - Reopening this bug as we are still seeing the issue with b40. This is the 3rd time we are able to reproduce the issue on same setup (at SCA) with either SSL or without SSL . (Last update from India: they also saw a crash on Windows setup with b40 and Richaccess+SSL , yet to get more details). With Abhijit's help, we have a jdk Engineer (Zhengyu Gu) monitoring the last run. He has updated bug http://monaco.sfbay.sun.com/detail.jsf?cr=7014061 with his findings so far . We are going to restart the run with flag XX:+ShowMessageBoxOnError as advised by Zhengyu for him to be able to attach a debugger. Thanks, Sudipa
        Hide
        Nazrul added a comment -

        Getting help from Alexey

        Show
        Nazrul added a comment - Getting help from Alexey
        Hide
        sonymanuel added a comment -

        The crash log and domain.xml from IEC setup is attached.

        Show
        sonymanuel added a comment - The crash log and domain.xml from IEC setup is attached.
        Hide
        oleksiys added a comment -

        According to the crash reports attached in 7014061, mentioned 4 crashes don't look related.
        Two times it happens in
        com.sun.messaging.jmq.util.UID.equals(...)

        I found the source[1] and it looks like:

        124 /**
        125 * Equals
        126 */
        127 public boolean equals(Object obj) {
        128 if (! (obj instanceof UID))

        { 129 return false; 130 }

        131 return (this.id == ((UID)obj).id);
        132 }

        don't see anything suspicious, no native calls etc.

        Other 2 crashes occur on different places: jmq TransactionReaper and socket accept.

        So, IMO, as i told crashes don't look related and probably they are caused by some general JDK/OS issue.

        WBR.

        [1] http://www.docjar.com/html/api/com/sun/messaging/jmq/util/UID.java.html

        Show
        oleksiys added a comment - According to the crash reports attached in 7014061, mentioned 4 crashes don't look related. Two times it happens in com.sun.messaging.jmq.util.UID.equals(...) I found the source [1] and it looks like: 124 /** 125 * Equals 126 */ 127 public boolean equals(Object obj) { 128 if (! (obj instanceof UID)) { 129 return false; 130 } 131 return (this.id == ((UID)obj).id); 132 } don't see anything suspicious, no native calls etc. Other 2 crashes occur on different places: jmq TransactionReaper and socket accept. So, IMO, as i told crashes don't look related and probably they are caused by some general JDK/OS issue. WBR. [1] http://www.docjar.com/html/api/com/sun/messaging/jmq/util/UID.java.html
        Hide
        amyk added a comment - - edited

        For the com.sun.messaging.jmq.util.UID.equals frame in hs_err logs, suggested GlassFish QE to try potential temporary workarounds described in the "Java Trouble-Shooting and Diagnostic Guide" while JVM engineer is looking into the issue (7014061/jvm).

        [information from Sony (IEC):
        non-SSL run completed 24x7 with JDK 1.6.0_22 64 bit.
        SSL - 1 instance crashed on 5th day. 2 instances completed 24x7. JDK used was 1.6.0_23.]

        Show
        amyk added a comment - - edited For the com.sun.messaging.jmq.util.UID.equals frame in hs_err logs, suggested GlassFish QE to try potential temporary workarounds described in the "Java Trouble-Shooting and Diagnostic Guide" while JVM engineer is looking into the issue (7014061/jvm). [information from Sony (IEC): non-SSL run completed 24x7 with JDK 1.6.0_22 64 bit. SSL - 1 instance crashed on 5th day. 2 instances completed 24x7. JDK used was 1.6.0_23.]
        Hide
        Chris Kasso added a comment -

        The test completed with SSL enabled using update 24. If we see the problem again on U24 we can reopen the issue.

        Show
        Chris Kasso added a comment - The test completed with SSL enabled using update 24. If we see the problem again on U24 we can reopen the issue.
        Hide
        Chris Kasso added a comment -

        Cannot reproduce on u24

        Show
        Chris Kasso added a comment - Cannot reproduce on u24

          People

          • Assignee:
            Chris Kasso
            Reporter:
            easarina
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: