glassfish
  1. glassfish
  2. GLASSFISH-15639

[Stress] richAcces + SSL stress test run on Win 2008, jvm crashed after 4 days of running.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1_b37
    • Fix Version/s: None
    • Component/s: other
    • Labels:
      None

      Description

      Windows 2008 machines. Nightly nuild 38 01/14 was installed on three amchines:
      asqe-oblade-15 DAS + in3
      bigapp-oblade-1 in1
      bigapp-oblade-2 in2

      Was executed richAcces stress test, SSL was enbaled. Was used adefault EMBEDDED MQ mode.
      After about 4 days of running without any issues, in3 jvm crashed on one asqe-oblade-15.
      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      The JVM crash on asqe-oblade-15 happened on 1/18/11 at 1:36 pm. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node3\in3\config (see this log bellow)
      ================================================================

      --------------- T H R E A D ---------------

      Current thread (0x00000000090bf000): JavaThread "TransactionReaper" [_thread_in_Java, id=2716, stack(0x000000000d140000,0x000000000d160000)]

      siginfo: ExceptionCode=0xc0000005, reading address 0x000000000066006e

      Registers:
      RAX=0x00000000c7a1e3e8, RBX=0x00000000d3274840, RCX=0x00000000d4209be0, RDX=0x00000000f170afd8
      RSP=0x000000000d15f4c0, RBP=0x00000000f170afd8, RSI=0x0000000000000000, RDI=0x00000000f1769cc8
      R8=0x00000000f1709698, R9=0x0000000000660036, R10=0x0000000000000000, R11=0x00000000f1709698
      R12=0x0000000000000000, R13=0x0000000000000000, R14=0x000000000d15f558, R15=0x00000000090bf000
      RIP=0x0000000002af57c2, EFLAGS=0x0000000000010202

      Register to memory mapping:

      RAX=0x00000000c7a1e3e8

      {instance class}
      - klass: {other class}

      RBX=0x00000000d3274840
      java.util.ArrayList
      - klass: 'java/util/ArrayList'

      RCX=0x00000000d4209be0
      [Ljava.lang.Object;
      - klass: 'java/lang/Object'[]
      - length: 38

      RDX=0x00000000f170afd8

      [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xe0000000]

      Stack: [0x000000000d140000,0x000000000d160000], sp=0x000000000d15f4c0, free space=125k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      J com.sun.messaging.jmq.util.UID.equals(Ljava/lang/Object;)Z

      ====================================
      On in2 bigapp-oblade-2 : There is a JVM crash on 1/19/11 1:22 am. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node1\in1\config

      --------------- T H R E A D ---------------

      Current thread (0x0000000006ac4800): JavaThread "TransactionReaper" daemon [_thread_in_Java, id=3928, stack(0x000000000ca00000,0x000000000ca20000)]

      siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000650085

      Registers:
      RAX=0x00000000c40c4b20, RBX=0x0000000000000003, RCX=0x00000000f0953400, RDX=0x00000000c76fe1f8
      RSP=0x000000000ca1f640, RBP=0x00000000d2dbd440, RSI=0x00000000c7a1d728, RDI=0x59d92d9d95057f07
      R8=0x0000000000000001, R9=0x00000000d4249428, R10=0x0000000000000004, R11=0x000000000065004d
      R12=0x0000000000000000, R13=0x00000000c78f3488, R14=0x00000000c40121b0, R15=0x0000000006ac4800
      RIP=0x0000000001d782c6, EFLAGS=0x0000000000010286

      Register to memory mapping:

      RAX=0x00000000c40c4b20{instance class}
      • klass: {other class}

        RBX=0x0000000000000003
        0x0000000000000003 is pointing to unknown location

        RCX=0x00000000f0953400

        [error occurred during error reporting (printing registers, top of stack, instructions near pc), id 0xe0000000]

        Stack: [0x000000000ca00000,0x000000000ca20000], sp=0x000000000ca1f640, free space=125k
        Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
        J com.sun.messaging.jmq.jmsserver.data.TransactionReaper.run()V

        ====================================
        On in1 - bigapp-oblade-1 : There is a JVM crash on 1/19/11 10:08 AM. Here is the reason for the crash. Please see jvm crash log under file C:\export\glassfish3\glassfish\nodes\node1\in1\config

        --------------- T H R E A D ---------------

        Current thread (0x0000000008aae000): JavaThread "Grizzly-kernel-thread(1)" daemon [_thread_in_vm, id=3572, stack(0x0000000008360000,0x0000000008380000)]

        siginfo: ExceptionCode=0xc0000005, reading address 0x0000000000000000

        Registers:
        RAX=0x0000000000000000, RBX=0x0000000008aae000, RCX=0x0000000000000e00, RDX=0x0000000000000000
        RSP=0x000000000837f2b0, RBP=0x000000000a85bc50, RSI=0x0000000000000000, RDI=0x0000000000000000
        R8=0x0000000000000032, R9=0x000000000a85bc50, R10=0x0000000000000000, R11=0x0000000000000246
        R12=0x0000000000000032, R13=0x00000000c53922d8, R14=0x000000000837f448, R15=0x0000000008aae000
        RIP=0x000000006dadf79e, EFLAGS=0x0000000000010202

        Register to memory mapping:

        RAX=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RBX=0x0000000008aae000
        "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
        java.lang.Thread.State: RUNNABLE

        RCX=0x0000000000000e00
        0x0000000000000e00 is pointing to unknown location

        RDX=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RSP=0x000000000837f2b0
        0x000000000837f2b0 is pointing into the stack for thread: 0x0000000008aae000
        "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
        java.lang.Thread.State: RUNNABLE

        RBP=0x000000000a85bc50
        0x000000000a85bc50 is pointing to unknown location

        RSI=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        RDI=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        R8 =0x0000000000000032
        0x0000000000000032 is pointing to unknown location

        R9 =0x000000000a85bc50
        0x000000000a85bc50 is pointing to unknown location

        R10=0x0000000000000000
        0x0000000000000000 is pointing to unknown location

        R11=0x0000000000000246
        0x0000000000000246 is pointing to unknown location

        R12=0x0000000000000032
        0x0000000000000032 is pointing to unknown location

        R13=0x00000000c53922d8 {constMethod}
        - klass: {other class}
      • method: 0x00000000c53922e0 {method}

        'accept0' '(Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;[Ljava/net/InetSocketAddress;)I' in 'sun/nio/ch/ServerSocketChannelImpl'

      • exceptions: 0x00000000c4001ef8

      R14=0x000000000837f448
      0x000000000837f448 is pointing into the stack for thread: 0x0000000008aae000
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
      java.lang.Thread.State: RUNNABLE

      R15=0x0000000008aae000
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x0000000008aae000 nid=0xdf4 runnable [0x000000000837f000]
      java.lang.Thread.State: RUNNABLE

      Top of Stack: (sp=0x000000000837f2b0)
      0x000000000837f2b0: 0000000008aae000 0000000000002048
      0x000000000837f2c0: 000000000837f438 000000000837f438
      0x000000000837f2d0: 00000000000009a0 000000000837f430
      0x000000000837f2e0: 0000000000000000 000000000837f438
      0x000000000837f2f0: 0000000008aae1c8 00000000c4832e18
      0x000000000837f300: 0000000000000000 000000006d6c3215
      0x000000000837f310: 000000006dee6520 000000000837f430
      0x000000000837f320: 0000000008aae1c8 00000000c4149600
      0x000000000837f330: 0000000000000001 0000000000000001
      0x000000000837f340: 0000000000000010 50bc850a8fa60002
      0x000000000837f350: 0000000000000000 0000b74226c117d6
      0x000000000837f360: 0000000000000001 00000000012258fa
      0x000000000837f370: 00000000c4149600 000000000837f410
      0x000000000837f380: 00000000c53922e0 00000000012312a0
      0x000000000837f390: 0000000000000001 00000000c53c1f98
      0x000000000837f3a0: 0000000008aae000 000000006da91dbd

      Instructions: (pc=0x000000006dadf79e)
      0x000000006dadf78e: 83 38 02 00 00 06 00 00 00 80 3d 2a ca 41 00 00
      0x000000006dadf79e: 48 8b 16 74 17 44 8b 4a 08 0f b6 0d 0a b0 40 00

      Stack: [0x0000000008360000,0x0000000008380000], sp=0x000000000837f2b0, free space=124k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [jvm.dll+0x24f79e]

      [error occurred during error reporting (printing native stack), id 0xc0000005]

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      j sun.nio.ch.ServerSocketChannelImpl.accept0(Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;[Ljava/net/InetSocketAddress;)I+0
      j sun.nio.ch.ServerSocketChannelImpl.accept()Ljava/nio/channels/SocketChannel;+94
      j com.sun.grizzly.TCPSelectorHandler.acceptWithoutRegistration(Ljava/nio/channels/SelectionKey;)Ljava/nio/channels/SelectableChannel;+11
      j com.sun.enterprise.v3.services.impl.monitor.MonitorableSelectorHandler.acceptWithoutRegistration(Ljava/nio/channels/SelectionKey;)Ljava/nio/channels/SelectableChannel;+2
      j com.sun.grizzly.http.SelectorThreadHandler.onAcceptInterest(Ljava/nio/channels/SelectionKey;Lcom/sun/grizzly/Context;)Z+2
      J com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(Ljava/nio/channels/SelectionKey;Lcom/sun/grizzly/SelectorHandler;Lcom/sun/grizzly/NIOContext;)Z
      J com.sun.grizzly.SelectorHandlerRunner.doSelect(Lcom/sun/grizzly/SelectorHandler;Lcom/sun/grizzly/NIOContext;)Z
      J com.sun.grizzly.SelectorHandlerRunner.run()V
      j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+59
      j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
      j java.lang.Thread.run()V+11
      v ~StubRoutines::call_stub

      1. domain.xml
        42 kB
        sonymanuel
      2. hs_err_pid4216.log
        38 kB
        sonymanuel

        Activity

        Hide
        sonymanuel added a comment -

        The crash log and domain.xml from IEC setup is attached.

        Show
        sonymanuel added a comment - The crash log and domain.xml from IEC setup is attached.
        Hide
        oleksiys added a comment -

        According to the crash reports attached in 7014061, mentioned 4 crashes don't look related.
        Two times it happens in
        com.sun.messaging.jmq.util.UID.equals(...)

        I found the source[1] and it looks like:

        124 /**
        125 * Equals
        126 */
        127 public boolean equals(Object obj) {
        128 if (! (obj instanceof UID))

        { 129 return false; 130 }

        131 return (this.id == ((UID)obj).id);
        132 }

        don't see anything suspicious, no native calls etc.

        Other 2 crashes occur on different places: jmq TransactionReaper and socket accept.

        So, IMO, as i told crashes don't look related and probably they are caused by some general JDK/OS issue.

        WBR.

        [1] http://www.docjar.com/html/api/com/sun/messaging/jmq/util/UID.java.html

        Show
        oleksiys added a comment - According to the crash reports attached in 7014061, mentioned 4 crashes don't look related. Two times it happens in com.sun.messaging.jmq.util.UID.equals(...) I found the source [1] and it looks like: 124 /** 125 * Equals 126 */ 127 public boolean equals(Object obj) { 128 if (! (obj instanceof UID)) { 129 return false; 130 } 131 return (this.id == ((UID)obj).id); 132 } don't see anything suspicious, no native calls etc. Other 2 crashes occur on different places: jmq TransactionReaper and socket accept. So, IMO, as i told crashes don't look related and probably they are caused by some general JDK/OS issue. WBR. [1] http://www.docjar.com/html/api/com/sun/messaging/jmq/util/UID.java.html
        Hide
        amyk added a comment - - edited

        For the com.sun.messaging.jmq.util.UID.equals frame in hs_err logs, suggested GlassFish QE to try potential temporary workarounds described in the "Java Trouble-Shooting and Diagnostic Guide" while JVM engineer is looking into the issue (7014061/jvm).

        [information from Sony (IEC):
        non-SSL run completed 24x7 with JDK 1.6.0_22 64 bit.
        SSL - 1 instance crashed on 5th day. 2 instances completed 24x7. JDK used was 1.6.0_23.]

        Show
        amyk added a comment - - edited For the com.sun.messaging.jmq.util.UID.equals frame in hs_err logs, suggested GlassFish QE to try potential temporary workarounds described in the "Java Trouble-Shooting and Diagnostic Guide" while JVM engineer is looking into the issue (7014061/jvm). [information from Sony (IEC): non-SSL run completed 24x7 with JDK 1.6.0_22 64 bit. SSL - 1 instance crashed on 5th day. 2 instances completed 24x7. JDK used was 1.6.0_23.]
        Hide
        Chris Kasso added a comment -

        The test completed with SSL enabled using update 24. If we see the problem again on U24 we can reopen the issue.

        Show
        Chris Kasso added a comment - The test completed with SSL enabled using update 24. If we see the problem again on U24 we can reopen the issue.
        Hide
        Chris Kasso added a comment -

        Cannot reproduce on u24

        Show
        Chris Kasso added a comment - Cannot reproduce on u24

          People

          • Assignee:
            Chris Kasso
            Reporter:
            easarina
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: