sailfin
  1. sailfin
  2. SAILFIN-1956

Memory growth observed in some instances after failure/restart for converged scenario

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: milestone 1
    • Component/s: session_replication
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

      Description

      ********************************************************************************

      • Template v0.1 ( 05/01/08 )
      • Sailfin Stress test issue
        ********************************************************************************
        Sailfin Build :30
        Cluster size : 10 instances
        Happens in a single instance (y/n) n/a:
        Test id : st5_1_conference
        Location of the test : as-telco-sqe/stress-ws/conference
        JDK version : 1.6.0_16
        CLB used : Yes
        SSR enabled : Yes
        CPS : 75 cps
        **********************************************************************

      At 30 minutes from start, instance and node-agent were killed. The instance was
      restarted after 30 minutes. The scenario completed 24x1 but memory growth was
      observed in 3 instances. Some thread deadlocks were also observed on these 3
      instances (issue 1953).

      Unfortunately the runmanager had stopped the instances so there was not chance
      to get a jmap. Attached is the runmanager cpu/memory plot which shows the growth.

      Will run again and attach the jmap info.

        Issue Links

          Activity

          Hide
          sonymanuel added a comment -

          Created an attachment (id=1106)
          cpu/memory plot for instance 104,105 & 106

          Show
          sonymanuel added a comment - Created an attachment (id=1106) cpu/memory plot for instance 104,105 & 106
          Hide
          sonymanuel added a comment -

          Created an attachment (id=1107)
          cpu/memory plot for instance 108,9 & 10

          Show
          sonymanuel added a comment - Created an attachment (id=1107) cpu/memory plot for instance 108,9 & 10
          Hide
          Mahesh Kannan added a comment -

          Assigning to Bhavani

          Show
          Mahesh Kannan added a comment - Assigning to Bhavani
          Hide
          sonymanuel added a comment -

          Created an attachment (id=1109)
          jmap histos from instance101 taken at 2 hour intervals

          Show
          sonymanuel added a comment - Created an attachment (id=1109) jmap histos from instance101 taken at 2 hour intervals
          Hide
          sonymanuel added a comment -

          Ran the test with 2-Sep-09 nightly build. 2 instances , instance101 &
          instance105(killed and restarted) showed memory growth. These are the instances
          which also had thread deadlocks (1953).

          Attached a set of jmap histo taken at 2 hours intervals on instance101.

          jmaps for for all the instance available at
          sf-x2200-14:/space/system-test/logs/issues/1956

          A heap dump for instance101 is available in
          /space/system-test/logs/issues/1956/sf-sb6000-101

          Show
          sonymanuel added a comment - Ran the test with 2-Sep-09 nightly build. 2 instances , instance101 & instance105(killed and restarted) showed memory growth. These are the instances which also had thread deadlocks (1953). Attached a set of jmap histo taken at 2 hours intervals on instance101. jmaps for for all the instance available at sf-x2200-14:/space/system-test/logs/issues/1956 A heap dump for instance101 is available in /space/system-test/logs/issues/1956/sf-sb6000-101
          Hide
          shreedhar_ganapathy added a comment -

          Assigned to Mahesh

          Show
          shreedhar_ganapathy added a comment - Assigned to Mahesh
          Hide
          Mahesh Kannan added a comment -

          The heap dump indicates that LinkedBlockingQueue is growing in size. Are there
          any logs available from instance101 and instance105?

          Also, have we run this with the fix for 1953?

          Show
          Mahesh Kannan added a comment - The heap dump indicates that LinkedBlockingQueue is growing in size. Are there any logs available from instance101 and instance105? Also, have we run this with the fix for 1953?
          Hide
          sonymanuel added a comment -

          Fix for 1953 seems to have fixed this. No memory growth observed over a 24x1
          period. Marking as resolved.

          Show
          sonymanuel added a comment - Fix for 1953 seems to have fixed this. No memory growth observed over a 24x1 period. Marking as resolved.
          Hide
          varunrupela added a comment -

          updated keyword

          Show
          varunrupela added a comment - updated keyword

            People

            • Assignee:
              Mahesh Kannan
              Reporter:
              sonymanuel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: