sailfin
  1. sailfin
  2. SAILFIN-1517

Replication disabled after restarting a stopped instance

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: b61
    • Component/s: session_replication
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

      Description

      This is a sub-issue of 1495. Tested with 15th dec nightly build.

      After 1.5 hours after restarting the stopped instance we see the following message.

      [#|2008-12-16T14:29:22.376+0530|INFO|sun-glassfish-comms-server1.5|com.sun.enterprise.ee.web.sessmgmt.pipe|_ThreadID=62;_ThreadName=ReplicationSenderExecutor

      • 181;|JxtaSenderPipeManager::pipePoolCount decremented = 0|#]

      [#|2008-12-16T14:29:22.376+0530|WARNING|sun-glassfish-comms-server1.5|javax.enterprise.system.container.web|_ThreadID=62;_ThreadName=ReplicationSenderExecutor

      • 181;_RequestID=f332cf14-7f23-4b9c-9470-628b7363f078;|Out of pipes in
        JxtaReplicationSender pipepool. Disabling replication.|#]

      Looks like replication is disabled and not resuming after this point.

        Issue Links

          Activity

          Hide
          sonymanuel added a comment -

          Created an attachment (id=883)
          instance104 server.log

          Show
          sonymanuel added a comment - Created an attachment (id=883) instance104 server.log
          Hide
          sonymanuel added a comment -

          On the 10 instance cluster , instance105 was stopped. Then restarted after 30
          minutes. On instance104 replication got disabled about 1.5 hours after restart.
          Then subsequently instance105 went out of memory. About one hour after the
          replication stopped.

          Will upload logs to
          http://agni-1.sfbay.sun.com/net/asqe-logs/export1/SailFin/Results/ST/b60d/1517

          Show
          sonymanuel added a comment - On the 10 instance cluster , instance105 was stopped. Then restarted after 30 minutes. On instance104 replication got disabled about 1.5 hours after restart. Then subsequently instance105 went out of memory. About one hour after the replication stopped. Will upload logs to http://agni-1.sfbay.sun.com/net/asqe-logs/export1/SailFin/Results/ST/b60d/1517
          Show
          sonymanuel added a comment - Logs available @ http://agni-1.sfbay.sun.com/net/asqe-logs/export1/SailFin/Results/ST/b60d-nb/1517/
          Hide
          Mahesh Kannan added a comment -

          The reason why replication didn't resume was, for some reason, this instance
          received a PlannedShutdownEvent. Also, many of the instances (107, 104, 110,
          103, 101, 102, 108) received GROUP_SHUTDOWN event whreas 109, 106 and 10-5 didn't.

          So, the fix that went in for 1436 and 1434 did not kick in.

          I will update the bug once I find out why some of them received shutdown event.

          Show
          Mahesh Kannan added a comment - The reason why replication didn't resume was, for some reason, this instance received a PlannedShutdownEvent. Also, many of the instances (107, 104, 110, 103, 101, 102, 108) received GROUP_SHUTDOWN event whreas 109, 106 and 10-5 didn't. So, the fix that went in for 1436 and 1434 did not kick in. I will update the bug once I find out why some of them received shutdown event.
          Hide
          prasads added a comment -

          "Reclassifying as P4 because these issues are not must fix for SailFin 1.0 release.
          This issue will be scrubbed after this release and will be given the right
          priority for SailFin 2.0 release."

          Show
          prasads added a comment - "Reclassifying as P4 because these issues are not must fix for SailFin 1.0 release. This issue will be scrubbed after this release and will be given the right priority for SailFin 2.0 release."
          Hide
          lwhite added a comment -

          These fixes were checked in to fix a number of umbrella issues including
          1477, 1478, 1486, 1495, 1517, 1536 and 1550

          Checking in DynamicOwnershipChangeHandler.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/DynamicOwnershipChangeHandler.java,v
          <-- DynamicOwnershipChangeHandler.java
          new revision: 1.1.2.4; previous revision: 1.1.2.3
          done
          Checking in DynamicOwnershipManager.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/DynamicOwnershipManager.java,v
          <-- DynamicOwnershipManager.java
          new revision: 1.1.2.4; previous revision: 1.1.2.3
          done
          Checking in InputPipeWrapper.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/InputPipeWrapper.java,v
          <-- InputPipeWrapper.java
          new revision: 1.8.6.12; previous revision: 1.8.6.11
          done
          Checking in JoinNotificationEventHandler.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JoinNotificationEventHandler.java,v
          <-- JoinNotificationEventHandler.java
          new revision: 1.11.2.12; previous revision: 1.11.2.11
          done
          Checking in JxtaBackingStoreImpl.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaBackingStoreImpl.java,v
          <-- JxtaBackingStoreImpl.java
          new revision: 1.18.2.24; previous revision: 1.18.2.23
          done
          Checking in JxtaBiDiPipeWrapper.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaBiDiPipeWrapper.java,v
          <-- JxtaBiDiPipeWrapper.java
          new revision: 1.17.2.25; previous revision: 1.17.2.24
          done
          Checking in JxtaReplicationReceiver.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationReceiver.java,v
          <-- JxtaReplicationReceiver.java
          new revision: 1.11.2.12; previous revision: 1.11.2.11
          done
          Checking in JxtaReplicationSender.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationSender.java,v
          <-- JxtaReplicationSender.java
          new revision: 1.18.2.31; previous revision: 1.18.2.30
          done
          Checking in JxtaReplicationUnicastSender.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/JxtaReplicationUnicastSender.java,v
          <-- JxtaReplicationUnicastSender.java
          new revision: 1.1.2.4; previous revision: 1.1.2.3
          done
          Checking in ReplicationHealthChecker.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationHealthChecker.java,v
          <-- ReplicationHealthChecker.java
          new revision: 1.9.2.25; previous revision: 1.9.2.24
          done
          Checking in ReplicationMessageRouter.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationMessageRouter.java,v
          <-- ReplicationMessageRouter.java
          new revision: 1.8.2.23; previous revision: 1.8.2.22
          done

          Checking in dialogmgmt/ReplicationDialogFragmentManager.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/dialogmgmt/ReplicationDialogFragmentManager.java,v
          <-- ReplicationDialogFragmentManager.java
          new revision: 1.67; previous revision: 1.66
          done
          Checking in sessmgmt/ActivationHelper.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ActivationHelper.java,v
          <-- ActivationHelper.java
          new revision: 1.17; previous revision: 1.16
          done
          Removing sessmgmt/ExpatListQueryTask.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ExpatListQueryTask.java,v
          <-- ExpatListQueryTask.java
          new revision: delete; previous revision: 1.1
          done
          Checking in sessmgmt/HAServletTimer.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/HAServletTimer.java,v
          <-- HAServletTimer.java
          new revision: 1.59; previous revision: 1.58
          done
          Checking in sessmgmt/HASipApplicationSession.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/HASipApplicationSession.java,v
          <-- HASipApplicationSession.java
          new revision: 1.78; previous revision: 1.77
          done
          Checking in sessmgmt/ServletTimerStoreImpl.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ServletTimerStoreImpl.java,v
          <-- ServletTimerStoreImpl.java
          new revision: 1.54; previous revision: 1.53
          done
          Checking in sessmgmt/SipApplicationSessionStoreImpl.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipApplicationSessionStoreImpl.java,v
          <-- SipApplicationSessionStoreImpl.java
          new revision: 1.61; previous revision: 1.60
          done
          Checking in sessmgmt/SipReplicationUtil.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipReplicationUtil.java,v
          <-- SipReplicationUtil.java
          new revision: 1.3; previous revision: 1.2
          done
          Checking in sessmgmt/SipTransactionPersistentManager.java;
          /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipTransactionPersistentManager.java,v
          <-- SipTransactionPersistentManager.java
          new revision: 1.162; previous revision: 1.161
          done

          Show
          lwhite added a comment - These fixes were checked in to fix a number of umbrella issues including 1477, 1478, 1486, 1495, 1517, 1536 and 1550 Checking in DynamicOwnershipChangeHandler.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/DynamicOwnershipChangeHandler.java,v <-- DynamicOwnershipChangeHandler.java new revision: 1.1.2.4; previous revision: 1.1.2.3 done Checking in DynamicOwnershipManager.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/DynamicOwnershipManager.java,v <-- DynamicOwnershipManager.java new revision: 1.1.2.4; previous revision: 1.1.2.3 done Checking in InputPipeWrapper.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/InputPipeWrapper.java,v <-- InputPipeWrapper.java new revision: 1.8.6.12; previous revision: 1.8.6.11 done Checking in JoinNotificationEventHandler.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JoinNotificationEventHandler.java,v <-- JoinNotificationEventHandler.java new revision: 1.11.2.12; previous revision: 1.11.2.11 done Checking in JxtaBackingStoreImpl.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaBackingStoreImpl.java,v <-- JxtaBackingStoreImpl.java new revision: 1.18.2.24; previous revision: 1.18.2.23 done Checking in JxtaBiDiPipeWrapper.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaBiDiPipeWrapper.java,v <-- JxtaBiDiPipeWrapper.java new revision: 1.17.2.25; previous revision: 1.17.2.24 done Checking in JxtaReplicationReceiver.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationReceiver.java,v <-- JxtaReplicationReceiver.java new revision: 1.11.2.12; previous revision: 1.11.2.11 done Checking in JxtaReplicationSender.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationSender.java,v <-- JxtaReplicationSender.java new revision: 1.18.2.31; previous revision: 1.18.2.30 done Checking in JxtaReplicationUnicastSender.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/Attic/JxtaReplicationUnicastSender.java,v <-- JxtaReplicationUnicastSender.java new revision: 1.1.2.4; previous revision: 1.1.2.3 done Checking in ReplicationHealthChecker.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationHealthChecker.java,v <-- ReplicationHealthChecker.java new revision: 1.9.2.25; previous revision: 1.9.2.24 done Checking in ReplicationMessageRouter.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationMessageRouter.java,v <-- ReplicationMessageRouter.java new revision: 1.8.2.23; previous revision: 1.8.2.22 done Checking in dialogmgmt/ReplicationDialogFragmentManager.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/dialogmgmt/ReplicationDialogFragmentManager.java,v <-- ReplicationDialogFragmentManager.java new revision: 1.67; previous revision: 1.66 done Checking in sessmgmt/ActivationHelper.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ActivationHelper.java,v <-- ActivationHelper.java new revision: 1.17; previous revision: 1.16 done Removing sessmgmt/ExpatListQueryTask.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ExpatListQueryTask.java,v <-- ExpatListQueryTask.java new revision: delete; previous revision: 1.1 done Checking in sessmgmt/HAServletTimer.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/HAServletTimer.java,v <-- HAServletTimer.java new revision: 1.59; previous revision: 1.58 done Checking in sessmgmt/HASipApplicationSession.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/HASipApplicationSession.java,v <-- HASipApplicationSession.java new revision: 1.78; previous revision: 1.77 done Checking in sessmgmt/ServletTimerStoreImpl.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/ServletTimerStoreImpl.java,v <-- ServletTimerStoreImpl.java new revision: 1.54; previous revision: 1.53 done Checking in sessmgmt/SipApplicationSessionStoreImpl.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipApplicationSessionStoreImpl.java,v <-- SipApplicationSessionStoreImpl.java new revision: 1.61; previous revision: 1.60 done Checking in sessmgmt/SipReplicationUtil.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipReplicationUtil.java,v <-- SipReplicationUtil.java new revision: 1.3; previous revision: 1.2 done Checking in sessmgmt/SipTransactionPersistentManager.java; /cvs/sailfin/replication/src/main/java/org/jvnet/glassfish/comms/replication/sessmgmt/SipTransactionPersistentManager.java,v <-- SipTransactionPersistentManager.java new revision: 1.162; previous revision: 1.161 done

            People

            • Assignee:
              jluehe
              Reporter:
              sonymanuel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: