sailfin
  1. sailfin
  2. SAILFIN-1953

Multiple deadlocks detected when running converged app.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: b32
    • Component/s: session_replication
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

      Description

      ********************************************************************************

      • Template v0.1 ( 05/01/08 )
      • Sailfin Stress test issue
        ********************************************************************************
        Sailfin Build :28
        Cluster size : 9 instances
        Happens in a single instance (y/n) n/a:
        Test id : st5_1_conference
        Location of the test : as-telco-sqe/stress-ws/conference
        JDK version : 1.6.0_16
        CLB used : Yes
        HW LB used : NO
        SSR enabled : Yes
        CPS : 75
        **********************************************************************

      Analyzing jstack from all the instances after a failure/restart of the scenario
      we see some threads deadlocked in 3 instances. Its hard to say when it happened
      but I am assuming it after failure or restart. See the attached jstacks for details.

      Found one Java-level deadlock:
      =============================
      "ReplicationSenderExecutor - 45":
      waiting to lock monitor 0x00002aac3aa37230 (object 0x00002aaac1214338, a
      java.lang.Object),
      which is held by "httpSSLWorkerThread-38080-0"
      "httpSSLWorkerThread-38080-0":
      waiting to lock monitor 0x00002aac3749a108 (object 0x00002aaac1214320, a
      java.lang.Object),
      which is held by "SipContainer-serversWorkerThread-5060-4"
      "SipContainer-serversWorkerThread-5060-4":
      waiting to lock monitor 0x00002aac3aa37230 (object 0x00002aaac1214338, a
      java.lang.Object),
      which is held by "httpSSLWorkerThread-38080-0"

      Java stack information for the threads listed above:
      ===================================================
      "ReplicationSenderExecutor - 45":
      at
      com.ericsson.ssa.sip.SipSessionManagerBase.initSipApplicationSession(SipSessionManagerBase.java:772)

      • waiting to lock <0x00002aaac1214338> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.SipSessionManagerBase.findSipApplicationSession(SipSessionManagerBase.java:238)
        at
        com.ericsson.ssa.sip.PersistentSipSessionManagerBase.findSipApplicationSessionFromCacheOnly(PersistentSipSessionManagerBase.java:301)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.findSipApplicationSessionFromCacheOnly(SipTransactionPersistentManager.java:2548)
      • locked <0x00002aaaca03cfc0> (a java.lang.Object)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.processLoadadvisorysas(SipTransactionPersistentManager.java:1539)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.processLoadadvisorysas(SipTransactionPersistentManager.java:1487)
        at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source)
        at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.doProcessQueryMessage(SipTransactionPersistentManager.java:6026)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.processQueryMessage(SipTransactionPersistentManager.java:6055)
        at
        com.sun.enterprise.ee.web.sessmgmt.ReplicationMessageRouter.routeQueryMessageForApp(ReplicationMessageRouter.java:857)
        at
        com.sun.enterprise.ee.web.sessmgmt.ReplicationMessageRouter.processQueryMessage(ReplicationMessageRouter.java:741)
        at
        com.sun.enterprise.ee.web.sessmgmt.MessageProcessor$ReceivedMessageWrapper.run(MessageProcessor.java:442)
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
        "httpSSLWorkerThread-38080-0":
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.findSipSession(SipTransactionPersistentManager.java:2919)
      • waiting to lock <0x00002aaac1214320> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.SipApplicationSessionImpl$1$1.hasNext(SipApplicationSessionImpl.java:387)
        at
        com.ericsson.ssa.sip.SipApplicationSessionImpl.hasValidSipSession(SipApplicationSessionImpl.java:556)
        at
        com.ericsson.ssa.sip.SipApplicationSessionImpl.isEmpty(SipApplicationSessionImpl.java:590)
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.isReadyToInvalidate(SipApplicationSessionBase.java:954)
      • locked <0x00002aaac1214338> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.checkSessionReadyToInvalidate(SipApplicationSessionBase.java:968)
        at
        com.ericsson.ssa.sip.SipApplicationSessionImpl.removePFieldHttpProtocolSession(SipApplicationSessionImpl.java:753)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.HASipApplicationSession.removePFieldHttpProtocolSession(HASipApplicationSession.java:652)
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.removeSession(SipApplicationSessionBase.java:729)
      • locked <0x00002aaac1214338> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.ConvergedHttpSessionFacade.removeFromSAS(ConvergedHttpSessionFacade.java:110)
        at
        com.ericsson.ssa.sip.ConvergedHttpSessionFacade.invalidate(ConvergedHttpSessionFacade.java:99)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.processFinalPage(ConferenceHTTPServlet.java:238)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.processRequest(ConferenceHTTPServlet.java:44)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.doPost(ConferenceHTTPServlet.java:262)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at
        org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
        at
        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
        at
        org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
        at
        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at
        com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at
        com.sun.enterprise.ee.web.sessmgmt.SessionLockingStandardPipeline.invoke(SessionLockingStandardPipeline.java:120)
        at
        org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at
        org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at
        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at
        org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at
        org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:872)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
        at
        org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
        at
        com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
        "SipContainer-serversWorkerThread-5060-4":
        at
        com.ericsson.ssa.sip.SipSessionManagerBase.initSipApplicationSession(SipSessionManagerBase.java:772)
      • waiting to lock <0x00002aaac1214338> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.SipSessionManagerBase.findSipApplicationSession(SipSessionManagerBase.java:238)
        at
        com.ericsson.ssa.sip.PersistentSipSessionManagerBase.findSipApplicationSession(PersistentSipSessionManagerBase.java:225)
        at
        com.ericsson.ssa.sip.PersistentSipSessionManagerBase.findSipApplicationSession(PersistentSipSessionManagerBase.java:210)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.HASipSession.validate(HASipSession.java:345)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipSessionStoreImpl.validateAndSave(SipSessionStoreImpl.java:194)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipSessionStoreImpl.loadFromBackingStore(SipSessionStoreImpl.java:177)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipSessionStoreImpl.load(SipSessionStoreImpl.java:133)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipSessionStoreImpl.load(SipSessionStoreImpl.java:118)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.swapInSipSession(SipTransactionPersistentManager.java:5301)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.swapInSipSession(SipTransactionPersistentManager.java:5256)
        at
        com.ericsson.ssa.sip.PersistentSipSessionManagerBase.findSipSession(PersistentSipSessionManagerBase.java:113)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.SipTransactionPersistentManager.findSipSession(SipTransactionPersistentManager.java:2920)
      • locked <0x00002aaac1214320> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.PersistentSipSessionManagerBase.findSipSession(PersistentSipSessionManagerBase.java:79)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.HADialogFragment.validate(HADialogFragment.java:311)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.DialogFragmentStoreImpl.validateAndSave(DialogFragmentStoreImpl.java:197)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.DialogFragmentStoreImpl.loadFromBackingStore(DialogFragmentStoreImpl.java:185)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.DialogFragmentStoreImpl.load(DialogFragmentStoreImpl.java:140)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.DialogFragmentStoreImpl.load(DialogFragmentStoreImpl.java:113)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.ReplicationDialogFragmentManager.swapInDialogFragment(ReplicationDialogFragmentManager.java:980)
        at
        org.jvnet.glassfish.comms.replication.dialogmgmt.ReplicationDialogFragmentManager.swapInDialogFragment(ReplicationDialogFragmentManager.java:941)
        at
        com.ericsson.ssa.sip.PersistentDialogFragmentManager.findDialogFragment(PersistentDialogFragmentManager.java:93)
        at com.ericsson.ssa.sip.DialogManager.getSession(DialogManager.java:1079)
        at
        com.ericsson.ssa.sip.DialogManager.setDialogContext(DialogManager.java:412)

      3747,2-9 87%
      at com.ericsson.ssa.sip.DialogManager.getSession(DialogManager.java:1079)
      at
      com.ericsson.ssa.sip.DialogManager.setDialogContext(DialogManager.java:412)
      at com.ericsson.ssa.sip.DialogManager.next(DialogManager.java:485)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at com.ericsson.ssa.sip.dns.ResolverManager.next(ResolverManager.java:169)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      com.ericsson.ssa.sip.persistence.ReplicationManager.next(ReplicationManager.java:102)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      com.ericsson.ssa.sip.transaction.TransactionManager.invokeCreatedOrFetchedServerTransaction(TransactionManager.java:235)
      at
      com.ericsson.ssa.sip.transaction.TransactionManager.next(TransactionManager.java:288)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      org.jvnet.glassfish.comms.clb.core.sip.SipLoadBalancerManagerBackEnd.next(SipLoadBalancerManagerBackEnd.java:193)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      org.jvnet.glassfish.comms.clb.core.sip.SipLoadBalancerManager.next(SipLoadBalancerManager.java:157)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      com.ericsson.ssa.container.OutboundFlowManager.processOutboundRequest(OutboundFlowManager.java:183)
      at
      com.ericsson.ssa.container.OutboundFlowManager.next(OutboundFlowManager.java:98)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      com.ericsson.ssa.container.GrizzlyNetworkManager.next(GrizzlyNetworkManager.java:1331)
      at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
      at
      com.ericsson.ssa.container.MessageProcessorFilter.processMessage(MessageProcessorFilter.java:416)
      at
      com.ericsson.ssa.container.MessageProcessorFilter.execute(MessageProcessorFilter.java:315)
      at
      com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
      at
      com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
      at
      com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
      at
      com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:67)
      at
      com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
      at
      com.sun.grizzly.util.WorkerThreadImpl.processTask(WorkerThreadImpl.java:325)
      at com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:184)

      Found one Java-level deadlock:

      1. jstack_10864_02-09-09_16-08-32
        312 kB
        sonymanuel
      2. jstack_11013_02-09-09_16-08-17
        313 kB
        sonymanuel
      3. jstack_12395_15-09-09_20-33-53
        330 kB
        sonymanuel
      4. jstack_18040_03-09-09_11-30-36
        398 kB
        sonymanuel
      5. jstack_1915_02-09-09_16-08-51
        317 kB
        sonymanuel

        Issue Links

          Activity

          Hide
          sonymanuel added a comment -

          Created an attachment (id=1116)
          14 sep nightly build. instance105 jstack

          Show
          sonymanuel added a comment - Created an attachment (id=1116) 14 sep nightly build. instance105 jstack
          Hide
          Bhavanishankar added a comment -

          The build you are using (14 sep) does not have my fix.

          Please use
          http://download.java.net/javaee5/sailfin/v2_branch/nightly/Linux/sailfin-image/jars/sailfin-image-v2-b31b-nightly-15_sep_2009.jar
          onwards (or upcoming b31b) to verify this issue.

          Show
          Bhavanishankar added a comment - The build you are using (14 sep) does not have my fix. Please use http://download.java.net/javaee5/sailfin/v2_branch/nightly/Linux/sailfin-image/jars/sailfin-image-v2-b31b-nightly-15_sep_2009.jar onwards (or upcoming b31b) to verify this issue.
          Hide
          sonymanuel added a comment -

          Verified with 15th September nightly build. Over a 24x1 period injected failure
          and restarted the instance. This deadlock was not seen. Saw another deadlock -
          issue 1982. Marking as resolved and verified.

          Show
          sonymanuel added a comment - Verified with 15th September nightly build. Over a 24x1 period injected failure and restarted the instance. This deadlock was not seen. Saw another deadlock - issue 1982. Marking as resolved and verified.
          Hide
          sonymanuel added a comment -

          Verified with 15th Sep nightly build.

          Show
          sonymanuel added a comment - Verified with 15th Sep nightly build.
          Hide
          varunrupela added a comment -

          updated keyword

          Show
          varunrupela added a comment - updated keyword

            People

            • Assignee:
              Bhavanishankar
              Reporter:
              sonymanuel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: