sailfin
  1. sailfin
  2. SAILFIN-1954

Possible deadlock if a converged application uses SAS object as synchronization lock.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0
    • Fix Version/s: milestone 1
    • Component/s: sip_container
    • Labels:
      None
    • Environment:

      Operating System: Linux
      Platform: Other

    • Issuezilla Id:
      1,954

      Description

      The converged conference application having http and sip servlets uses SAS as a
      lock to synchronize access between SIP and Http worker threads when updating
      attribute values stored in the SAS object. We observed the following deadlock in
      one of the instances.

      Found one Java-level deadlock:
      =============================
      "httpSSLWorkerThread-38080-9":
      waiting to lock monitor 0x00002aac37fc9470 (object 0x00002aac29e31780, a
      java.lang.Class),
      which is held by "SipContainer-serversWorkerThread-5060-7"
      "SipContainer-serversWorkerThread-5060-7":
      waiting to lock monitor 0x00002aac45aa4e78 (object 0x00002aaad00ecd48, a
      java.lang.Object),
      which is held by "httpSSLWorkerThread-38080-2"
      "httpSSLWorkerThread-38080-2":
      waiting to lock monitor 0x00002aac37fc89f0 (object 0x00002aaaf92bb670, a
      org.jvnet.glassfish.comms.replication.sessmgmt.FullHASipApplicationSession),
      which is held by "SipContainer-serversWorkerThread-5060-7"

      Java stack information for the threads listed above:
      ===================================================
      "httpSSLWorkerThread-38080-9":
      at com.sun.asqe.systemtest.conference.util.Util.getAndSetAttribute(Util.java:62)

      • waiting to lock <0x00002aac29e31780> (a java.lang.Class for
        com.sun.asqe.systemtest.conference.util.Util)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.processLinkSessions(ConferenceHTTPServlet.java:142)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.processRequest(ConferenceHTTPServlet.java:40)
        at
        com.sun.asqe.systemtest.conference.http.ConferenceHTTPServlet.doPost(ConferenceHTTPServlet.java:262)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
        at
        org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
        at
        org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:315)
        at
        org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
        at
        org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
        at
        com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
        at
        com.sun.enterprise.ee.web.sessmgmt.SessionLockingStandardPipeline.invoke(SessionLockingStandardPipeline.java:120)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at
        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:872)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
        at
        org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
        at
        com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
        "SipContainer-serversWorkerThread-5060-7":
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.getAttribute(SipApplicationSessionBase.java:185)
      • waiting to lock <0x00002aaad00ecd48> (a java.lang.Object)
        at com.sun.asqe.systemtest.conference.util.Util.getAndSetAttribute(Util.java:63)
      • locked <0x00002aaaf92bb670> (a
        org.jvnet.glassfish.comms.replication.sessmgmt.FullHASipApplicationSession)
      • locked <0x00002aac29e31780> (a java.lang.Class for
        com.sun.asqe.systemtest.conference.util.Util)
        at
        com.sun.asqe.systemtest.conference.sip.ConferenceSipServlet.doAck(ConferenceSipServlet.java:72)
        at javax.servlet.sip.SipServlet.doRequest(SipServlet.java:59)
        at javax.servlet.sip.SipServlet.service(SipServlet.java:46)
        at
        com.ericsson.ssa.container.sim.SipServletFacade.service(SipServletFacade.java:121)
        at com.ericsson.ssa.sip.INVITESession.dispatch(INVITESession.java:1365)
        at com.ericsson.ssa.sip.UA.dispatch(UA.java:657)
        at
        com.ericsson.ssa.container.sim.ApplicationDispatcher.dispatchViaStatelessProxy(ApplicationDispatcher.java:638)
        at
        com.ericsson.ssa.container.sim.ApplicationDispatcher.dispatch(ApplicationDispatcher.java:178)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:53)
        at
        com.ericsson.ssa.container.sim.ApplicationDispatcher.next(ApplicationDispatcher.java:697)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at com.ericsson.ssa.sip.LocalRouteManager.next(LocalRouteManager.java:98)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at com.ericsson.ssa.sip.DialogManager.next(DialogManager.java:568)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at com.ericsson.ssa.sip.dns.ResolverManager.next(ResolverManager.java:169)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        com.ericsson.ssa.sip.persistence.ReplicationManager.next(ReplicationManager.java:102)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        com.ericsson.ssa.sip.transaction.TransactionManager.next(TransactionManager.java:283)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        org.jvnet.glassfish.comms.clb.core.sip.SipLoadBalancerManagerBackEnd.next(SipLoadBalancerManagerBackEnd.java:193)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        org.jvnet.glassfish.comms.clb.core.sip.SipLoadBalancerManager.next(SipLoadBalancerManager.java:157)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        com.ericsson.ssa.container.OutboundFlowManager.processOutboundRequest(OutboundFlowManager.java:183)
        at com.ericsson.ssa.container.OutboundFlowManager.next(OutboundFlowManager.java:98)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        com.ericsson.ssa.container.GrizzlyNetworkManager.next(GrizzlyNetworkManager.java:1331)
        at com.ericsson.ssa.sip.LayerHelper.next(LayerHelper.java:60)
        at
        com.ericsson.ssa.container.MessageProcessorFilter.processMessage(MessageProcessorFilter.java:416)
        at
        com.ericsson.ssa.container.MessageProcessorFilter.execute(MessageProcessorFilter.java:315)
        at
        com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:136)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:103)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:89)
        at
        com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:67)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:56)
        at com.sun.grizzly.util.WorkerThreadImpl.processTask(WorkerThreadImpl.java:325)
        at com.sun.grizzly.util.WorkerThreadImpl.run(WorkerThreadImpl.java:184)
        "httpSSLWorkerThread-38080-2":
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.lockForeground(SipApplicationSessionBase.java:1222)
      • waiting to lock <0x00002aaaf92bb670> (a
        org.jvnet.glassfish.comms.replication.sessmgmt.FullHASipApplicationSession)
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.lockForegroundWithRetry(SipApplicationSessionBase.java:1176)
        at com.sun.appserv.ha.uow.ReplicationUnitOfWork.add(ReplicationUnitOfWork.java:123)
      • locked <0x00002aaad2803828> (a
        com.ericsson.ssa.sip.persistence.ReplicationUnitOfWork)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.HASipApplicationSession.addToUnitOfWork(HASipApplicationSession.java:676)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.HASipApplicationSession.setShouldBePersisted(HASipApplicationSession.java:576)
        at
        com.ericsson.ssa.sip.SipApplicationSessionImpl.addPFieldHttpProtocolSession(SipApplicationSessionImpl.java:767)
        at
        org.jvnet.glassfish.comms.replication.sessmgmt.HASipApplicationSession.addPFieldHttpProtocolSession(HASipApplicationSession.java:634)
        at
        com.ericsson.ssa.sip.SipApplicationSessionBase.addSession(SipApplicationSessionBase.java:719)
      • locked <0x00002aaad00ecd48> (a java.lang.Object)
        at
        com.ericsson.ssa.sip.ConvergedHttpSessionFacade.getApplicationSession(ConvergedHttpSessionFacade.java:198)
        at
        com.ericsson.ssa.config.ConvergedContextImpl.beginPipelineInvoke(ConvergedContextImpl.java:606)
        at
        com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:96)
        at
        com.sun.enterprise.ee.web.sessmgmt.SessionLockingStandardPipeline.invoke(SessionLockingStandardPipeline.java:120)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at
        org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
        at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
        at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:872)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
        at
        com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
        at
        org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
        at
        com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

      Found 1 deadlock.

        Activity

        Hide
        sonymanuel added a comment -

        Created an attachment (id=1104)
        jstack

        Show
        sonymanuel added a comment - Created an attachment (id=1104) jstack
        Hide
        sonymanuel added a comment -

        Workaround suggested by Sankar/Bhavani.

        Locking SAS may not be good idea as spec doesn't mention any session locking
        semantics and it is not guaranteed to work. See the below snippet for more
        information. Unfortunately there is no easy way to achieve the same, but how
        this has to be achieved is out side the scope. A good container implementation
        should support to achieve this functionality.
        You can achieve this functionality by doing the following snippet of code

        SAS Listener.sessionCreate()

        { sas.setAttribute("lock", mySASLocak); }

        in your servlet:

        synchronized(sas.getAttribute("lock"))

        { sas.setAttribute(..., ...); }

        The test is running fine after following the suggested approach.

        Show
        sonymanuel added a comment - Workaround suggested by Sankar/Bhavani. Locking SAS may not be good idea as spec doesn't mention any session locking semantics and it is not guaranteed to work. See the below snippet for more information. Unfortunately there is no easy way to achieve the same, but how this has to be achieved is out side the scope. A good container implementation should support to achieve this functionality. You can achieve this functionality by doing the following snippet of code SAS Listener.sessionCreate() { sas.setAttribute("lock", mySASLocak); } in your servlet: synchronized(sas.getAttribute("lock")) { sas.setAttribute(..., ...); } The test is running fine after following the suggested approach.
        Hide
        sankara added a comment -

        Scheduling it for next release.

        Show
        sankara added a comment - Scheduling it for next release.
        Hide
        chinmayee_srivathsa added a comment -

        This issue is documented in the SailFin 2.0 Release Notes as a known issue.

        Show
        chinmayee_srivathsa added a comment - This issue is documented in the SailFin 2.0 Release Notes as a known issue.

          People

          • Assignee:
            sankara
            Reporter:
            sonymanuel
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: