sailfin
  1. sailfin
  2. SAILFIN-1742

IllegalArguementException on the restarted instance at DataCentricUtilImpl.getFailoverServerInstance(DataCentricUtilImpl.java:161)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: milestone 1
    • Component/s: session_replication
    • Labels:
      None
    • Environment:

      Operating System: Linux
      Platform: Other

      Description

      Parent Issue: 1739

      Time Line:
      2009-04-24T16:41:07 - Killed instance110 and its NA process
      2009-04-25T18:31:41 - restarted instance110

      Logs Location:
      /net/asqe-logs.sfbay.sun.com/export1/SailFin/Results/ST/2.0/converged-failure/fail/b11-1failure-1restart/

      The following IllegalArguementException appears on the restarted instance soon
      after it is started back up.

      [#|2009-04-25T18:32:37.173+0530|SEVERE|sun-glassfish-comms-server1.5|org.apache.coyote.tomcat5.CoyoteAdapter|_ThreadID=27;_ThreadName=httpSSLWorkerThread-38080-3;_RequestID=1b72dafe-fa13-439f-bbdf-5f40e4afb844;|PWC3989:
      An exception or error occurred in the container during the request processing
      java.lang.IllegalArgumentException: Null key provided to lookup failover server
      instance
      at
      org.jvnet.glassfish.comms.clb.core.sip.DataCentricUtilImpl.getFailoverServerInstance(DataCentricUtilImpl.java:161)
      at
      com.ericsson.ssa.sip.SipApplicationSessionUtil.getFailoverServerInstanceForBeKey(SipApplicationSessionUtil.java:108)
      at
      org.jvnet.glassfish.comms.replication.sessmgmt.ExtendedReplicationUtil.getFailoverServerInstanceForBeKey(ExtendedReplicationUtil.java:109)
      at
      com.sun.enterprise.ee.web.sessmgmt.JxtaReplicationSender.sendReplicationState(JxtaReplicationSender.java:396)
      at
      com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.doTransmit(JxtaBackingStoreImpl.java:825)
      at
      com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.saveSimple(JxtaBackingStoreImpl.java:317)
      at
      com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.save(JxtaBackingStoreImpl.java:263)
      at
      com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.doValveSave(ReplicationStore.java:193)
      at
      com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.valveSave(ReplicationStore.java:144)
      at
      com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.load(ReplicationStore.java:649)
      at
      org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1088)
      at
      com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.doSwapIn(ReplicationManagerBase.java:1900)
      at
      org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1054)
      at
      com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.findSession(ReplicationManagerBase.java:3055)
      at
      org.apache.coyote.tomcat5.CoyoteRequest.doGetSession(CoyoteRequest.java:2828)
      at
      org.apache.coyote.tomcat5.CoyoteRequest.getSession(CoyoteRequest.java:2620)
      at
      org.apache.coyote.tomcat5.CoyoteRequestFacade.getSession(CoyoteRequestFacade.java:912)
      at
      org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:217)
      at
      org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
      at
      org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
      at
      org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
      at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
      at
      org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
      at
      org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
      at
      org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
      at
      org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
      at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
      at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
      at
      com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

      #]

        Issue Links

          Activity

          Hide
          varunrupela added a comment -

          added keyword and dependency

          Show
          varunrupela added a comment - added keyword and dependency
          Hide
          Scott Oaks added a comment -
              • Issue 1653 has been marked as a duplicate of this issue. ***
          Show
          Scott Oaks added a comment - Issue 1653 has been marked as a duplicate of this issue. ***
          Hide
          lwhite added a comment -

          sending patch jar to Scott/Steve for testing.

          Show
          lwhite added a comment - sending patch jar to Scott/Steve for testing.
          Hide
          varunrupela added a comment -

          Seeing this with build 13. The first exception appears 60 seconds after
          restarting the failed instance. I think that is when the first request will land
          on it as the load-factor kicks in. The exception continues for about 8 minutes
          and then stops. There are 1676 occurances in the restarted instance.

          The stack trace is a bit different with build 13. Here it is:

          [#|2009-05-08T12:18:52.227+0530|SEVERE|sun-glassfish-comms-server1.5|org.apache.coyote.tomcat5.CoyoteAdapter|_ThreadID=29;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=5d18d435-c8d6-4cf4-a09f-f7d5585b792f;|PWC3989:
          An exception or error occurred in the container during the request processing
          java.lang.IllegalArgumentException: Null key provided to lookup failover server
          instance
          at
          org.jvnet.glassfish.comms.clb.core.sip.DataCentricUtilImpl.getFailoverServerInstance(DataCentricUtilImpl.java:161)
          at
          com.ericsson.ssa.sip.SipApplicationSessionUtil.getFailoverServerInstanceForBeKey(SipApplicationSessionUtil.java:108)
          at
          org.jvnet.glassfish.comms.replication.sessmgmt.ExtendedReplicationUtil.getFailoverServerInstanceForBeKey(ExtendedReplicationUtil.java:121)
          at
          com.sun.enterprise.ee.web.sessmgmt.JxtaReplicationSender.sendReplicationState(JxtaReplicationSender.java:394)
          at
          com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.doTransmit(JxtaBackingStoreImpl.java:825)
          at
          com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.saveSimple(JxtaBackingStoreImpl.java:317)
          at
          com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.save(JxtaBackingStoreImpl.java:263)
          at
          com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.doValveSave(ReplicationStore.java:193)
          at
          com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.valveSave(ReplicationStore.java:144)
          at
          com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.load(ReplicationStore.java:649)
          at
          org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1088)
          at
          com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.doSwapIn(ReplicationManagerBase.java:1900)
          at
          org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1054)
          at
          com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.findSession(ReplicationManagerBase.java:3062)
          at
          org.apache.coyote.tomcat5.CoyoteRequest.doGetSession(CoyoteRequest.java:2828)
          at
          org.apache.coyote.tomcat5.CoyoteRequest.getSession(CoyoteRequest.java:2620)
          at
          org.apache.coyote.tomcat5.CoyoteRequestFacade.getSession(CoyoteRequestFacade.java:912)
          at
          org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:217)
          at
          org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
          at
          org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
          at
          org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
          at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
          at
          org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
          at
          org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
          at
          org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
          at
          org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
          at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)
          at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:290)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:871)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
          at
          com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
          at
          org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532)
          at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
          at
          com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

          #]
          Show
          varunrupela added a comment - Seeing this with build 13. The first exception appears 60 seconds after restarting the failed instance. I think that is when the first request will land on it as the load-factor kicks in. The exception continues for about 8 minutes and then stops. There are 1676 occurances in the restarted instance. The stack trace is a bit different with build 13. Here it is: [#|2009-05-08T12:18:52.227+0530|SEVERE|sun-glassfish-comms-server1.5|org.apache.coyote.tomcat5.CoyoteAdapter|_ThreadID=29;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=5d18d435-c8d6-4cf4-a09f-f7d5585b792f;|PWC3989: An exception or error occurred in the container during the request processing java.lang.IllegalArgumentException: Null key provided to lookup failover server instance at org.jvnet.glassfish.comms.clb.core.sip.DataCentricUtilImpl.getFailoverServerInstance(DataCentricUtilImpl.java:161) at com.ericsson.ssa.sip.SipApplicationSessionUtil.getFailoverServerInstanceForBeKey(SipApplicationSessionUtil.java:108) at org.jvnet.glassfish.comms.replication.sessmgmt.ExtendedReplicationUtil.getFailoverServerInstanceForBeKey(ExtendedReplicationUtil.java:121) at com.sun.enterprise.ee.web.sessmgmt.JxtaReplicationSender.sendReplicationState(JxtaReplicationSender.java:394) at com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.doTransmit(JxtaBackingStoreImpl.java:825) at com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.saveSimple(JxtaBackingStoreImpl.java:317) at com.sun.enterprise.ee.web.sessmgmt.JxtaBackingStoreImpl.save(JxtaBackingStoreImpl.java:263) at com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.doValveSave(ReplicationStore.java:193) at com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.valveSave(ReplicationStore.java:144) at com.sun.enterprise.ee.web.sessmgmt.ReplicationStore.load(ReplicationStore.java:649) at org.apache.catalina.session.PersistentManagerBase.doSwapIn(PersistentManagerBase.java:1088) at com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.doSwapIn(ReplicationManagerBase.java:1900) at org.apache.catalina.session.PersistentManagerBase.swapIn(PersistentManagerBase.java:1054) at com.sun.enterprise.ee.web.sessmgmt.ReplicationManagerBase.findSession(ReplicationManagerBase.java:3062) at org.apache.coyote.tomcat5.CoyoteRequest.doGetSession(CoyoteRequest.java:2828) at org.apache.coyote.tomcat5.CoyoteRequest.getSession(CoyoteRequest.java:2620) at org.apache.coyote.tomcat5.CoyoteRequestFacade.getSession(CoyoteRequestFacade.java:912) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:217) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:290) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:871) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:532) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) #]
          Hide
          lwhite added a comment -

          Fix checked in. Converted from use of session parent sasId
          to direct use of beKey of the session.

          Checking in JxtaReplicationSender.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationSender.java,v
          <-- JxtaReplicationSender.java
          new revision: 1.18.2.45; previous revision: 1.18.2.44
          done
          Checking in ReplicationAttributeStore.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationAttributeStore.java,v
          <-- ReplicationAttributeStore.java
          new revision: 1.9.2.13; previous revision: 1.9.2.12
          done
          Checking in ReplicationManagerBase.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationManagerBase.java,v
          <-- ReplicationManagerBase.java
          new revision: 1.31.2.70; previous revision: 1.31.2.69
          done
          Checking in ReplicationStore.java;
          /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationStore.java,v
          <-- ReplicationStore.java
          new revision: 1.20.2.28; previous revision: 1.20.2.27
          done

          Show
          lwhite added a comment - Fix checked in. Converted from use of session parent sasId to direct use of beKey of the session. Checking in JxtaReplicationSender.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/JxtaReplicationSender.java,v <-- JxtaReplicationSender.java new revision: 1.18.2.45; previous revision: 1.18.2.44 done Checking in ReplicationAttributeStore.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationAttributeStore.java,v <-- ReplicationAttributeStore.java new revision: 1.9.2.13; previous revision: 1.9.2.12 done Checking in ReplicationManagerBase.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationManagerBase.java,v <-- ReplicationManagerBase.java new revision: 1.31.2.70; previous revision: 1.31.2.69 done Checking in ReplicationStore.java; /cvs/glassfish/appserv-core-ee/http-session-persistence/src/java/com/sun/enterprise/ee/web/sessmgmt/ReplicationStore.java,v <-- ReplicationStore.java new revision: 1.20.2.28; previous revision: 1.20.2.27 done
          Hide
          Scott Oaks added a comment -
              • Issue 1765 has been marked as a duplicate of this issue. ***
          Show
          Scott Oaks added a comment - Issue 1765 has been marked as a duplicate of this issue. ***

            People

            • Assignee:
              Scott Oaks
              Reporter:
              varunrupela
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: