sailfin
  1. sailfin
  2. SAILFIN-597

Null Pointer Exception in Concurrent Queue

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0
    • Fix Version/s: milestone 1
    • Component/s: load_balancer
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: Sun

    • Issuezilla Id:
      597

      Description

      Running longevity testing using a web application called Richaccess on Sailfin
      promoted B21. During the loading of the users, seeing lots of these NPE's in
      instance logs which is causing lots of client requests to fail.

      [#|2008-02-21T17:14:14.720-0800|WARNING|sun-comms-appserver1.0|javax.enterprise.system.stream.err|_ThreadID=21;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=892bdb2c-86cb-42f6-ab6c-b23078c2b426;|
      java.lang.NullPointerException
      at
      com.sun.grizzly.connectioncache.impl.concurrent.ConcurrentQueueImpl.poll(ConcurrentQueueImpl.java:135)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.util.ObjectManager.pollTask(ObjectManager.java:70)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:100)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:83)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:454)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

      #]

        Activity

        Hide
        jluehe added a comment -

        hi ramesh, can you please take a look or reassign as appropriate? thanks, jan

        Show
        jluehe added a comment - hi ramesh, can you please take a look or reassign as appropriate? thanks, jan
        Hide
        meenap added a comment -

        I am also seeing the following Exceptions.

        The following IO Exception is seen after about 30 mins into the run and it has
        happened once on the 1st instance for the past 24 hours.

        [#|2008-02-21T17:25:16.891-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.web|_ThreadID=27;_ThreadName=SelectorThread-38080;_RequestID=a9032998-a124-4193-bae6-18c460b3b8db;|WEB0756:
        Caught exception during HTTP processing.
        java.io.IOException: Too many open files
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at
        sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAccept(SelectorThread.java:1460)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConnection(SelectorThread.java:1439)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1350)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247)
        at
        com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)

        #]

        [#|2008-02-21T17:25:16.943-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=21;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=892bdb2c-86cb-42f6-ab6c-b23078c2b426;|Fatal
        Error in doProxyHttp, task is null, terminating task.|#]

        [#|2008-02-21T17:25:16.943-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=21;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=892bdb2c-86cb-42f6-ab6c-b23078c2b426;|Failed
        to release the request handler to pool.|#]

        This EOF Exceptions from clb containter has happened 4 times on the 1st
        instance. This is seen after 5 mins into the run

        [#|2008-02-21T17:09:52.800-0800|INFO|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=22;_ThreadName=httpSSLWorkerThread-38080-2;|Endpoint
        is null and status is 200, local.|#]

        [#|2008-02-21T17:09:52.803-0800|WARNING|sun-comms-appserver1.0|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=8cb661a1-18ca-4101-af59-6b649b1c56ae;|java.io.EOFException:
        EOF_error
        at
        org.jvnet.glassfish.comms.clb.proxy.http.util.HttpInputBuffer.parseRequestLine(HttpInputBuffer.java:259)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.layers.ParserLayer.invoke(ParserLayer.java:61)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyLayerHandler.doInvoke(ProxyLayerHandler.java:74)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.process(ProxyRequestHandler.java:266)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.doTask(ProxyRequestHandler.java:237)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:109)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:83)
        at
        org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:454)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at
        com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

        #]

        [#|2008-02-21T17:09:52.813-0800|WARNING|sun-comms-appserver1.0|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=8cb661a1-18ca-4101-af59-6b649b1c56ae;|
        java.io.EOFException: EOF_error
        at
        org.jvnet.glassfish.comms.clb.proxy.http.util.HttpInputBuffer.parseRequestLine(HttpInputBuffer.java:259)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.layers.ParserLayer.invoke(ParserLayer.java:61)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyLayerHandler.doInvoke(ProxyLayerHandler.java:74)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.process(ProxyRequestHandler.java:266)
        at
        org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.doTask(ProxyRequestHandler.java:237)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:109)
        at
        org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:83)
        at
        org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:454)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
        at
        com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

        #]

        [#|2008-02-21T17:09:52.813-0800|INFO|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;|Endpoint
        is null, and status is not 200, error response.|#]

        Show
        meenap added a comment - I am also seeing the following Exceptions. The following IO Exception is seen after about 30 mins into the run and it has happened once on the 1st instance for the past 24 hours. [#|2008-02-21T17:25:16.891-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.web|_ThreadID=27;_ThreadName=SelectorThread-38080;_RequestID=a9032998-a124-4193-bae6-18c460b3b8db;|WEB0756: Caught exception during HTTP processing. java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleAccept(SelectorThread.java:1460) at com.sun.enterprise.web.connector.grizzly.SelectorThread.handleConnection(SelectorThread.java:1439) at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1350) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247) at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223) #] [#|2008-02-21T17:25:16.943-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=21;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=892bdb2c-86cb-42f6-ab6c-b23078c2b426;|Fatal Error in doProxyHttp, task is null, terminating task.|#] [#|2008-02-21T17:25:16.943-0800|SEVERE|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=21;_ThreadName=httpSSLWorkerThread-38080-1;_RequestID=892bdb2c-86cb-42f6-ab6c-b23078c2b426;|Failed to release the request handler to pool.|#] This EOF Exceptions from clb containter has happened 4 times on the 1st instance. This is seen after 5 mins into the run [#|2008-02-21T17:09:52.800-0800|INFO|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=22;_ThreadName=httpSSLWorkerThread-38080-2;|Endpoint is null and status is 200, local.|#] [#|2008-02-21T17:09:52.803-0800|WARNING|sun-comms-appserver1.0|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=8cb661a1-18ca-4101-af59-6b649b1c56ae;|java.io.EOFException: EOF_error at org.jvnet.glassfish.comms.clb.proxy.http.util.HttpInputBuffer.parseRequestLine(HttpInputBuffer.java:259) at org.jvnet.glassfish.comms.clb.proxy.http.layers.ParserLayer.invoke(ParserLayer.java:61) at org.jvnet.glassfish.comms.clb.proxy.ProxyLayerHandler.doInvoke(ProxyLayerHandler.java:74) at org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.process(ProxyRequestHandler.java:266) at org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.doTask(ProxyRequestHandler.java:237) at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:109) at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:83) at org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:454) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) #] [#|2008-02-21T17:09:52.813-0800|WARNING|sun-comms-appserver1.0|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;_RequestID=8cb661a1-18ca-4101-af59-6b649b1c56ae;| java.io.EOFException: EOF_error at org.jvnet.glassfish.comms.clb.proxy.http.util.HttpInputBuffer.parseRequestLine(HttpInputBuffer.java:259) at org.jvnet.glassfish.comms.clb.proxy.http.layers.ParserLayer.invoke(ParserLayer.java:61) at org.jvnet.glassfish.comms.clb.proxy.ProxyLayerHandler.doInvoke(ProxyLayerHandler.java:74) at org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.process(ProxyRequestHandler.java:266) at org.jvnet.glassfish.comms.clb.proxy.ProxyRequestHandler.doTask(ProxyRequestHandler.java:237) at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:109) at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:83) at org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:454) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) #] [#|2008-02-21T17:09:52.813-0800|INFO|sun-comms-appserver1.0|javax.enterprise.system.container.clb|_ThreadID=20;_ThreadName=httpSSLWorkerThread-38080-0;|Endpoint is null, and status is not 200, error response.|#]
        Hide
        meenap added a comment -

        Changing the subcomponent to clb.

        Show
        meenap added a comment - Changing the subcomponent to clb.
        Hide
        rampsarathy added a comment -

        this would be retested in b23 by the submitter

        Show
        rampsarathy added a comment - this would be retested in b23 by the submitter
        Hide
        rampsarathy added a comment -

        The NPE was because the ConcurrentQueueImpl from grizzly was not thread safe, so
        modified ObjectManager to use ArrayBlockingQueue.
        Fixed in b24, wednesday (mar 5) nightly

        Checking in http/util/ObjectManager.java;
        /cvs/sailfin/clb/src/main/java/org/jvnet/glassfish/comms/clb/proxy/http/util/ObjectManager.java,v
        <-- ObjectManager.java
        new revision: 1.10; previous revision: 1.9

        Show
        rampsarathy added a comment - The NPE was because the ConcurrentQueueImpl from grizzly was not thread safe, so modified ObjectManager to use ArrayBlockingQueue. Fixed in b24, wednesday (mar 5) nightly Checking in http/util/ObjectManager.java; /cvs/sailfin/clb/src/main/java/org/jvnet/glassfish/comms/clb/proxy/http/util/ObjectManager.java,v <-- ObjectManager.java new revision: 1.10; previous revision: 1.9
        Hide
        meenap added a comment -

        Ran the test on B24a and this issue is not reproducible anymore. Marking this
        bug as verified.

        Show
        meenap added a comment - Ran the test on B24a and this issue is not reproducible anymore. Marking this bug as verified.

          People

          • Assignee:
            rampsarathy
            Reporter:
            meenap
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: