Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 2.0
    • Fix Version/s: milestone 1
    • Component/s: load_balancer
    • Labels:
      None
    • Environment:

      Operating System: Linux
      Platform: All

    • Issuezilla Id:
      2,015

      Description

      EFFECT:
      jvm hangs and clb throws exceptions when running XCAP/HTTP traffic. Running PGM
      Traffic mix at 1200 TPS on three PL's ( ~12 Mbit/s XCAP/HTTP ) any of the PL's
      hang after 10 minutes - 5 1/2 hours.
      When the jvm hangs the other payloads stops handling most traffic so almost no
      traffic is handled.

      DESCRIPTION:
      Sooner or later when running XCAP/HTTP traffic to the cluster any of the
      PayLoad jvm's hang. When this occur below is logged in server.log:
      [#|2009-10-07T15:34:06.398+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=66;_ThreadName=httpSSL
      WorkerThread-8080-4;_RequestID=30f41ca5-0d56-459c-8a7a-
      1e8003231c44;|Remote_termination
      java.lang.NullPointerException
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermina
      tion(LoadBalancerProxyFinder.java:154)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find
      (LoadBalancerProxyFinder.java:112)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask
      (ClbProxyPipeline.java:459)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run
      (TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run
      (SSLWorkerThread.java:106)

      #]

      [#|2009-10-07T15:34:06.411+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=67;_ThreadName=httpSSL
      WorkerThread-8080-2;_RequestID=71bc4976-dd1f-4a43-86e9-
      037649a20e30;|Remote_termination
      java.lang.NullPointerException
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermina
      tion(LoadBalancerProxyFinder.java:154)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find
      (LoadBalancerProxyFinder.java:112)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask
      (ClbProxyPipeline.java:459)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run
      (TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run
      (SSLWorkerThread.java:106)

      #]

      [#|2009-10-07T15:34:06.421+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=70;_ThreadName=httpSSL
      WorkerThread-8080-1;_RequestID=85edc547-480e-4d95-9b46-
      0e7c50f991a7;|Remote_termination
      java.lang.NullPointerException
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermina
      tion(LoadBalancerProxyFinder.java:154)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find
      (LoadBalancerProxyFinder.java:112)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask
      (ClbProxyPipeline.java:459)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run
      (TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run
      (SSLWorkerThread.java:106)

      #]

      [#|2009-10-07T15:34:06.421+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=69;_ThreadName=httpSSL
      WorkerThread-8080-0;_RequestID=bf69ed15-6ce7-4c59-953f-
      61b0a3071e9a;|Remote_termination
      java.lang.NullPointerException
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermina
      tion(LoadBalancerProxyFinder.java:154)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find
      (LoadBalancerProxyFinder.java:112)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask
      (ClbProxyPipeline.java:459)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run
      (TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run
      (SSLWorkerThread.java:106)

      #]

      [#|2009-10-07T15:34:06.412+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=68;_ThreadName=httpSSL
      WorkerThread-8080-3;_RequestID=425b468a-cd29-4f16-b4bb-
      517adacef8ec;|Remote_termination
      java.lang.NullPointerException
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermina
      tion(LoadBalancerProxyFinder.java:154)
      at
      org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find
      (LoadBalancerProxyFinder.java:112)
      at
      org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask
      (ClbProxyPipeline.java:459)
      at com.sun.enterprise.web.connector.grizzly.TaskBase.run
      (TaskBase.java:265)
      at
      com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run
      (SSLWorkerThread.java:106)

      #]

      [#|2009-10-07T15:34:07.921+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=70;_ThreadName=httpSSL
      WorkerThread-8080-1;_RequestID=85edc547-480e-4d95-9b46-0e7c50f991a7;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.921+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=70;_ThreadName=httpSSL
      WorkerThread-8080-1;_RequestID=85edc547-480e-4d95-9b46-0e7c50f991a7;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.921+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=70;_ThreadName=httpSSL
      WorkerThread-8080-1;_RequestID=85edc547-480e-4d95-9b46-0e7c50f991a7;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:07.922+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=66;_ThreadName=httpSSL
      WorkerThread-8080-4;_RequestID=30f41ca5-0d56-459c-8a7a-1e8003231c44;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.922+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=67;_ThreadName=httpSSL
      WorkerThread-8080-2;_RequestID=71bc4976-dd1f-4a43-86e9-037649a20e30;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.922+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=66;_ThreadName=httpSSL
      WorkerThread-8080-4;_RequestID=30f41ca5-0d56-459c-8a7a-1e8003231c44;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:07.923+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=68;_ThreadName=httpSSL
      WorkerThread-8080-3;_RequestID=425b468a-cd29-4f16-b4bb-517adacef8ec;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.923+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=68;_ThreadName=httpSSL
      WorkerThread-8080-3;_RequestID=425b468a-cd29-4f16-b4bb-517adacef8ec;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:07.923+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=69;_ThreadName=httpSSL
      WorkerThread-8080-0;_RequestID=bf69ed15-6ce7-4c59-953f-61b0a3071e9a;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.924+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=69;_ThreadName=httpSSL
      WorkerThread-8080-0;_RequestID=bf69ed15-6ce7-4c59-953f-61b0a3071e9a;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:07.923+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=67;_ThreadName=httpSSL
      WorkerThread-8080-2;_RequestID=71bc4976-dd1f-4a43-86e9-037649a20e30;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:07.925+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=67;_ThreadName=httpSSL
      WorkerThread-8080-2;_RequestID=71bc4976-dd1f-4a43-86e9-037649a20e30;|Fatal
      Error in doProxyHttp, task is null, terminating task.|#]

      [#|2009-10-07T15:34:07.925+0200|SEVERE|sun-glassfish-comms-
      server2.0|javax.enterprise.system.container.clb|_ThreadID=67;_ThreadName=httpSSL
      WorkerThread-8080-2;_RequestID=71bc4976-dd1f-4a43-86e9-037649a20e30;|Failed
      to release the request handler to pool.|#]

      [#|2009-10-07T15:34:35.931+0200|INFO|sun-glassfish-comms-
      server2.0|javax.enterprise.system.stream.out|_ThreadID=30;_ThreadName=pool-4-
      thread-14;|
      Mem usage: 9|#]

      When this occur, loads of sockets end up in CLOSE_WAIT on this payload ( could
      be any PL ). These sockets are both to the TTCN loadgenerator but also to other
      payloads. The receive queues are not emptied on this payload. This leads to
      send queues on the other payloads ( my speculation ), causing these payloads
      handle traffic badly ( I suppose only traffic ending on these PL's gets handled
      ok ).

      Stopping the traffic generator will not solve the situation. All sockets stay
      on CLOSE_WAIT at least 16 hours, probably forever. The easiest way to solve the
      situation is to restart the jvm with /opt/ericsson/payload stop.
      The other PayLoads

      MEASURES:
      I've tried to run this traffic with CLB loglevel set to FINE, but since the a
      new 2 GB server.log file is created every second at this traffic rate it is
      extremely difficult to handle.

      Attached is a compressed tar file containing the followig files:
      ./AppConfig.xml
      ./domain.xml
      ./jvmsettings_091008.txt
      ./pl_2_4_clb_error_091007.log This is the server log containing exceptions.
      ,/thread_dump.txt Generated with kill -QUIT <jvm_pid>
      ./traffic_lb_config.xml.v660 Current version of DCR file

      INSTALLED SW:
      Sailfin: sailfin-v2-b20

      SC_2_1: swm -p | sort
      APP ADMINSTARTPKG-CXP9013410_1-R11A01
      APP CAPS_XDMS-CXP9011509-DEV
      APP CONTENT_XDMS-CXP9014103-DEV
      APP DIR_XDMS-CXP9012628-DEV
      APP GROUP_XDMS-CXP9012414-DEV
      APP HARDSTATE_XDMS-CXP9012537-DEV
      APP JAVA_CAF_x86_64-CXP9013050_2-R7A02
      APP MMASPKG-CXP9013481_1-R11A01
      APP OAMSTARTPKG-CXP9013823_1-R11A01
      APP PGM-MYSQL-CONNECTOR
      APP PGM-MYSQL-DATANODE_R4A
      APP PGM-MYSQL-MANAGEMENT_R3A
      APP PGM-MYSQL-SQLNODE_R3A
      APP PGM_ACTIVEUSERS-CXP9020003-DEV
      APP PGM_PS-CXP9011382-DEV
      APP PGM_RLS-CXP9011385-DEV
      APP PGM_SHARED_SERVICES-CXP901TODO-DEV
      APP PNA_GMPC_CLIENT-CXP9013549-DEV
      APP PNA_SERVICE-ABC123-DEV
      APP PRESENCE_XDMS-CXP9011388-DEV
      APP PXD_Inventory-CXP90126785-DEV
      APP RLS_XDMS-CXP9011391-DEV
      APP SCASPKG-CXP9013518_1-R11A01
      APP SHARED_GROUP_XDMS-CXP9014155-DEV
      APP SHARED_POLICY_XDMS-CXP9014158-DEV
      APP SHARED_XDMS-CXP9011394-DEV
      APP TRAFFICSTARTPKG-CXP9013824_1-R11A01
      APP XDM_INVENTORY-CXP90126795-DEV
      APP XDM_SERVICE_API-CXP9011772-DEV
      CONFIG CONFIGPKG-CXP9013822_1-R11A01
      CONFIG CONFIG_CAPS_XDMS-CXP9011509-DEV
      CONFIG CONFIG_CONTENT_XDMS-CXP9014103-DEV
      CONFIG CONFIG_DIR_XDMS-CXP9012628-DEV
      CONFIG CONFIG_GROUP_XDMS-CXP9012414-DEV
      CONFIG CONFIG_HARDSTATE_XDMS-CXP9012537-DEV
      CONFIG CONFIG_PGM_PS-CXP9011382-DEV
      CONFIG CONFIG_PGM_RLS-CXP9011385-DEV
      CONFIG CONFIG_PNA_GMPC_CLIENT-CXP9013549-DEV
      CONFIG CONFIG_PRESENCE_XDMS-CXP9011388-DEV
      CONFIG CONFIG_RLS_XDMS-CXP9011391-DEV
      CONFIG CONFIG_SHARED_GROUP_XDMS-CXP9014155-DEV
      CONFIG CONFIG_SHARED_POLICY_XDMS-CXP9014158-DEV
      CONFIG CONFIG_SHARED_XDMS-CXP9011394-DEV
      CONFIG SAFCFG_x86_64-CXP9013711_2-R10B01
      MW OPENSAF_PL_x86_64-CXP9013044_2-R10B01
      MW OPENSAF_SC_x86_64-CXP9013043_2-R10B01
      MW SAF_OAM_x86_64-CXP9013625_2-R10C01
      MW SAF_SWM_x86_64-CXP9013626_2-R10C01
      MW TSPSAF_AF_x86_64-CXP9013045_2-R10C01
      MW VIP_x86_64-CXP9013048_2-R8A01
      OS LINUX_CONTROL-CXP9013151_1-R2J02
      OS LINUX_PAYLOAD-CXP9013152_1-R2J02

        Activity

        Hide
        lmcqfan added a comment -

        Created an attachment (id=1134)
        Associated log files...

        Show
        lmcqfan added a comment - Created an attachment (id=1134) Associated log files...
        Hide
        rampsarathy added a comment -

        I see the following configuration for traffic-config

        <request-processing header-buffer-length-in-bytes="8192"
        initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5"
        thread-increment="1"/>

        Could you please set

        asadmin set traffic.http-service.keep-alive.max-connections=-1
        asadmin set traffic.http-service.request-processing.initial-thread-count=20
        asadmin set traffic.http-service.request-processing.thread-count=20

        Also, increasing the "request-pool-size" (under clb->proxy) to 100 might help

        Show
        rampsarathy added a comment - I see the following configuration for traffic-config <request-processing header-buffer-length-in-bytes="8192" initial-thread-count="2" request-timeout-in-seconds="30" thread-count="5" thread-increment="1"/> Could you please set asadmin set traffic.http-service.keep-alive.max-connections=-1 asadmin set traffic.http-service.request-processing.initial-thread-count=20 asadmin set traffic.http-service.request-processing.thread-count=20 Also, increasing the "request-pool-size" (under clb->proxy) to 100 might help
        Hide
        rampsarathy added a comment -

        The NPE should be fixed, but applying the tuning should improve the performance
        and user will not hit the NPE path,
        Downgrading this,

        Show
        rampsarathy added a comment - The NPE should be fixed, but applying the tuning should improve the performance and user will not hit the NPE path, Downgrading this,
        Hide
        jesoncoobo added a comment -

        Hi Ramesh,

        We have optimized the configuration as you told.
        However, we are suffering this problem again, even with the optimized configuration.

        We got 2SC+2PL deployment, traffic is not much, but the result is similar. Many http sockets got stucked in CLOSE_WAIT on both PLs.

        load generator ip address: 10.6.0.151
        local ip address: 10.6.0.150

        netstat:

        PL-2-3: 135 connections stayed on CLOSE_WAIT
        tcp 1 0 10.6.0.150:8080 10.6.0.151:49377 CLOSE_WAIT

        PL-2-4: 58 connections stayed on CLOSE_WAIT
        tcp 923 0 10.6.0.150:8080 10.6.0.151:49520 CLOSE_WAIT

        336 CLB connections from PL-2-3 to PL-2-4
        tcp 922 0 192.168.0.4:8080 192.168.0.3:50844 ESTABLISHED

        No CLB connection from PL-2-4 to PL-2-3

        So looks like much more traffic are handled only on PL-2-4. I don't know the reason yet, but I will dig into it.

        server.log:

        PL-2-3:
        [#|2012-03-22T16:58:20.409+0100|SEVERE|javax.enterprise.system.container.clb|_ThreadID=59;_ThreadName=http-proxy-outboundWorkerThread-8080-8;_RequestID=789d5dd6-b08c-43bf-925b-f414b2cc3c7e
        Client channel closed.|#]

        PL-2-4:
        [#|2012-03-22T16:57:33.713+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=74;_ThreadName=http-proxy-outboundWorkerThread-8080-8;_RequestID=4b0451e0-8890-4729-bd90-72a672f261a0;|Client channel closed.|#]

        [#|2012-03-22T16:58:20.659+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=84;_ThreadName=httpSSLWorkerThread-8080-11;_RequestID=f06d42af-90ac-4bfc-b2a0-36ffad53a91f;|Failed to release the request handler to pool.|#]

        [#|2012-03-22T16:58:20.667+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=65;_ThreadName=http-proxy-outboundWorkerThread-8080-3;_RequestID=a0b13599-8d68-4594-854d-e1571e346c58;|Client channel closed.|#]

        [#|2012-03-22T16:58:50.665+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Remote_termination
        java.lang.NullPointerException
        at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:154)
        at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:112)
        at org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:471)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269)
        at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111)

        #]
        [#|2012-03-22T16:58:50.666+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Fatal Error in doProxyHttp, task is null, terminating task.|#]
        [#|2012-03-22T16:58:50.666+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Failed to release the request handler to pool.|#]

        The server.log in PL-2-3 is pretty clear except a client close.
        But there are lots of exceptions on PL-2-4.
        It looks like we were running out of ProxyRequestHandler which is configured as 100 max.

        Then I found the places where returing ProxyRequestHandler.
        objManager.offerTask(task, protocolInfo.isSecure);

        in LoadBalancerProxyHandler, remove clientkey and return task as:
        HttpProxy.getInstance().getConnectionManager().removeClientEndpoint(task.getSelectionKey());
        task.recycle();
        objManager.offerTask(task, protocolInfo.isSecure);

        However, in other places which calls removeClientEndpoint() without returning task.

        public void removeClientEndpoint(SelectionKey key) {
        if (key != null)

        { clientToproxy.remove(key); }

        }

        Could this cause these tasks gone?
        Should we remove client key and meanwhile check if task is not null and return task?

        Correct me if I am wrong.

        Thanks
        Brs

        Show
        jesoncoobo added a comment - Hi Ramesh, We have optimized the configuration as you told. However, we are suffering this problem again, even with the optimized configuration. We got 2SC+2PL deployment, traffic is not much, but the result is similar. Many http sockets got stucked in CLOSE_WAIT on both PLs. load generator ip address: 10.6.0.151 local ip address: 10.6.0.150 netstat: PL-2-3: 135 connections stayed on CLOSE_WAIT tcp 1 0 10.6.0.150:8080 10.6.0.151:49377 CLOSE_WAIT PL-2-4: 58 connections stayed on CLOSE_WAIT tcp 923 0 10.6.0.150:8080 10.6.0.151:49520 CLOSE_WAIT 336 CLB connections from PL-2-3 to PL-2-4 tcp 922 0 192.168.0.4:8080 192.168.0.3:50844 ESTABLISHED No CLB connection from PL-2-4 to PL-2-3 So looks like much more traffic are handled only on PL-2-4. I don't know the reason yet, but I will dig into it. server.log: PL-2-3: [#|2012-03-22T16:58:20.409+0100|SEVERE|javax.enterprise.system.container.clb|_ThreadID=59;_ThreadName=http-proxy-outboundWorkerThread-8080-8;_RequestID=789d5dd6-b08c-43bf-925b-f414b2cc3c7e Client channel closed.|#] PL-2-4: [#|2012-03-22T16:57:33.713+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=74;_ThreadName=http-proxy-outboundWorkerThread-8080-8;_RequestID=4b0451e0-8890-4729-bd90-72a672f261a0;|Client channel closed.|#] [#|2012-03-22T16:58:20.659+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=84;_ThreadName=httpSSLWorkerThread-8080-11;_RequestID=f06d42af-90ac-4bfc-b2a0-36ffad53a91f;|Failed to release the request handler to pool.|#] [#|2012-03-22T16:58:20.667+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=65;_ThreadName=http-proxy-outboundWorkerThread-8080-3;_RequestID=a0b13599-8d68-4594-854d-e1571e346c58;|Client channel closed.|#] [#|2012-03-22T16:58:50.665+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Remote_termination java.lang.NullPointerException at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.isRemoteTermination(LoadBalancerProxyFinder.java:154) at org.jvnet.glassfish.comms.clb.proxy.http.LoadBalancerProxyFinder.find(LoadBalancerProxyFinder.java:112) at org.jvnet.glassfish.comms.clb.proxy.portunif.ClbProxyPipeline$PUTask.doTask(ClbProxyPipeline.java:471) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:269) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:111) #] [#|2012-03-22T16:58:50.666+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Fatal Error in doProxyHttp, task is null, terminating task.|#] [#|2012-03-22T16:58:50.666+0100|SEVERE|sun-glassfish-comms-server2.0|javax.enterprise.system.container.clb|_ThreadID=50;_ThreadName=httpSSLWorkerThread-8080-0;_RequestID=50d06be5-a355-45e0-ba74-54fd1e29f8cb;|Failed to release the request handler to pool.|#] The server.log in PL-2-3 is pretty clear except a client close. But there are lots of exceptions on PL-2-4. It looks like we were running out of ProxyRequestHandler which is configured as 100 max. Then I found the places where returing ProxyRequestHandler. objManager.offerTask(task, protocolInfo.isSecure); in LoadBalancerProxyHandler, remove clientkey and return task as: HttpProxy.getInstance().getConnectionManager().removeClientEndpoint(task.getSelectionKey()); task.recycle(); objManager.offerTask(task, protocolInfo.isSecure); However, in other places which calls removeClientEndpoint() without returning task. public void removeClientEndpoint(SelectionKey key) { if (key != null) { clientToproxy.remove(key); } } Could this cause these tasks gone? Should we remove client key and meanwhile check if task is not null and return task? Correct me if I am wrong. Thanks Brs

          People

          • Assignee:
            rampsarathy
            Reporter:
            lmcqfan
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: