glassfish
  1. glassfish
  2. GLASSFISH-18311

Periodical error on Glassfish after a small load and running for a few hours : java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1_b12
    • Fix Version/s: 4.0
    • Component/s: grizzly-kernel
    • Labels:
      None
    • Environment:

      Operating System : SunOS push002ppd 5.10 Generic_142909-17 sun4v sparc SUNW,T5140

      JVM user : Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) Server VM (build 20.5-b03, mixed mode)

      Description

      Periodical error on Glassfish after a small load and running for a few hours. No one is connected to the server anymore and the error appear a few hours after a stress test.

      The questions are :

      • how is it possible that the such an error appear when there is no more load ?
      • since no requests are made, what is filling the pool ?
      • can we see which request fill in the pool ? can we activate any logs ?

      Thanks.

      The error :

      [#|2012-02-02T19:43:24.375+0100|SEVERE|glassfish3.1.1|grizzly|_ThreadID=119;_ThreadName=Thread-2;|doSelect exception
      java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
              at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
              at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
              at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
              at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
              at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
              at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
              at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
              at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
              at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
              at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
              at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      |#]
      

      Glassfish connector config :

      [#|2012-02-02T19:43:26.844+0100|INFO|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=133;_ThreadName=Thread-2;|GRIZZLY0021: "
      Grizzly running on SunOS - 5.10 under JDK version: 1.6.0_30 - Sun Microsystems Inc.
      port: 8280
      Thread Pool: com.sun.enterprise.v3.services.impl.monitor.MonitorableThreadPool@4b9772e2, port=8280
      Read Selector: 3
      ByteBuffer size: 4096
      maxHttpHeaderSize: 4096
      sendBufferSize: 8192
      maxKeepAliveRequests: 10,000
      keepAliveTimeoutInSeconds: 120
      Static File Cache enabled: false
      Static resources directory: /opt/server/glassfish3/glassfish/domains/comet/docroot
      Adapter : com.sun.enterprise.v3.services.impl.ContainerMapper
      Asynchronous Request Processing enabled: true|#]

      1. domain.xml
        31 kB
        guillaume.d
      2. gf-reject-2997.dump
        1.16 MB
        guillaume.d
      3. jstack_queue_full
        1.29 MB
        guillaume.d

        Activity

        Hide
        oleksiys added a comment -

        You can easily figure out what is blocking threads by making threads dump.

        1) run "jps" to find out PID of Glassfish.
        2) run "jstack <PID>" to get threads dump.

        Once you have a dump - pls. attach it here.

        Show
        oleksiys added a comment - You can easily figure out what is blocking threads by making threads dump. 1) run "jps" to find out PID of Glassfish. 2) run "jstack <PID>" to get threads dump. Once you have a dump - pls. attach it here.
        Hide
        scatari added a comment -

        We are not seeing this issue on 3.1.2, hence reducing the priority to P3. We will still evaluate this bug as we wait for data from submitter to proceed.

        Show
        scatari added a comment - We are not seeing this issue on 3.1.2, hence reducing the priority to P3. We will still evaluate this bug as we wait for data from submitter to proceed.
        Hide
        guillaume.d added a comment -

        Unfortunately the server has been restarted and the error disappeared. I did not keep the thread dump I made from Visual VM but I did not see any thread in BLOCKED state. I also tried to see the thread pool state in the JMX MBean panel but I could not find any usefull information like the current thread use count or the current queue size.

        One more precision : comet is enable on the listener.

        Do you recommend to put more trace in logs in order to diagnose the problem more quickly ? I have enable access logging in order to see which requests are made on the server.

        Show
        guillaume.d added a comment - Unfortunately the server has been restarted and the error disappeared. I did not keep the thread dump I made from Visual VM but I did not see any thread in BLOCKED state. I also tried to see the thread pool state in the JMX MBean panel but I could not find any usefull information like the current thread use count or the current queue size. One more precision : comet is enable on the listener. Do you recommend to put more trace in logs in order to diagnose the problem more quickly ? I have enable access logging in order to see which requests are made on the server.
        Hide
        guillaume.d added a comment -

        It seems that the same bug has been reported once GLASSFISH-15426.

        Why is it in "won't fix" state. Did you find any configuration and/or use of Glassfish that can make this problem appear ? Is there another workaround than increasing the queue size ?

        Show
        guillaume.d added a comment - It seems that the same bug has been reported once GLASSFISH-15426 . Why is it in "won't fix" state. Did you find any configuration and/or use of Glassfish that can make this problem appear ? Is there another workaround than increasing the queue size ?
        Hide
        oleksiys added a comment -

        Usually thread-pool overflow is caused by application (servlet) logic, some database connections get blocked or so.
        That's why we need thread dump to understand who occupies threads. If it would really help if you can provide one.

        Show
        oleksiys added a comment - Usually thread-pool overflow is caused by application (servlet) logic, some database connections get blocked or so. That's why we need thread dump to understand who occupies threads. If it would really help if you can provide one.
        Hide
        Joe Di Pol added a comment -

        Too late to consider for 3.1.2

        Show
        Joe Di Pol added a comment - Too late to consider for 3.1.2
        Hide
        guillaume.d added a comment -

        Problem reproduced. jstack attached. The port that doesn't listent is 8280 : http-thread-pool-client-8280*.

        All the thread are available and the error is still here.

        Perhaps a counter problem on the queue size ??

        Show
        guillaume.d added a comment - Problem reproduced. jstack attached. The port that doesn't listent is 8280 : http-thread-pool-client-8280*. All the thread are available and the error is still here. Perhaps a counter problem on the queue size ??
        Hide
        guillaume.d added a comment -

        Is there a way to see the value of the current queue size ?

        Show
        guillaume.d added a comment - Is there a way to see the value of the current queue size ?
        Hide
        guillaume.d added a comment - - edited

        The weird thing in the thread dump is there are 2 thread pools for the same port (see thread-pool-client(n)) appear 2 times.

        The pool is configured with 100 actives thread but there are 200 threads. How is it possible ?

        Show
        guillaume.d added a comment - - edited The weird thing in the thread dump is there are 2 thread pools for the same port (see thread-pool-client(n)) appear 2 times. The pool is configured with 100 actives thread but there are 200 threads. How is it possible ?
        Hide
        guillaume.d added a comment -

        This is a serious problem, please consider my request.

        Thanks.

        Show
        guillaume.d added a comment - This is a serious problem, please consider my request. Thanks.
        Hide
        oleksiys added a comment -

        you're right thread dump looks clean (not sure why you have thread duplications though).
        to monitor threads and task queue, you have to enable monitoring.

        1) In the admin gui go to Configurations->server-config->Monitoring
        Set Web Container, Thread Pool and Http Service monitoring to HIGH
        2) Restart GF

        To check the busy threads number run this:
        asadmin get -m server.network.http-listener-1.thread-pool.currentthreadsbusy-count

        To check the queue size:
        asadmin get -m server.network.http-listener-1.connection-queue.countqueued-count

        • replace http-listener-1 w/ the name you use for your listener
        Show
        oleksiys added a comment - you're right thread dump looks clean (not sure why you have thread duplications though). to monitor threads and task queue, you have to enable monitoring. 1) In the admin gui go to Configurations->server-config->Monitoring Set Web Container, Thread Pool and Http Service monitoring to HIGH 2) Restart GF To check the busy threads number run this: asadmin get -m server.network.http-listener-1.thread-pool.currentthreadsbusy-count To check the queue size: asadmin get -m server.network.http-listener-1.connection-queue.countqueued-count replace http-listener-1 w/ the name you use for your listener
        Hide
        guillaume.d added a comment -

        In another JIRA, you suggest to put the queue size to -1 (unlimited). I will try that since all my threads are availabled.

        Do you think this can bypass the problem ?

        Can we upgrade to Glassfish 3.1.2 ?

        Show
        guillaume.d added a comment - In another JIRA, you suggest to put the queue size to -1 (unlimited). I will try that since all my threads are availabled. Do you think this can bypass the problem ? Can we upgrade to Glassfish 3.1.2 ?
        Hide
        oleksiys added a comment -

        well, you can, but it may just delay the problem. The issue is pretty weird, I doubt it's a bug in thread-pool implementation, because it would have been discovered earlier during stress testing.

        I see lots of threads like "NonStopCache [sessions] Executor Thread-204 for 'http-thread-pool-client-8280(61)'"

        can you try to disable NonStopCache and repeat the test?

        Show
        oleksiys added a comment - well, you can, but it may just delay the problem. The issue is pretty weird, I doubt it's a bug in thread-pool implementation, because it would have been discovered earlier during stress testing. I see lots of threads like "NonStopCache [sessions] Executor Thread-204 for 'http-thread-pool-client-8280(61)'" can you try to disable NonStopCache and repeat the test?
        Hide
        guillaume.d added a comment -

        I'll try to disable it but it is a Terracotta feature that should not cause any problem.

        I have 6 servers and the problem appear not on heavy load and only on some of them. Last time this problem occured it was 2 hours after a restart. A simple reconfiguration of the thread pool without restart solve the problem. The most weird thing is the double http thread pool that was not present on the others servers (but the others servers have NonStopCache).

        My Glassfish 3.1 is patched with 2 jars from GLASSFISH-17150 and GRIZZLY-1034. These 2 patches are include in GF 3.1.2 right ? Maybe I can try with Glassfish 3.1.2.

        Show
        guillaume.d added a comment - I'll try to disable it but it is a Terracotta feature that should not cause any problem. I have 6 servers and the problem appear not on heavy load and only on some of them. Last time this problem occured it was 2 hours after a restart. A simple reconfiguration of the thread pool without restart solve the problem. The most weird thing is the double http thread pool that was not present on the others servers (but the others servers have NonStopCache). My Glassfish 3.1 is patched with 2 jars from GLASSFISH-17150 and GRIZZLY-1034 . These 2 patches are include in GF 3.1.2 right ? Maybe I can try with Glassfish 3.1.2.
        Hide
        oleksiys added a comment -

        Sure, you can try 3.1.2,
        can you pls. also attach domain.xml?

        Thanks.

        Show
        oleksiys added a comment - Sure, you can try 3.1.2, can you pls. also attach domain.xml? Thanks.
        Hide
        guillaume.d added a comment - - edited

        domain.xml attached.

        I can't reproduce the problem, it should something not weel initialized at startup that cause this random problem.

        I also put some node with -1 queue size and others with 4096 to check what if the problem appear also with a -1 queue size.

        Show
        guillaume.d added a comment - - edited domain.xml attached. I can't reproduce the problem, it should something not weel initialized at startup that cause this random problem. I also put some node with -1 queue size and others with 4096 to check what if the problem appear also with a -1 queue size.
        Hide
        oleksiys added a comment -

        thread-pool configuration in the domain.xml looks fine.

        if you have time, can you pls. try to reproduce the issue, on a thread-pool with following config:

        • max-queue-size = 4096
        • min-thread-pool-size = 8
        • max-thread-pool-size = 100

        so the only diff. in that min-thread-pool-size is 8 instead of 100.

        Thanks.

        Show
        oleksiys added a comment - thread-pool configuration in the domain.xml looks fine. if you have time, can you pls. try to reproduce the issue, on a thread-pool with following config: max-queue-size = 4096 min-thread-pool-size = 8 max-thread-pool-size = 100 so the only diff. in that min-thread-pool-size is 8 instead of 100. Thanks.
        Hide
        guillaume.d added a comment -

        Reproduced again. No load except a load balancer that do a ping to check if the server is still here.

        Result of asadmin command

        server.network.http-listener-client.thread-pool.corethreads-count = 0
        server.network.http-listener-client.thread-pool.corethreads-description = Core number of threads in the thread pool
        server.network.http-listener-client.thread-pool.corethreads-lastsampletime = -1
        server.network.http-listener-client.thread-pool.corethreads-name = CoreThreads
        server.network.http-listener-client.thread-pool.corethreads-starttime = 1331232242528
        server.network.http-listener-client.thread-pool.corethreads-unit = count
        server.network.http-listener-client.thread-pool.currentthreadcount-count = 200
        server.network.http-listener-client.thread-pool.currentthreadcount-description = Provides the number of request processing threads currently in the listener thread pool
        server.network.http-listener-client.thread-pool.currentthreadcount-lastsampletime = 1331232243802
        server.network.http-listener-client.thread-pool.currentthreadcount-name = CurrentThreadCount
        server.network.http-listener-client.thread-pool.currentthreadcount-starttime = 1331232242529
        server.network.http-listener-client.thread-pool.currentthreadcount-unit = count
        server.network.http-listener-client.thread-pool.currentthreadsbusy-count = 0
        server.network.http-listener-client.thread-pool.currentthreadsbusy-description = Provides the number of request processing threads currently in use in the listener thread pool serving requests
        server.network.http-listener-client.thread-pool.currentthreadsbusy-lastsampletime = 1331250783604
        server.network.http-listener-client.thread-pool.currentthreadsbusy-name = CurrentThreadsBusy
        server.network.http-listener-client.thread-pool.currentthreadsbusy-starttime = 1331232242529
        server.network.http-listener-client.thread-pool.currentthreadsbusy-unit = count
        server.network.http-listener-client.thread-pool.dotted-name = server.network.http-listener-client.thread-pool
        server.network.http-listener-client.thread-pool.maxthreads-count = 0
        server.network.http-listener-client.thread-pool.maxthreads-description = Maximum number of threads allowed in the thread pool
        server.network.http-listener-client.thread-pool.maxthreads-lastsampletime = -1
        server.network.http-listener-client.thread-pool.maxthreads-name = MaxThreads
        server.network.http-listener-client.thread-pool.maxthreads-starttime = 1331232242528
        server.network.http-listener-client.thread-pool.maxthreads-unit = count
        server.network.http-listener-client.thread-pool.totalexecutedtasks-count = 8097
        server.network.http-listener-client.thread-pool.totalexecutedtasks-description = Provides the total number of tasks, which were executed by the thread pool
        server.network.http-listener-client.thread-pool.totalexecutedtasks-lastsampletime = 1331250783604
        server.network.http-listener-client.thread-pool.totalexecutedtasks-name = TotalExecutedTasksCount
        server.network.http-listener-client.thread-pool.totalexecutedtasks-starttime = 1331232242529
        server.network.http-listener-client.thread-pool.totalexecutedtasks-unit = count
        
        Show
        guillaume.d added a comment - Reproduced again. No load except a load balancer that do a ping to check if the server is still here. Result of asadmin command server.network.http-listener-client.thread-pool.corethreads-count = 0 server.network.http-listener-client.thread-pool.corethreads-description = Core number of threads in the thread pool server.network.http-listener-client.thread-pool.corethreads-lastsampletime = -1 server.network.http-listener-client.thread-pool.corethreads-name = CoreThreads server.network.http-listener-client.thread-pool.corethreads-starttime = 1331232242528 server.network.http-listener-client.thread-pool.corethreads-unit = count server.network.http-listener-client.thread-pool.currentthreadcount-count = 200 server.network.http-listener-client.thread-pool.currentthreadcount-description = Provides the number of request processing threads currently in the listener thread pool server.network.http-listener-client.thread-pool.currentthreadcount-lastsampletime = 1331232243802 server.network.http-listener-client.thread-pool.currentthreadcount-name = CurrentThreadCount server.network.http-listener-client.thread-pool.currentthreadcount-starttime = 1331232242529 server.network.http-listener-client.thread-pool.currentthreadcount-unit = count server.network.http-listener-client.thread-pool.currentthreadsbusy-count = 0 server.network.http-listener-client.thread-pool.currentthreadsbusy-description = Provides the number of request processing threads currently in use in the listener thread pool serving requests server.network.http-listener-client.thread-pool.currentthreadsbusy-lastsampletime = 1331250783604 server.network.http-listener-client.thread-pool.currentthreadsbusy-name = CurrentThreadsBusy server.network.http-listener-client.thread-pool.currentthreadsbusy-starttime = 1331232242529 server.network.http-listener-client.thread-pool.currentthreadsbusy-unit = count server.network.http-listener-client.thread-pool.dotted-name = server.network.http-listener-client.thread-pool server.network.http-listener-client.thread-pool.maxthreads-count = 0 server.network.http-listener-client.thread-pool.maxthreads-description = Maximum number of threads allowed in the thread pool server.network.http-listener-client.thread-pool.maxthreads-lastsampletime = -1 server.network.http-listener-client.thread-pool.maxthreads-name = MaxThreads server.network.http-listener-client.thread-pool.maxthreads-starttime = 1331232242528 server.network.http-listener-client.thread-pool.maxthreads-unit = count server.network.http-listener-client.thread-pool.totalexecutedtasks-count = 8097 server.network.http-listener-client.thread-pool.totalexecutedtasks-description = Provides the total number of tasks, which were executed by the thread pool server.network.http-listener-client.thread-pool.totalexecutedtasks-lastsampletime = 1331250783604 server.network.http-listener-client.thread-pool.totalexecutedtasks-name = TotalExecutedTasksCount server.network.http-listener-client.thread-pool.totalexecutedtasks-starttime = 1331232242529 server.network.http-listener-client.thread-pool.totalexecutedtasks-unit = count
        Hide
        guillaume.d added a comment -

        Other stats :

        server.network.http-listener-client.connection-queue.countopenconnections-count = 0
        server.network.http-listener-client.connection-queue.countopenconnections-description = The number of open/active connections
        server.network.http-listener-client.connection-queue.countopenconnections-lastsampletime = 1331370087836
        server.network.http-listener-client.connection-queue.countopenconnections-name = CountOpenConnections
        server.network.http-listener-client.connection-queue.countopenconnections-starttime = 1331370087836
        server.network.http-listener-client.connection-queue.countopenconnections-unit = count
        server.network.http-listener-client.connection-queue.countoverflows-count = 52093
        server.network.http-listener-client.connection-queue.countoverflows-description = Number of times the queue has been too full to accommodate a connection
        server.network.http-listener-client.connection-queue.countoverflows-lastsampletime = 1331370087144
        server.network.http-listener-client.connection-queue.countoverflows-name = CountOverflows
        server.network.http-listener-client.connection-queue.countoverflows-starttime = 1331232242972
        server.network.http-listener-client.connection-queue.countoverflows-unit = count
        server.network.http-listener-client.connection-queue.countqueued-count = 0
        server.network.http-listener-client.connection-queue.countqueued-description = Number of connections currently in the queue
        server.network.http-listener-client.connection-queue.countqueued-lastsampletime = 1331250783599
        server.network.http-listener-client.connection-queue.countqueued-name = CountQueued
        server.network.http-listener-client.connection-queue.countqueued-starttime = 1331232242972
        server.network.http-listener-client.connection-queue.countqueued-unit = count
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-count = 0
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-description = Average number of connections queued in the last 15 minutes
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-lastsampletime = 1331370087838
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-name = CountQueued15MinutesAverage
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-starttime = 1331370087838
        server.network.http-listener-client.connection-queue.countqueued15minutesaverage-unit = count
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-count = 0
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-description = Average number of connections queued in the last 1 minute
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-lastsampletime = 1331370087839
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-name = CountQueued1MinuteAverage
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-starttime = 1331370087839
        server.network.http-listener-client.connection-queue.countqueued1minuteaverage-unit = count
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-count = 0
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-description = Average number of connections queued in the last 5 minutes
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-lastsampletime = 1331370087840
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-name = CountQueued5MinutesAverage
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-starttime = 1331370087839
        server.network.http-listener-client.connection-queue.countqueued5minutesaverage-unit = count
        server.network.http-listener-client.connection-queue.counttotalconnections-count = 60190
        server.network.http-listener-client.connection-queue.counttotalconnections-description = Total number of connections that have been accepted
        server.network.http-listener-client.connection-queue.counttotalconnections-lastsampletime = 1331370087143
        server.network.http-listener-client.connection-queue.counttotalconnections-name = CountTotalConnections
        server.network.http-listener-client.connection-queue.counttotalconnections-starttime = 1331232242971
        server.network.http-listener-client.connection-queue.counttotalconnections-unit = count
        server.network.http-listener-client.connection-queue.counttotalqueued-count = 8097
        server.network.http-listener-client.connection-queue.counttotalqueued-description = Total number of connections that have been queued
        server.network.http-listener-client.connection-queue.counttotalqueued-lastsampletime = 1331250783599
        server.network.http-listener-client.connection-queue.counttotalqueued-name = CountTotalQueued
        server.network.http-listener-client.connection-queue.counttotalqueued-starttime = 1331232242972
        server.network.http-listener-client.connection-queue.counttotalqueued-unit = count
        server.network.http-listener-client.connection-queue.dotted-name = server.network.http-listener-client.connection-queue
        server.network.http-listener-client.connection-queue.maxqueued-count = 0
        server.network.http-listener-client.connection-queue.maxqueued-description = Maximum size of the connection queue
        server.network.http-listener-client.connection-queue.maxqueued-lastsampletime = -1
        server.network.http-listener-client.connection-queue.maxqueued-name = MaxQueued
        server.network.http-listener-client.connection-queue.maxqueued-starttime = 1331232242972
        server.network.http-listener-client.connection-queue.maxqueued-unit = count
        server.network.http-listener-client.connection-queue.peakqueued-count = 2
        server.network.http-listener-client.connection-queue.peakqueued-description = Largest number of connections that were in the queue simultaneously
        server.network.http-listener-client.connection-queue.peakqueued-lastsampletime = 1331236068463
        server.network.http-listener-client.connection-queue.peakqueued-name = PeakQueued
        server.network.http-listener-client.connection-queue.peakqueued-starttime = 1331232242972
        server.network.http-listener-client.connection-queue.peakqueued-unit = count
        server.network.http-listener-client.connection-queue.tickstotalqueued-count = 0
        server.network.http-listener-client.connection-queue.tickstotalqueued-description = (Unsupported) Total number of ticks that connections have spent in the queue
        server.network.http-listener-client.connection-queue.tickstotalqueued-lastsampletime = 1331232243027
        server.network.http-listener-client.connection-queue.tickstotalqueued-name = TicksTotalQueued
        server.network.http-listener-client.connection-queue.tickstotalqueued-starttime = 1331232242973
        server.network.http-listener-client.connection-queue.tickstotalqueued-unit = count
        
        
        Show
        guillaume.d added a comment - Other stats : server.network.http-listener-client.connection-queue.countopenconnections-count = 0 server.network.http-listener-client.connection-queue.countopenconnections-description = The number of open/active connections server.network.http-listener-client.connection-queue.countopenconnections-lastsampletime = 1331370087836 server.network.http-listener-client.connection-queue.countopenconnections-name = CountOpenConnections server.network.http-listener-client.connection-queue.countopenconnections-starttime = 1331370087836 server.network.http-listener-client.connection-queue.countopenconnections-unit = count server.network.http-listener-client.connection-queue.countoverflows-count = 52093 server.network.http-listener-client.connection-queue.countoverflows-description = Number of times the queue has been too full to accommodate a connection server.network.http-listener-client.connection-queue.countoverflows-lastsampletime = 1331370087144 server.network.http-listener-client.connection-queue.countoverflows-name = CountOverflows server.network.http-listener-client.connection-queue.countoverflows-starttime = 1331232242972 server.network.http-listener-client.connection-queue.countoverflows-unit = count server.network.http-listener-client.connection-queue.countqueued-count = 0 server.network.http-listener-client.connection-queue.countqueued-description = Number of connections currently in the queue server.network.http-listener-client.connection-queue.countqueued-lastsampletime = 1331250783599 server.network.http-listener-client.connection-queue.countqueued-name = CountQueued server.network.http-listener-client.connection-queue.countqueued-starttime = 1331232242972 server.network.http-listener-client.connection-queue.countqueued-unit = count server.network.http-listener-client.connection-queue.countqueued15minutesaverage-count = 0 server.network.http-listener-client.connection-queue.countqueued15minutesaverage-description = Average number of connections queued in the last 15 minutes server.network.http-listener-client.connection-queue.countqueued15minutesaverage-lastsampletime = 1331370087838 server.network.http-listener-client.connection-queue.countqueued15minutesaverage-name = CountQueued15MinutesAverage server.network.http-listener-client.connection-queue.countqueued15minutesaverage-starttime = 1331370087838 server.network.http-listener-client.connection-queue.countqueued15minutesaverage-unit = count server.network.http-listener-client.connection-queue.countqueued1minuteaverage-count = 0 server.network.http-listener-client.connection-queue.countqueued1minuteaverage-description = Average number of connections queued in the last 1 minute server.network.http-listener-client.connection-queue.countqueued1minuteaverage-lastsampletime = 1331370087839 server.network.http-listener-client.connection-queue.countqueued1minuteaverage-name = CountQueued1MinuteAverage server.network.http-listener-client.connection-queue.countqueued1minuteaverage-starttime = 1331370087839 server.network.http-listener-client.connection-queue.countqueued1minuteaverage-unit = count server.network.http-listener-client.connection-queue.countqueued5minutesaverage-count = 0 server.network.http-listener-client.connection-queue.countqueued5minutesaverage-description = Average number of connections queued in the last 5 minutes server.network.http-listener-client.connection-queue.countqueued5minutesaverage-lastsampletime = 1331370087840 server.network.http-listener-client.connection-queue.countqueued5minutesaverage-name = CountQueued5MinutesAverage server.network.http-listener-client.connection-queue.countqueued5minutesaverage-starttime = 1331370087839 server.network.http-listener-client.connection-queue.countqueued5minutesaverage-unit = count server.network.http-listener-client.connection-queue.counttotalconnections-count = 60190 server.network.http-listener-client.connection-queue.counttotalconnections-description = Total number of connections that have been accepted server.network.http-listener-client.connection-queue.counttotalconnections-lastsampletime = 1331370087143 server.network.http-listener-client.connection-queue.counttotalconnections-name = CountTotalConnections server.network.http-listener-client.connection-queue.counttotalconnections-starttime = 1331232242971 server.network.http-listener-client.connection-queue.counttotalconnections-unit = count server.network.http-listener-client.connection-queue.counttotalqueued-count = 8097 server.network.http-listener-client.connection-queue.counttotalqueued-description = Total number of connections that have been queued server.network.http-listener-client.connection-queue.counttotalqueued-lastsampletime = 1331250783599 server.network.http-listener-client.connection-queue.counttotalqueued-name = CountTotalQueued server.network.http-listener-client.connection-queue.counttotalqueued-starttime = 1331232242972 server.network.http-listener-client.connection-queue.counttotalqueued-unit = count server.network.http-listener-client.connection-queue.dotted-name = server.network.http-listener-client.connection-queue server.network.http-listener-client.connection-queue.maxqueued-count = 0 server.network.http-listener-client.connection-queue.maxqueued-description = Maximum size of the connection queue server.network.http-listener-client.connection-queue.maxqueued-lastsampletime = -1 server.network.http-listener-client.connection-queue.maxqueued-name = MaxQueued server.network.http-listener-client.connection-queue.maxqueued-starttime = 1331232242972 server.network.http-listener-client.connection-queue.maxqueued-unit = count server.network.http-listener-client.connection-queue.peakqueued-count = 2 server.network.http-listener-client.connection-queue.peakqueued-description = Largest number of connections that were in the queue simultaneously server.network.http-listener-client.connection-queue.peakqueued-lastsampletime = 1331236068463 server.network.http-listener-client.connection-queue.peakqueued-name = PeakQueued server.network.http-listener-client.connection-queue.peakqueued-starttime = 1331232242972 server.network.http-listener-client.connection-queue.peakqueued-unit = count server.network.http-listener-client.connection-queue.tickstotalqueued-count = 0 server.network.http-listener-client.connection-queue.tickstotalqueued-description = (Unsupported) Total number of ticks that connections have spent in the queue server.network.http-listener-client.connection-queue.tickstotalqueued-lastsampletime = 1331232243027 server.network.http-listener-client.connection-queue.tickstotalqueued-name = TicksTotalQueued server.network.http-listener-client.connection-queue.tickstotalqueued-starttime = 1331232242973 server.network.http-listener-client.connection-queue.tickstotalqueued-unit = count
        Hide
        oleksiys added a comment -

        According to this

        server.network.http-listener-client.connection-queue.countoverflows-count = 52093
        server.network.http-listener-client.connection-queue.countoverflows-description = Number of times the queue has been too full to accommodate a connection

        we see that queue was really overflowed (52093 times)!
        Though, according the statistics you posted, the queue is empty now and ready to process requests.

        Show
        oleksiys added a comment - According to this server.network.http-listener-client.connection-queue.countoverflows-count = 52093 server.network.http-listener-client.connection-queue.countoverflows-description = Number of times the queue has been too full to accommodate a connection we see that queue was really overflowed (52093 times)! Though, according the statistics you posted, the queue is empty now and ready to process requests.
        Hide
        guillaume.d added a comment -

        So it is a bug ?

        Show
        guillaume.d added a comment - So it is a bug ?
        Hide
        oleksiys added a comment -

        Let's do this.
        I can send you a patch, which takes threads snapshot each time thread-pool is overflowed.

        You use Glassfish 3.1.1 release, right?

        Show
        oleksiys added a comment - Let's do this. I can send you a patch, which takes threads snapshot each time thread-pool is overflowed. You use Glassfish 3.1.1 release, right?
        Hide
        guillaume.d added a comment -

        Just to be clear, request were discarded when I took the statistics and were still discarded after.

        Show
        guillaume.d added a comment - Just to be clear, request were discarded when I took the statistics and were still discarded after.
        Hide
        oleksiys added a comment - - edited

        Can you pls. attach same stats for all listeners?

        thanks.

        Show
        oleksiys added a comment - - edited Can you pls. attach same stats for all listeners? thanks.
        Hide
        guillaume.d added a comment -

        I didn't keep the statistics of all the listener and the server has been restarted since.

        But I'm sure that client listener has the problem because :

        • all the request, from the browser were ok except on port 8280
        • after changing a setting on the thead pool of client listener everyhting gets back to normal
        • the probe that was failing and detect the problem is only on port 8280

        So the problem is on that thread pool.

        Can you give a patch for 3.1.1 and 3.1.2 web profile. I am juste migrating the application.

        Show
        guillaume.d added a comment - I didn't keep the statistics of all the listener and the server has been restarted since. But I'm sure that client listener has the problem because : all the request, from the browser were ok except on port 8280 after changing a setting on the thead pool of client listener everyhting gets back to normal the probe that was failing and detect the problem is only on port 8280 So the problem is on that thread pool. Can you give a patch for 3.1.1 and 3.1.2 web profile. I am juste migrating the application.
        Hide
        oleksiys added a comment -

        patch for 3.1.2

        Show
        oleksiys added a comment - patch for 3.1.2
        Hide
        oleksiys added a comment -

        attaching the patch for GF 3.1.2.
        Once task is getting refused - GF will create a threads dump file in the current folder (for me it's gfv3/glassfish/domains/domain1/config/).

        Show
        oleksiys added a comment - attaching the patch for GF 3.1.2. Once task is getting refused - GF will create a threads dump file in the current folder (for me it's gfv3/glassfish/domains/domain1/config/).
        Hide
        guillaume.d added a comment -

        There is still the bug in GF 3.1.2.

        It appears no longer after a start. Unfortunately, the bugs has been diagnose on the only server that does not have the patch.

        Witing for the next time is shows up.

        Show
        guillaume.d added a comment - There is still the bug in GF 3.1.2. It appears no longer after a start. Unfortunately, the bugs has been diagnose on the only server that does not have the patch. Witing for the next time is shows up.
        Hide
        oleksiys added a comment -

        any updates on the issue?

        thanks.

        Show
        oleksiys added a comment - any updates on the issue? thanks.
        Hide
        guillaume.d added a comment -

        Problem experience again right after a restart of the server.

        [#|2012-07-08T12:29:50.115+0200|SEVERE|glassfish3.1.2|grizzly|_ThreadID=74;_ThreadName=Thread-2;|doSelect exception
        java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
                at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
                at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
                at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
                at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
                at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:134)
                at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
                at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
                at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
                at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
                at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
                at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
        |#]
        

        I have many thead dumps (that filled the whole file system). I have attached one.

        Show
        guillaume.d added a comment - Problem experience again right after a restart of the server. [#|2012-07-08T12:29:50.115+0200|SEVERE|glassfish3.1.2|grizzly|_ThreadID=74;_ThreadName=Thread-2;|doSelect exception java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096 at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473) at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97) at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88) at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162) at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:134) at com.sun.grizzly.NIOContext.execute(NIOContext.java:510) at com.sun.grizzly.NIOContext.execute(NIOContext.java:488) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263) at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200) at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) |#] I have many thead dumps (that filled the whole file system). I have attached one.
        Hide
        oleksiys added a comment -

        If you used the patch (attached here), can you pls. send server.log* file(s)?
        Thanks.

        Show
        oleksiys added a comment - If you used the patch (attached here), can you pls. send server.log* file(s)? Thanks.
        Hide
        guillaume.d added a comment -

        Here are all the server.log files

        Show
        guillaume.d added a comment - Here are all the server.log files
        Hide
        oleksiys added a comment -

        looks like thread-pool for port 8281 and 8248 haven't been created, at least I don't see them in the gf-reject*.dump. So most probably requests to these ports were queued up but never processed... this finally caused RejectedExecutionException.

        You didn't change thread-pool config, so they are still configured (in the domain.xml) like:
        <thread-pools>
        <thread-pool name="admin-thread-pool" max-thread-pool-size="50" max-queue-size="256"/>
        <thread-pool name="http-thread-pool-casino" max-thread-pool-size="100" min-thread-pool-size="100"/>
        <thread-pool name="http-thread-pool-client" max-thread-pool-size="100" max-queue-size="-1" min-thread-pool-size="100"/><thread-pool name="http-thread-pool"/>
        </thread-pools>

        ?

        Show
        oleksiys added a comment - looks like thread-pool for port 8281 and 8248 haven't been created, at least I don't see them in the gf-reject*.dump. So most probably requests to these ports were queued up but never processed... this finally caused RejectedExecutionException. You didn't change thread-pool config, so they are still configured (in the domain.xml) like: <thread-pools> <thread-pool name="admin-thread-pool" max-thread-pool-size="50" max-queue-size="256"/> <thread-pool name="http-thread-pool-casino" max-thread-pool-size="100" min-thread-pool-size="100"/> <thread-pool name="http-thread-pool-client" max-thread-pool-size="100" max-queue-size="-1" min-thread-pool-size="100"/><thread-pool name="http-thread-pool"/> </thread-pools> ?
        Hide
        oleksiys added a comment -

        imo you're right, something is not getting started properly (during GF startup).
        Here is another patch, which should give us more information on thread pools and their threads even if they have not been started.
        It will produce gf-reject-.dump and gf-reject--extra.dump.

        Will appreciate if you can try it.

        Show
        oleksiys added a comment - imo you're right, something is not getting started properly (during GF startup). Here is another patch, which should give us more information on thread pools and their threads even if they have not been started. It will produce gf-reject- .dump and gf-reject- -extra.dump. Will appreciate if you can try it.
        Hide
        guillaume.d added a comment -

        Ok we will try it asap.

        Thanks.

        Show
        guillaume.d added a comment - Ok we will try it asap. Thanks.
        Hide
        guillaume.d added a comment -

        We reproduced the problem wery often when the load is increasing.

        Here are the logs generated by your patch. Can you have a look at those file ?

        Thanks.

        Show
        guillaume.d added a comment - We reproduced the problem wery often when the load is increasing. Here are the logs generated by your patch. Can you have a look at those file ? Thanks.
        Hide
        oleksiys added a comment -

        Very interesting...
        Looks like the counter is correct, the real thread-pool queue size is 4096...
        May be it's a bug in LinkedTransferQueue implementation, let's try to substitute it w/ LinkedBlockingQueue.

        I'm attaching grizzly-utils.jar, which has to override grizzly-utils.jar from the patch I attached earlier.

        Pls. send me the logs, if you see the problem again.

        Thanks.

        Show
        oleksiys added a comment - Very interesting... Looks like the counter is correct, the real thread-pool queue size is 4096... May be it's a bug in LinkedTransferQueue implementation, let's try to substitute it w/ LinkedBlockingQueue. I'm attaching grizzly-utils.jar, which has to override grizzly-utils.jar from the patch I attached earlier. Pls. send me the logs, if you see the problem again. Thanks.
        Hide
        yavuzs added a comment -

        hi all,
        we're facing the same problem with GlassFish Server Open Source Edition 3.1.2 (build 23)
        guillaume.d, is the problem fixed with grizzly-utils.jar that oleksiys attached?
        what is the last status for this issue?
        regards,

        Show
        yavuzs added a comment - hi all, we're facing the same problem with GlassFish Server Open Source Edition 3.1.2 (build 23) guillaume.d, is the problem fixed with grizzly-utils.jar that oleksiys attached? what is the last status for this issue? regards,
        Hide
        ggates1020 added a comment -

        We are also still seeing this issue after upgrading to GlassFish Server Open Source Edition 3.1.2.2 (build 5). It is happening on a large customer production site and is becoming painful.

        Show
        ggates1020 added a comment - We are also still seeing this issue after upgrading to GlassFish Server Open Source Edition 3.1.2.2 (build 5). It is happening on a large customer production site and is becoming painful.
        Hide
        oleksiys added a comment -

        hi guys,

        when you see the issue - pls. provide more info on OS you're running, attach the server.log, domain.xml files and jstack report, taken when the problem happens, so we can doublecheck if it's really the same issue.

        Thanks.

        Show
        oleksiys added a comment - hi guys, when you see the issue - pls. provide more info on OS you're running, attach the server.log, domain.xml files and jstack report, taken when the problem happens, so we can doublecheck if it's really the same issue. Thanks.
        Hide
        kpasgma added a comment -

        Hi we have just run into this same issue. From the logs it only happened on "ThreadID=11".
        We are running on a Ubuntu, Glassfish 3.1.2 (build 23).
        Being that this error looks like it is occurring in grizzly, is this a grizzly issue or Glassfish?
        Restarting the server did fix the issue, but what is the cause?

        [#|2013-03-23T12:57:18.730+0800|SEVERE|glassfish3.1.2|grizzly|_ThreadID=11;_ThreadName=Thread-2;|doSelect exception
        java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
        at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
        at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
        at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
        at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
        at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
        at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
        at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

        Show
        kpasgma added a comment - Hi we have just run into this same issue. From the logs it only happened on "ThreadID=11". We are running on a Ubuntu, Glassfish 3.1.2 (build 23). Being that this error looks like it is occurring in grizzly, is this a grizzly issue or Glassfish? Restarting the server did fix the issue, but what is the cause? [#|2013-03-23T12:57:18.730+0800|SEVERE|glassfish3.1.2|grizzly|_ThreadID=11;_ThreadName=Thread-2;|doSelect exception java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096 at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473) at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97) at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88) at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162) at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127) at com.sun.grizzly.NIOContext.execute(NIOContext.java:510) at com.sun.grizzly.NIOContext.execute(NIOContext.java:488) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263) at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200) at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
        Hide
        oleksiys added a comment -

        I'm not sure it's either Grizzly or Glassfish issue...
        Might be just configuration or application problem.

        This issue happens when thread-pool queue is overloaded, default max thread-pool queue size is 4096 elements (tasks).
        First of all you can try to extend the max queue size:

        asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-queue-size=<NNN>
        

        also when it happens - pls. take a threads dump using

        jstack <GLASSFISH_PID>
        

        because may be all the threads are stuck working on some tasks and new tasks are getting queued but never processed.

        Thanks.

        Show
        oleksiys added a comment - I'm not sure it's either Grizzly or Glassfish issue... Might be just configuration or application problem. This issue happens when thread-pool queue is overloaded, default max thread-pool queue size is 4096 elements (tasks). First of all you can try to extend the max queue size: asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-queue-size=<NNN> also when it happens - pls. take a threads dump using jstack <GLASSFISH_PID> because may be all the threads are stuck working on some tasks and new tasks are getting queued but never processed. Thanks.
        Hide
        pbelbin added a comment - - edited

        I too am seeing this.

        I'm using jdk 7 u 17 x64 on solaris 10 x64

        glassfish is 3.1.2.2 (build 5)

        I see in server.log:

        [#|2013-06-20T15:47:21.062+0000|SEVERE|glassfish3.1.2|grizzly|_ThreadID=34;_ThreadName=Thread-2;|doSelect exception
        java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
        at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
        at com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191)
        at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
        at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
        at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
        at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
        at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
        at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)

        this is a production system, and would really appreciate some help with this.

        my attached jstack <glassfish pid> output is attached as: pbelbin_stack_2013-06-20a.gz

        I take it back - I can't attach to this issue, it seems.

        I may be able to create a sub-issue and attach there.

        nope! sorry, I have a jstack output file to upload, but am unable to attach it. please advise.

        I even created a sub-task to see if I could attach there, but no, it won't let me.

        Show
        pbelbin added a comment - - edited I too am seeing this. I'm using jdk 7 u 17 x64 on solaris 10 x64 glassfish is 3.1.2.2 (build 5) I see in server.log: [#|2013-06-20T15:47:21.062+0000|SEVERE|glassfish3.1.2|grizzly|_ThreadID=34;_ThreadName=Thread-2;|doSelect exception java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096 at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473) at com.sun.grizzly.util.SyncThreadPool.execute(SyncThreadPool.java:191) at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162) at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127) at com.sun.grizzly.NIOContext.execute(NIOContext.java:510) at com.sun.grizzly.NIOContext.execute(NIOContext.java:488) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370) at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263) at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200) at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) this is a production system, and would really appreciate some help with this. my attached jstack <glassfish pid> output is attached as: pbelbin_stack_2013-06-20a.gz I take it back - I can't attach to this issue, it seems. I may be able to create a sub-issue and attach there. nope! sorry, I have a jstack output file to upload, but am unable to attach it. please advise. I even created a sub-task to see if I could attach there, but no, it won't let me.
        Hide
        oleksiys added a comment -

        I moved GLASSFISH-20652 to a separate issue, because it's different.
        In GLASSFISH-20652 all the Glassfish threads are blocked inside user's application code, so it looks like application (not Glassfish issue)

        Show
        oleksiys added a comment - I moved GLASSFISH-20652 to a separate issue, because it's different. In GLASSFISH-20652 all the Glassfish threads are blocked inside user's application code, so it looks like application (not Glassfish issue)
        Hide
        sirio7g added a comment -

        Did the patch provided on Oct, 19 2012 work?

        Show
        sirio7g added a comment - Did the patch provided on Oct, 19 2012 work?
        Hide
        oleksiys added a comment -

        @sirio7g are you using JDK6?

        You may want to apply the patch, looks like it's a bug in LinkedTransferQueue.

        Show
        oleksiys added a comment - @sirio7g are you using JDK6? You may want to apply the patch, looks like it's a bug in LinkedTransferQueue.
        Hide
        sirio7g added a comment -

        Os:
        Linux pd022 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

        JVM:
        java version "1.7.0_25"
        Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
        Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

        Show
        sirio7g added a comment - Os: Linux pd022 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux JVM: java version "1.7.0_25" Java(TM) SE Runtime Environment (build 1.7.0_25-b15) Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
        Hide
        oleksiys added a comment -

        the latest grizzly-http.jar has to fix this problem.

        Show
        oleksiys added a comment - the latest grizzly-http.jar has to fix this problem.
        Hide
        oleksiys added a comment -

        correspondent Grizzly issue [1] has been fixed.
        the patch for 3.1.2.2 is available in the attachment.

        [1] https://java.net/jira/browse/GRIZZLY-1581

        Show
        oleksiys added a comment - correspondent Grizzly issue [1] has been fixed. the patch for 3.1.2.2 is available in the attachment. [1] https://java.net/jira/browse/GRIZZLY-1581

          People

          • Assignee:
            oleksiys
            Reporter:
            guillaume.d
          • Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: