glassfish
  1. glassfish
  2. GLASSFISH-18311

Periodical error on Glassfish after a small load and running for a few hours : java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1_b12
    • Fix Version/s: 4.0
    • Component/s: grizzly-kernel
    • Labels:
      None
    • Environment:

      Operating System : SunOS push002ppd 5.10 Generic_142909-17 sun4v sparc SUNW,T5140

      JVM user : Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) Server VM (build 20.5-b03, mixed mode)

      Description

      Periodical error on Glassfish after a small load and running for a few hours. No one is connected to the server anymore and the error appear a few hours after a stress test.

      The questions are :

      • how is it possible that the such an error appear when there is no more load ?
      • since no requests are made, what is filling the pool ?
      • can we see which request fill in the pool ? can we activate any logs ?

      Thanks.

      The error :

      [#|2012-02-02T19:43:24.375+0100|SEVERE|glassfish3.1.1|grizzly|_ThreadID=119;_ThreadName=Thread-2;|doSelect exception
      java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
              at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
              at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
              at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
              at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
              at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
              at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
              at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
              at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
              at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:263)
              at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
              at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      |#]
      

      Glassfish connector config :

      [#|2012-02-02T19:43:26.844+0100|INFO|glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=133;_ThreadName=Thread-2;|GRIZZLY0021: "
      Grizzly running on SunOS - 5.10 under JDK version: 1.6.0_30 - Sun Microsystems Inc.
      port: 8280
      Thread Pool: com.sun.enterprise.v3.services.impl.monitor.MonitorableThreadPool@4b9772e2, port=8280
      Read Selector: 3
      ByteBuffer size: 4096
      maxHttpHeaderSize: 4096
      sendBufferSize: 8192
      maxKeepAliveRequests: 10,000
      keepAliveTimeoutInSeconds: 120
      Static File Cache enabled: false
      Static resources directory: /opt/server/glassfish3/glassfish/domains/comet/docroot
      Adapter : com.sun.enterprise.v3.services.impl.ContainerMapper
      Asynchronous Request Processing enabled: true|#]

      1. domain.xml
        31 kB
        guillaume.d
      2. gf-reject-2997.dump
        1.16 MB
        guillaume.d
      3. jstack_queue_full
        1.29 MB
        guillaume.d

        Activity

        Hide
        sirio7g added a comment -

        Did the patch provided on Oct, 19 2012 work?

        Show
        sirio7g added a comment - Did the patch provided on Oct, 19 2012 work?
        Hide
        oleksiys added a comment -

        @sirio7g are you using JDK6?

        You may want to apply the patch, looks like it's a bug in LinkedTransferQueue.

        Show
        oleksiys added a comment - @sirio7g are you using JDK6? You may want to apply the patch, looks like it's a bug in LinkedTransferQueue.
        Hide
        sirio7g added a comment -

        Os:
        Linux pd022 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

        JVM:
        java version "1.7.0_25"
        Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
        Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

        Show
        sirio7g added a comment - Os: Linux pd022 2.6.32-358.11.1.el6.x86_64 #1 SMP Wed Jun 12 03:34:52 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux JVM: java version "1.7.0_25" Java(TM) SE Runtime Environment (build 1.7.0_25-b15) Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
        Hide
        oleksiys added a comment -

        the latest grizzly-http.jar has to fix this problem.

        Show
        oleksiys added a comment - the latest grizzly-http.jar has to fix this problem.
        Hide
        oleksiys added a comment -

        correspondent Grizzly issue [1] has been fixed.
        the patch for 3.1.2.2 is available in the attachment.

        [1] https://java.net/jira/browse/GRIZZLY-1581

        Show
        oleksiys added a comment - correspondent Grizzly issue [1] has been fixed. the patch for 3.1.2.2 is available in the attachment. [1] https://java.net/jira/browse/GRIZZLY-1581

          People

          • Assignee:
            oleksiys
            Reporter:
            guillaume.d
          • Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: