glassfish
  1. glassfish
  2. GLASSFISH-15426

[STRESS] "java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full" message appears in the logs, 4 days into the run. Seems to cause the instance to crash.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: 3.1_b35
    • Fix Version/s: None
    • Component/s: grizzly-kernel
    • Labels:
      None

      Description

      Please see the parent issue 15425 for the details of the scenario that caused this bug.

      The following exception appears in one of the instance's logs before the instance crashes:

      [#|2011-01-02T22:15:52.483+0530|SEVERE|glassfish3.1|grizzly|_ThreadID=22;_ThreadName=Thread-3;|doSelect exception
      java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
      at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
      at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
      at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
      at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
      at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
      at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
      at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
      at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
      at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
      at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)
      at java.lang.Thread.run(Thread.java:662)

      #]

      The complete server log for that instance is attached.

        Issue Links

          Activity

          Hide
          oleksiys added a comment -

          This behavior is expected, we need to find out why GF container(s) block Grizzly thread-pool.
          For now I can not do anything about this issue.

          Show
          oleksiys added a comment - This behavior is expected, we need to find out why GF container(s) block Grizzly thread-pool. For now I can not do anything about this issue.
          Hide
          oleksiys added a comment -

          Posting what I wrote in the email conversation.

          that exception just mean that thread-pool task queue reached its maximum (4096 tasks in a queue).
          This usually means that thread-pool is not able to process tasks that fast as they're getting added. Or most probably thread-pool's threads are either blocked with some tasks, or task processing takes too long.

          you can workaround this "bug" by setting higher limit of thread-pool max queue size (which is 4096 by default).

          in the domain.xml
          <thread-pool max-queue-size="<NN>" ....>

          Setting <NN> to -1 - will create unlimited thread-pool queue.

          This should solve the RejectedExecutionException, but in some other place we may find OutOfMemoryError.

          Anyway, don't think this is Grizzly bug, it could be good if someone can check which modules/tasks block thread-pool's threads and fix that.

          Show
          oleksiys added a comment - Posting what I wrote in the email conversation. that exception just mean that thread-pool task queue reached its maximum (4096 tasks in a queue). This usually means that thread-pool is not able to process tasks that fast as they're getting added. Or most probably thread-pool's threads are either blocked with some tasks, or task processing takes too long. you can workaround this "bug" by setting higher limit of thread-pool max queue size (which is 4096 by default). in the domain.xml <thread-pool max-queue-size="<NN>" ....> Setting <NN> to -1 - will create unlimited thread-pool queue. This should solve the RejectedExecutionException, but in some other place we may find OutOfMemoryError. Anyway, don't think this is Grizzly bug, it could be good if someone can check which modules/tasks block thread-pool's threads and fix that.
          Hide
          varunrupela added a comment -

          We have started a re-run of this scenario with the latest build to collect more information on why thread pool task queue reaches its maximum.

          Show
          varunrupela added a comment - We have started a re-run of this scenario with the latest build to collect more information on why thread pool task queue reaches its maximum.
          Hide
          varunrupela added a comment -

          The instances seem to crash quite some time after this issue appears in the log. The debug run has also provided indications that this bug is s symptom. We need to figure out why grizzly threads get blocked.

          For details see http://java.net/jira/browse/GLASSFISH-15425

          Show
          varunrupela added a comment - The instances seem to crash quite some time after this issue appears in the log. The debug run has also provided indications that this bug is s symptom. We need to figure out why grizzly threads get blocked. For details see http://java.net/jira/browse/GLASSFISH-15425

            People

            • Assignee:
              oleksiys
              Reporter:
              varunrupela
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: