[GLASSFISH-15426] [STRESS] "java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full" message appears in the logs, 4 days into the run. Seems to cause the instance to crash. Created: 04/Jan/11  Updated: 10/Jan/11  Resolved: 10/Jan/11

Status: Closed
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1_b35
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: varunrupela Assignee: oleksiys
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File server.log    
Issue Links:
blocks GLASSFISH-15425 [STRESS][umbrella] 24x7 RichAccess ru... Open


Please see the parent issue 15425 for the details of the scenario that caused this bug.

The following exception appears in one of the instance's logs before the instance crashes:

[#|2011-01-02T22:15:52.483+0530|SEVERE|glassfish3.1|grizzly|_ThreadID=22;_ThreadName=Thread-3;|doSelect exception
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
at com.sun.grizzly.util.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:473)
at com.sun.grizzly.util.QueueLimitedThreadPool.onTaskQueueOverflow(QueueLimitedThreadPool.java:97)
at com.sun.grizzly.util.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:88)
at com.sun.grizzly.util.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:162)
at com.sun.grizzly.http.StatsThreadPool.execute(StatsThreadPool.java:127)
at com.sun.grizzly.NIOContext.execute(NIOContext.java:510)
at com.sun.grizzly.NIOContext.execute(NIOContext.java:488)
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:370)
at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:200)
at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:909)
at java.lang.Thread.run(Thread.java:662)


The complete server log for that instance is attached.

Comment by oleksiys [ 04/Jan/11 ]

This behavior is expected, we need to find out why GF container(s) block Grizzly thread-pool.
For now I can not do anything about this issue.

Comment by oleksiys [ 04/Jan/11 ]

Posting what I wrote in the email conversation.

that exception just mean that thread-pool task queue reached its maximum (4096 tasks in a queue).
This usually means that thread-pool is not able to process tasks that fast as they're getting added. Or most probably thread-pool's threads are either blocked with some tasks, or task processing takes too long.

you can workaround this "bug" by setting higher limit of thread-pool max queue size (which is 4096 by default).

in the domain.xml
<thread-pool max-queue-size="<NN>" ....>

Setting <NN> to -1 - will create unlimited thread-pool queue.

This should solve the RejectedExecutionException, but in some other place we may find OutOfMemoryError.

Anyway, don't think this is Grizzly bug, it could be good if someone can check which modules/tasks block thread-pool's threads and fix that.

Comment by varunrupela [ 05/Jan/11 ]

We have started a re-run of this scenario with the latest build to collect more information on why thread pool task queue reaches its maximum.

Comment by varunrupela [ 10/Jan/11 ]

The instances seem to crash quite some time after this issue appears in the log. The debug run has also provided indications that this bug is s symptom. We need to figure out why grizzly threads get blocked.

For details see http://java.net/jira/browse/GLASSFISH-15425

Generated at Fri Oct 09 17:19:26 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.