Issue Details (XML | Word | Printable)

Key: GLASSFISH-15017
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: oleksiys
Reporter: johanvos
Votes: 0
Watchers: 2
Operations

If you were logged in you would be able to see more operations.
glassfish

cpu 100% after grizzly-comet sends/receives large chunks of data

Created: 07/Dec/10 12:36 AM   Updated: 10/Dec/10 05:28 AM   Resolved: 10/Dec/10 05:28 AM
Component/s: grizzly-kernel
Affects Version/s: 3.1_b30
Fix Version/s: 3.1_ms07

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive comettest.zip (35 kB) 07/Dec/10 12:36 AM - johanvos
2. Java Archive File grizzly-comet-1.9.24.jar (29 kB) 09/Dec/10 08:43 AM - oleksiys
3. Zip Archive server.log.zip (91 kB) 09/Dec/10 12:04 PM - Joeri Sykora

Environment:

Linux marc 2.6.32-26-generic-pae #48-Ubuntu SMP Wed Nov 24 10:31:20 UTC 2010 i686 GNU/Linux


Tags:
Participants: Joeri Sykora, johanvos and oleksiys


 Description  « Hide

I have a problem when sending/receiving large Strings from/to Comet-grizzly. I've uploaded a simple example at http://lodgon.com/johan/comettest.zip
The example consists of 2 parts:

1. CometTestServer: a WAR with 2 servlets and a CometHandler.

2. CometTestClient: a Java client with a Main class and a AsyncRedReader class that calls a servlet and receives input.

The basic idea is to send a String from the client to the server, and the server uses Comet to send it to all registered clients.

In the Main class on the client, I can set the size of the message. In case the message is small (< 100K characters, it works fine). In case the message is large (e.g. > 100K characters), no input is received. Sometimes, this already happens with 1 client, while during other attempts I have to start e.g. 4 clients before it stops working.

The Glassfish server uses 100% CPU once the problem occurs, a Grizzly TCP Selector thread keeps running and uses a CPU. The stacktrace below is a snapshot generated by

asadmin generate-jvm-report --type=thread

Thread "Thread-41" thread-id: 92 thread-state: RUNNABLE
at: sun.nio.ch.FileDispatcher.read0(Native Method)
at: sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at: sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at: sun.nio.ch.IOUtil.read(IOUtil.java:210)
at: sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at: com.sun.grizzly.comet.CometTask.checkIfClientClosedConnection(CometTask.java:192)
at: com.sun.grizzly.comet.CometTask.handleSelectedKey(CometTask.java:178)
at: com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:274)
at: com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:258)
at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:195)
at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:130)
at: java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at: java.lang.Thread.run(Thread.java:619)

Most of the snapshots I take have this Thread in CometTask.checkIfClientClosedConnection, but it seems to be looping forever somewhere. The execution time of this thread is equal to the time since the client-hang occurs.

Note that the CometHandler.onEvent() tells that everything has been sent to the client, but the client didn't receive anything.

I added some debug info to Comet (grizzly-comet 1.9.18o) and I have more info:

when sending small messages, the CometTask.handleSelectedKey(SelectionKey selectionKey) only gets called when the client closes the connection.

With large messages, this method is called infintely after comet has written the message to the output:

[#|2010-12-06T21:17:07.897+0100|SEVERE|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=31;_ThreadName=Thread-1;|java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1206)
at com.sun.grizzly.comet.CometTask.handleSelectedKey(CometTask.java:163)
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKey(SelectorHandlerRunner.java:274)
at com.sun.grizzly.SelectorHandlerRunner.handleSelectedKeys(SelectorHandlerRunner.java:258)
at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:195)
at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:130)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)



Sort Order: Ascending order - Click to sort in descending order
oleksiys made changes - 09/Dec/10 08:43 AM
Field Original Value New Value
Attachment grizzly-comet-1.9.24.jar [ 23819 ]
Joeri Sykora made changes - 09/Dec/10 12:04 PM
Attachment server.log.zip [ 23823 ]
oleksiys made changes - 10/Dec/10 05:28 AM
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 3.1_ms07 [ 11016 ]
Resolution Fixed [ 1 ]