[GLASSFISH-21005] GlassFish stops responding to http connections if client aborts downloads (grizzly or http service i suppose) Created: 08/Mar/14  Updated: 13/Mar/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: pranahata Assignee: oleksiys
Resolution: Unresolved Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CenOs 64 bit, Java 7 u 51



 Description   

To reproduce the issue

Create a Serlvet that serves a file.

From a different box, do this a few hundred times.

HttpURLConnection connection = null;

URL url = new URL("http://lala:8080/myapp/myfileservingservlet");
connection = (HttpURLConnection)url.openConnection();

connection.setDoInput(true);
connection.setRequestMethod("GET");
InputStream is = connection.getInputStream();
connection.getHeaderField("Content-Type);
connection.disconnect();

GlassFish logs shows a fair few of these:

Caused by: java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:298) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:613) ~[nucleus-grizzly-all.jar:na]

And also a fair few of these

java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.flushByteBuffer(TCPNIOTransport.java:1252)

Trying to get anything else from the server shows:

Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

Note that it is easier to reproduce if the request goes over the internet but doesn't happen so frequently if using localhost, no sure about another box in the local network.

I have also seen the same errors in the logs when launching jws applications via JnlpDownloadServlet. To my undertanding the jws container does a similar thing by requesting the http headers of a jar but then aborting the download if the jar is in the jws cache and hasn't been modified.

Setting as blocker as it can make the server unusable and there one can not control how http clients interact with the server.



 Comments   
Comment by oleksiys [ 12/Mar/14 ]

Can you please try this patch [1] and check if it's still reproducible?

Thanks.

[1] https://dl.dropboxusercontent.com/u/7319744/nucleus-grizzly-all.jar

Comment by pranahata [ 13/Mar/14 ]

Olexsys,

I am quite busy at the moment, the test case is pretty simple. Where you able to reproduce the error without the patch?





[GLASSFISH-18803] no response for a chunked package Created: 13/Jun/12  Updated: 14/Dec/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b23
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: yavuzs Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 1 day
Time Spent: Not Specified
Original Estimate: 1 day
Environment:

SunOs, x86, 64bit


Attachments: File seac808012.cap    
Tags: http, http-listener

 Description   

there is a server that sends us chunked requests (by POST), our application on glassfish takes these packages and process them. but somehow, some of the packages are not processed by the application, also there is no log activity about these packages. but when we capture the network activity we can see that the request has come to the server.
we tried "-Dcom.sun.enterprise.web.connector.grizzly.enableSnoop=true" parameter. with this parameter, for the lost request there is no log.
here is a lost request:
Host: 192.168.100.10:49205
Transfer-Encoding: chunked

185
<?xml version='1.0' encoding='UTF-8'?>
<env:Envelope xmlns:env='http://www.w3.org/2003/05/soap-envelope' xmlns:seadac='http://www.schange.com/2004/SeaDAC/triggers'>
<env:Header><seadac:transaction transaction-id='

{20321632-B758-442a-ADFE-6E4607343AE3}

' origination-time='2012-06-12T08:29:04Z' env:role='http://www.w3.org/2003/05/soap-envelope/ultimateReceiver'/></env:Header>
<env:Body>
80
<seadac:subscription-notification subscription-id='

{1942B6B8-158F-4279-B9E9-0429671BEC67}

' current-request='3' next-request='4'>
c7
<seadac:hierarchy-changed hierarchy-uid='539401'><seadac:node-added hierarchy-uid='539710' asset-uid='528673'/></seadac:hierarchy-changed></seadac:subscription-notification></env:Body></env:Envelope>
0

you can find the capture attached.
frame 7824 and 7825 are the same request that the server sends to all subscribed clients. for this example there are two client subscribed. the gf application and the tomcat application. as you see, at the frame 7824 tomcat responded the request but at the frame 7825 fg did not respond the request.
regards,



 Comments   
Comment by yavuzs [ 19/Jun/12 ]

any response?

Comment by oleksiys [ 14/Dec/12 ]

sorry for the late response.
is it still the case w/ Glassfish 3.1.2.2?

thanks.

Comment by yavuzs [ 14/Dec/12 ]

the version is : GlassFish Server Open Source Edition 3.1.2 (build 23)

Comment by oleksiys [ 14/Dec/12 ]

yes, I understand, but there is a newer version (patch release) where this issue might have been fixed.
are you able to reproduce the issue consistently, or it happens sporadically?

Comment by yavuzs [ 14/Dec/12 ]

it happens sporadically. we didn't try new patch yet.





[GLASSFISH-21211] Grizzly NIO selectors running postponed tasks indefinitely causing high CPU usage Created: 19/Sep/14  Updated: 22/Oct/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1_b10
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: afcarv Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 4.1_b13, JDK 1.7u67, Linux CentOS 3.10.48 x64



 Description   

I have an nginx http server load balancing a Glassfish cluster composed of 2 instances in different machines. These instances serve a Jersey application composed of several RESTful webservices.

After some time running, I noticed a great increase in CPU load in one of the instances, even though client request throughput remained the same. Looking at the server threads I see that two of them are constantly running, being the cause of this increase.

Checking a heap dump I see that both of these threads are running selectors with postponed TCPNIOConnection tasks, which in turn do not seem to be processed correctly.

After turning on Grizzly logs I see lots of entries like the one below:

[2014-09-19T14:07:42.393+0000] [glassfish 4.1] [FINEST] [] [org.glassfish.grizzly.ProcessorExecutor] [tid: _ThreadID=105 _ThreadName=http-listener-1-kernel(8) SelectorRunner] [timeMillis: 1411135662393] [levelValue: 300] [CLASSNAME: org.glassfish.grizzly.ProcessorExecutor] [METHODNAME: execute] [[
executing connection (TCPNIOConnection{localSocketAddress=

{/10.248.232.5:28080}

, peerSocketAddress={/10.248.233.91:42864}}). IOEvent=WRITE processor=org.glassfish.grizzly.filterchain.DefaultFilterChain@3d4f0b0b]]

Where the first IP address is the Glassfish instance and the second the nginx load balancer.

Checking the nginx machine I see no open sockets with the listed number.

My theory is that nginx closed the socket for some reason (timeout?) and now Glassfish is unable to connect to it. The task will keep running and consuming resources until a server restart (as selectors with postponed tasks will call non-blocking selectNow() method).

Is this a bug or is my configuration not supported?

Thank you!



 Comments   
Comment by oleksiys [ 22/Sep/14 ]

I added more logging to investigate the problem.
Can you pls. apply the patched jar [1] and enable FINEST logging for org.glassfish.grizzly.nio.DefaultSelectorHandler
After you reproduce the problem - pls. share the server.log* files.

Thank you!
[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

Comment by afcarv [ 22/Sep/14 ]

I have done as requested and restarted the cluster but it may take a while for the issue to manifest itself as I'm not sure what toggles it. Once it does, though, I'll get the log files.

Thanks!

Comment by oleksiys [ 23/Sep/14 ]

Thank you!

Comment by afcarv [ 24/Sep/14 ]

Ok, the event happened again and another thread is looping. Unfortunately due to the heavy log rotation, I could not get the beginning of it. Server log can be downloaded below but it's mostly composed of the same entries.

https://dl.dropboxusercontent.com/u/106573849/server.zip

Let me know how you want to proceed, thanks!

Comment by oleksiys [ 26/Sep/14 ]

Can you pls. apply this patch
https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

it has to give us more stacktraces logged, thank you.

Comment by afcarv [ 26/Sep/14 ]

Patch applied. Do I keep the same logger settings as before? Will post log files once it happens again, thanks!

Comment by oleksiys [ 27/Sep/14 ]

Yep, same logger settings. Thank you!

Comment by afcarv [ 27/Sep/14 ]

Happened again with one more thread. Logged some more info now, though I still couldn't get the beginning of it.

https://dl.dropboxusercontent.com/u/106573849/server-27-09.zip

Thanks

Comment by oleksiys [ 30/Sep/14 ]

pls. try this patch [1], let me know if you still see the problem.

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

Comment by afcarv [ 01/Oct/14 ]

Ok, applied the patch - will let it run for a few days and let you know if it happens again.

Thanks!

Comment by afcarv [ 02/Oct/14 ]

Looks like it's still happening - got another thread looping. Is logging still in place? Thanks

Comment by lcmoreno [ 02/Oct/14 ]

Not sure if this is the same problem I'm having, but it seems like it is. I found on Stackoverflow another user reporting this high cpu usage the same way I experienced: http://stackoverflow.com/questions/25922809/glassfish-4-grizzly-threads-heavy-cpu-usage

It looks like it happens when this sun.nio.ch.SelectorImpl.selectNow is called. When I have threads using just sun.nio.ch.SelectorImpl.select I don't have the high cpu usage problem.

Comment by oleksiys [ 03/Oct/14 ]

Can you pls. apply this patch [1].
Please enable FINEST logging for "org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter". The logging you changed earlier could be disabled.
Now you'll see even more logging messages

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

Comment by afcarv [ 06/Oct/14 ]

Patch applied, will let you know once I have the new data, thanks!

@lcmoreno

Likely the same issue, another user reported it at http://stackoverflow.com/questions/26107101/glassfish-4-1-high-cpu-usage

Comment by dirkroets [ 06/Oct/14 ]

We've also been experiencing this issue since we upgraded to Glassfish 4.1b13 last week. We're running single (non clustered) Glassfish instances behind nginx proxies.

I've applied the latest patch and enabled logging on one of the dev servers in the office. Will post the logs here when it reoccurs.

Comment by oleksiys [ 06/Oct/14 ]

thank you guys. pls. post the log files once you experience the problem again.

Comment by dirkroets [ 06/Oct/14 ]

Log files have been uploaded to dropbox - https://dl.dropboxusercontent.com/u/46663766/logs.zip

As far as I can see all the entries in the first log file in the sequence - server.log_2014-10-06T22-12-38 - were logged before the utilisation went up. Then around 2014-10-06T22:13:16.417+0200 in the log file - server.log_2014-10-06T22-13-16 - the problem starts appearing.

Comment by oleksiys [ 07/Oct/14 ]

I have a suspect now, can you pls. run this patch [1], it should clarify if I'm right or not.

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/nucleus-grizzly-all.jar

Comment by dirkroets [ 07/Oct/14 ]

Patch has been loaded on one of our dev servers. Will monitor throughout the day and post the logs!

Comment by dirkroets [ 07/Oct/14 ]

Ok, log files have been uploaded to dropbox again - https://dl.dropboxusercontent.com/u/46663766/logs-2.zip

First signs of the issue I can see are in server.log_2014-10-07T10-22-36 It took a while for me to realise that the issue had reappeared, so there are plenty more log files, but they just seem to repeat the same entry.

Comment by oleksiys [ 08/Oct/14 ]

Ok, please try this patch [1], hopefully it will resolve the problem.

Thank you.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

Comment by dirkroets [ 08/Oct/14 ]

Thanks!

I've loaded the patch onto one of the servers. Will monitor and report back.

Comment by afcarv [ 08/Oct/14 ]

I've applied this latest patch as well, will let you know if it happens again - thanks

Comment by dirkroets [ 09/Oct/14 ]

Yesterday I applied the patch to one of our dev servers and to one of our production servers. The dev server is still running smoothly, but the issue has reappeared on the production server. It is possible that the dev server just doesn't see enough traffic for the issue to have been triggered again. In the past the dev server has also sometimes run smoothly for a day without the issue appearing.

I can turn on logging on the production servers, but unfortunately restarts have to be done between 22:00 and 06:00 CAT if I have to patch again. For the moment I'm going to ask someone in my team to try and reproduce on the dev server.

Comment by oleksiys [ 09/Oct/14 ]

@dirkroets yes, pls. turn on FINEST logging on:
"org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter"
"org.glassfish.grizzly.nio.TCPNIOAsyncQueueWriter"

Comment by dirkroets [ 10/Oct/14 ]

Here are the server log files for a server running the latest patch.

We've now succeeded in reproducing this issue "at will" on one of our dev servers. The setup is a very basic nginx proxy that forwards http requests to a GF4.1 server running a simple web application with at least one page that takes longer than 1s to process. The nginx configuration for the hostname being forwarded looks as follows:

server {
  listen          80;
  server_name     gf.localdomain;
  location / {
    proxy_pass              http://192.168.200.7:8080;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Forwarded-Host $host;
    proxy_set_header        X-Forwarded-Server $host;
    proxy_set_header        Host $http_host;
    proxy_connect_timeout         1s;
    proxy_send_timeout         1s;
    proxy_read_timeout         1s;
    client_max_body_size    20M;
  }
}

By setting the connect, send and read time outs to 1 second each it guarantees that nginx will handle requests for every page that takes longer than 1 second to respond as a time out. With this configuration we can trigger the issue consistently be requesting a simple page from a servlet that has Thread.sleep(2000).

https://dl.dropboxusercontent.com/u/46663766/logs-3.zip

EDIT: It turns out that it is not quite as simple to reproduce this issue as I thought. This morning we were able to easily reproduce on the test server, but after a restart of the test GF server and the nginx proxy server we cannot reproduce again with the simple Thread.sleep servlet. If we do manage to consistently reproduce it I'll post the web app here.

Comment by afcarv [ 10/Oct/14 ]

@dirkroets

Out of curiosity, which nginx version are you using? Have you tried to set proxy_ignore_client_abort to check if it solves or mitigates the issue? I think I may try that to see what happens. Thanks

Comment by dirkroets [ 10/Oct/14 ]

@afcarv , we're running nginx 1.4.6 and 1.6 in front of the two servers where this issue most frequently occurs. We have not tried proxy_ignore_client_abort yet. I will test that tonight and report back.

Comment by oleksiys [ 11/Oct/14 ]

Unfortunately this log doesn't help much, because I don't see the moment when it started failing, looks like there is some part missed?
Anyway I've updated the patch to add logs [1].

One more update, I was inaccurate with one logging domain, the correct ones are:
"org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter"
"org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter"

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21211/nucleus-grizzly-all.jar

Comment by dirkroets [ 13/Oct/14 ]

I've loaded the new patch and fixed the logging. Will upload the logs as soon as this reappears!

Comment by dirkroets [ 13/Oct/14 ]

Ok, that was quite quick. I've uploaded the latest logs to dropbox and provided the link below this comment.

The issue was still triggered using an nginx configuration without proxy_ignore_client_abort I've now added the directive to nginx to see if it still gets triggered.

[1] https://dl.dropboxusercontent.com/u/46663766/logs-4.zip

Comment by dirkroets [ 13/Oct/14 ]

We've been able to reproduce this issue again with proxy_ignore_client_abort set for all locations except for locations used exclusively by websockets.

[1] https://dl.dropboxusercontent.com/u/46663766/logs-5.zip

Comment by oleksiys [ 13/Oct/14 ]

Thank you!
One more attempt to fix it [1]. If doesn't work - please attach the logs.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/nucleus-grizzly-all.jar

Comment by dirkroets [ 14/Oct/14 ]

Thanks! I've loaded the patch onto one of our dev servers. Will try to trigger the issue and report back.

Comment by dirkroets [ 15/Oct/14 ]

Just a quick update...

We've not been able to trigger this issue again on the dev server since I've loaded the patch. It is still a bit inconclusive though because there were other times when we could also not reliable reproduce the issue on the dev server in 24 hours. We'll keep putting some effort into this on the dev server today and tomorrow and if everything seems good I'll give this a try on one of our production servers from Thursday night.

Comment by oleksiys [ 15/Oct/14 ]

thank you for the update!

Comment by afcarv [ 15/Oct/14 ]

I've applied the new patch as well, will post results later. Thanks!

Comment by dirkroets [ 16/Oct/14 ]

Everything is still looking good on the dev server!

I'm going to try the patch on one of our production servers in a couple of hours from now. Although we've not been able to directly trigger the issue on our production servers in the past the issue has shown up consistently within a few minutes of starting up any of the production servers, presumably because of the higher volumes of traffic. I should therefore tomorrow be able to say with quite a lot of certainty whether the latest patch has solved the problem for us.

Comment by oleksiys [ 16/Oct/14 ]

sounds good! Thank you for the update!

Comment by dirkroets [ 17/Oct/14 ]

After having closely monitored all the servers where we've loaded the latest patch I'm happy to report that the issue hasn't reappeared on any of these servers. We are seeing a great improvement in CPU utilisation as well as a great improvement in the number of requests timing out between our nginx proxies and app servers.

The bug is fixed as far as I'm concerned. Thanks, @oleksiys !

Comment by afcarv [ 17/Oct/14 ]

Looking good to me as well – no issues so far; will leave it running throughout the weekend and check again on Monday. Thanks Oleksiy!

Comment by dirkroets [ 20/Oct/14 ]

We've now deployed the patch to all our production servers and the issue hasn't reappeared on any of those servers.

Comment by afcarv [ 20/Oct/14 ]

Yup, no sign of the issue yet - pretty sure this has been fixed. Thanks!

Comment by oleksiys [ 20/Oct/14 ]

grizzly issue
https://java.net/jira/browse/GRIZZLY-1712

Comment by afcarv [ 22/Oct/14 ]

By the way - previously, given enough time, instances would crash with an exception like the one below:

[2014-10-22T02:42:30.987+0000] [glassfish 4.1] [SEVERE] [] [org.glassfish.grizzly.nio.SelectorRunner] [tid: _ThreadID=98 _ThreadName=http-listener-1-kernel(4) SelectorRunner] [timeMillis: 1413945750987] [levelValue: 1000] [[
doSelect exception
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
at org.glassfish.grizzly.threadpool.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:466)
at org.glassfish.grizzly.threadpool.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:81)
at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:161)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.executeIoEvent(WorkerThreadIOStrategy.java:100)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:414)
at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:383)
at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:347)
at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
at java.lang.Thread.run(Thread.java:745)
]]

Thought this was due to the pile of TCPNIOConnection tasks that kept accumulating, so I didn't mind it too much, but it looks like this is still happening, even after the patch, so I'm not sure if it's related to this issue now. Haven't tried to change max queue size yet.

Comment by oleksiys [ 22/Oct/14 ]

I think it can be not related to the fixed issue.
You can open a new one and we can take a look at it.

Comment by afcarv [ 22/Oct/14 ]

Ok, thanks for clarification - will try to get some more details first.





[GLASSFISH-21246] Grizzly thread pool waiting on LinkedTransferQueue hangs application Created: 03/Nov/14  Updated: 10/Nov/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1_b10
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: afcarv Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 4.1_b13, JDK 1.7u67, Linux CentOS 3.10.48 x64



 Description   

Elaborating on the issue I posted on GLASSFISH-21211.

After some time running, http-listener-1 thread pool stops responding. Checking the logs, I see multiple messages like the one below:

[2014-11-03T18:50:47.114+0000] [glassfish 4.1] [SEVERE] [] [org.glassfish.grizzly.nio.SelectorRunner] [tid: _ThreadID=60 _ThreadName=http-listener-1-kernel(1) SelectorRunner] [timeMillis: 1415040647114] [levelValue: 1000] [[
  doSelect exception
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
        at org.glassfish.grizzly.threadpool.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:464)
        at org.glassfish.grizzly.threadpool.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:81)
        at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:161)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.executeIoEvent(WorkerThreadIOStrategy.java:100)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:414)
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:383)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:347)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:744)
]]

I then took a thread dump which shows all threads in http-listener-1 in a state like the below:

"http-listener-1(31)" daemon prio=10 tid=0x00007ff19422b000 nid=0x6c61 waiting on condition [0x00007ff1f6eed000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006036fd360> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
        at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:557)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:744)

They seem to be waiting on LinkedTransferQueue take() method, which appears to be not responding (waiting for some item to arrive in the queue?). I also took a memory dump that I can look into if someone points me in the right direction.

Thought this could be related to GRIZZLY-1711 but after applying the latest version (2.3.17-SNAPSHOT) this behavior is still happening.

Let me know if I should get more data - thanks!



 Comments   
Comment by davidwinters1980 [ 03/Nov/14 ]

This appears to be occurring as a result of the http thread pool reaching its limit of 4096.

As per the default domain.xml setting:

<thread-pools>
<thread-pool name="admin-thread-pool" max-thread-pool-size="50" max-queue-size="256"></thread-pool>
<thread-pool name="http-thread-pool" max-queue-size="4096"></thread-pool>
<thread-pool name="thread-pool-1" max-thread-pool-size="200"/>
</thread-pools>

You could configure this to -1 or unlimited but this will likely lead to other issues such as out of memory etc

It would appear as though tasks are being added to this queue faster than the tasks can be processed. I would investigate why this occurring? If you could take a number of thread dumps over a period of time whereby we capture this issue occurring that would be useful.

Furthermore, it maybe worth increasing the value of your http request processing threads as this will allow more tasks to be handled concurrently off the http thread task queue and so it may reach the 4096 limit so quickly. What is the current value of the number of http request processing threads??

The issue you have here is very dependent on the number of http requests being pushed to the http task pool and how quickly these requests are being handled so appropriate tuning of the http parameters may resolve this issue. If this fails then looking at the state of all threads running over a period of time should give you a hint as to why requests are taking so long to process.

Comment by afcarv [ 03/Nov/14 ]

I don't think it's a configuration issue as the environment is able to handle much heavier loads - it may be due to specific and temporary environmental reasons (network latency, database slowdown, etc) but you'd think it would resume work as soon as this is normalized. This does not happen (waited for some hours to be sure), even if the consumer is stopped and there's no inbound traffic anymore - the only way to bring it back online is with a server/cluster restart. I did notice a lot of connections in the CLOSE_WAIT state in the server, though.

Haven't tried to raise the queue limit or remove it yet, but I suspect it would only delay the occurrence or eventually reach an OOM error.

I see no obvious reason for this looking at the thread pool dump, but will post it later for analysis. Thanks!

Comment by afcarv [ 04/Nov/14 ]

Please find attached the full thread dump and the server log file. The dump was taken about 30 minutes after the issue manifested itself - you can see all threads in the http-listener-1-kernel pool reach the queue limit and stop responding. http-thread-pool contains 32 threads; there are 8 acceptor threads.

Interestingly, http-listener-2 (SSL) continues to respond normally. http-listener-1 hanged and required a server restart. The configuration is a 2 instance cluster with a nginx load balancer in front; there are Jersey applications deployed serving RESTful webservices. The configuration can handle about 5x the average load on the server - no increased throughput was observed leading to the issue, which persisted after stopping all inbound traffic.

I will schedule a cron job to take a thread dump every 5 minutes to check on any odd behavior leading to the issue - thanks!

Thread dump: https://www.dropbox.com/s/qejdzhwejlv1zzp/threadump.txt?dl=0
Server log: https://www.dropbox.com/s/rbu24cep9y25aol/server.log?dl=0

Comment by afcarv [ 04/Nov/14 ]

Some more info I got from the memory dump -

I see lots of instances of TCPNIOConnection (1500+) in what appears to be a closing state; as the latest snapshot was applied I can see these with a CloseReason of "LOCALLY". May explain the connections in the CLOSE_WAIT state I saw? Is it possible that nginx closed the socket before a FIN packet could be sent from the server, and now it is not able to end the connection properly? Not sure if this could be just a consequence of some other issue, though.

Comment by oleksiys [ 06/Nov/14 ]

Which Grizzly version is used in 4.1_b10?

Thank you.

Comment by afcarv [ 06/Nov/14 ]

We're using b13 (couldn't find the tag for it so I selected the closest), but I believe it's the same version - 2.3.15.

Currently, 2.3.17-SNAPSHOT is applied.

By the way - we found out that what triggers this behavior is a DB performance degradation, causing the requests to queue and eventually reach the limit. No additional info on why queue isn't being reduced, though.

Comment by davidwinters1980 [ 06/Nov/14 ]

Could you take a number of thread dumps say every 5 minutes so that we can compare the different thread dump files leading up to the hang? Also could you send us on the contents of the <transports> which should contain the different tcp parameters used by Grizzly.

<transports>
<transport byte-buffer-type="HEAP" display-configuration="true" name="tcp" .....></transport>
</transports>

Useful to know db degradation issues are causing requests in the queue to fill up. It might be useful to get some debug grizzly logging on here: org.glassfish.grizzly.level=FINEST

Thanks.

Comment by afcarv [ 10/Nov/14 ]

I had some monitoring in place but we've been focusing on fixing the db degradation so the issue won't be triggered - so far we've been successful, so it looks unlikely I'll be able to collect more data in the production system.

What I'm going to try next is to create a sample application and configuration that will be able to recreate the issue - doesn't look too hard; just a sample RESTful webservice that performs a db query that has an artificial delay in response, small db connection pool and a sample JMeter test script with http requests that will timeout before the app has a chance to respond. Should replicate the conditions pretty closely (don't know if nginx in front plays a role in this or not, so will try without it first).

Will report on any news, thanks!

Comment by oleksiys [ 10/Nov/14 ]

pls. try to patch GFv4.1 with this patch [1].
I need server.log output message(s) like:
"The queue is overflowed. queuePermits=......."

Thank you.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21246/nucleus-grizzly-all.jar





[GLASSFISH-21202] Glassfish/Grizzly Memory Leak in Glassfish 4? Created: 16/Sep/14  Updated: 12/Dec/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: talk2umakanth Assignee: oleksiys
Resolution: Unresolved Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

GlassFish Server Open Source Edition 4.0 (build 89)
Red Hat Enterprise Linux Server release 5.9 Beta (Tikanga)
Linux 2.6.18-339.el5 #1 SMP Mon Aug 27 15:42:17 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux



 Description   

We're facing very frequent out of Memory problems in our Glassfish Installation of an enterprise web application.

The application is running in a cluster of 18 JVMs. Initial heap size was 1 GB which resulted in the JVMs going down almost every hour and we have increased the heap size to 1.5 GB which makes the JVMs to crash every 3-4 hours.

The Heap dump does not show any application objects at all whenever the JVMs crash and shows the same objects from Glassfish/Grizzly consuming majority of the memory all the time. Below object is found to consume more than 65% of heap in most of the heap dumps.

762,277,600 bytes (52.71 %) of Java heap is used by 5,632,379 instances of org/glassfish/grizzly/memory/HeapMemoryManager$TrimmableHeapBuffer

Following error is also seen in the logs frequently (which is due a malformed URL having && in the request query string??). Can this be a potential cause of the problem? Running a 12 hours load test using similar URL does not produce out of memory in non production systems.

GRIZZLY0155: Invalid chunk starting at byte [387] and ending at byte [387] with a value of [null] ignored]]

The error is not reproducible in non-production environment but heap dumps from production can be provided if required.

  • Are there any know memory leaks in Glassfish 4.0 (Build 89)/Grizzly 2.3.1?
  • What does the object HeapMemoryManager$TrimmableHeapBuffer constitute of? Is there any way this can be avoided if there are no known/feasible fixes for Glassfish or Grizzly?

Additional Information (may not be relevant)

  • Each JVM is fronted by an Apache 2.4.9 HTTP Server using mod_jk to communicate with glassfish which are in turn balanced by Netscale load balancer.


 Comments   
Comment by oleksiys [ 18/Sep/14 ]

Heap dump tree could help us understand the problem.

Thank you.

Comment by talk2umakanth [ 18/Sep/14 ]

You can see the heap tree in the image: http://i.stack.imgur.com/tJEQV.png

All the heapdumps show the same objects leaking as is being shown in the image.

Comment by oleksiys [ 18/Sep/14 ]

Can you pls. apply the following patch (2 jars) [1] and [2] and enable FINER logging level for org.glassfish.grizzly.http.io.InputBuffer
When/if it fails pls. attach the thread dump and logs.

Thank you.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.0/nucleus-grizzly-all.jar
[2] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.0/glassfish-grizzly-extra-all.jar

Comment by anshul.kumar [ 18/Sep/14 ]

Thanks oleksiys. Does this patch contain any fixes as well or it is just to gather additional logging information? We just want to make sure before promoting it in the production system. It has tested successfully in non-production environment.

Comment by oleksiys [ 19/Sep/14 ]

It does have different glassfish4 fixes, not just logging.

Comment by anshul.kumar [ 19/Sep/14 ]

We applied this patch on one node containing 3 JVM instances of the application and the instances are still crashing after applying the patch.

The crashdumps including JVM Log files, thread dumps and the images of heap and permgen space leading up to out of memory of the instances can be downloaded from this link: https://drive.google.com/file/d/0B52890ypA1_bYlR4Snh5bUxLMzg/edit?usp=sharing

Note: The JVM Time Zone is one hour behind the timestamps that are visible in the images of heap size and permgen size. Another observation that we are seeing is that the JVM crashes just when it tries to resize the permgen space. Max Permgen space for JVM is set to 512 MB. Can this be a part of problem and should we try setting min and max perm size to be the same value to avoid permgen resize?

Comment by oleksiys [ 22/Sep/14 ]

Thank you for the info.
Can you pls. apply this patch [1] and also enable FINEST logging for
org.glassfish.grizzly.http.util.Parameters?

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/nucleus-grizzly-all.jar

Comment by anshul.kumar [ 22/Sep/14 ]

Thank you for the information.

Do you want to keep the logging level of org.glassfish.grizzly.http.io.InputBuffer at FINER as well or we should switch that logger off now?

Comment by oleksiys [ 23/Sep/14 ]

please keep it as well.

thank you.

Comment by anshul.kumar [ 23/Sep/14 ]

The crash dumps of two instance crashes from today after applying the updated jar file can be downloaded from here:
https://drive.google.com/file/d/0B52890ypA1_bbEFpLS1zZkxpbzg/edit?usp=sharing

(Thread dumps are also included in the zip file). If you need more data, please let me know.

Thanks

Comment by anshul.kumar [ 25/Sep/14 ]

Is there anything that could be found in the latest crash dumps? Should we collect any other data as the instances are still crashing every 2-3 hours

Comment by oleksiys [ 25/Sep/14 ]

The logs look fine the only piece of information missed (from the very latest log) is thread http-listener-3(44), from the thread dump I see this thread is constantly reading data, but I don't see any logging activity from this thread in the server.log* files.
For now it looks like the problem might be caused by a huge HTTP POST request, which represents form parameters.

Can you pls. share the raw heap dump file so I can learn it locally?

Thank you!

Comment by anshul.kumar [ 26/Sep/14 ]

Thank you! A crashdump including the server.log* files, thread dumps and heapdump from the latest crash can be downloaded from here: https://drive.google.com/file/d/0B52890ypA1_bVzBvaGtjTHNYTE0/edit?usp=sharing

Please let me know if you need any additional data. I am going to reduce the maximum post size for the glassfish instances to 1-2 MB and will enable the option to print the request size in HTTP Server (apache) to see if there was a big POST request leading up to the crash.

Comment by anshul.kumar [ 26/Sep/14 ]

I enabled the request size directive in HTTP Server. The maximum size of GET or POST data is 171 KB leading up to the latest crash. Is that POST size big enough for the JVM to consume 1.5 GB memory and should I trey reducing this size?

You can see the log of requests for the latest crash by downloading the spreadsheet from here: https://drive.google.com/file/d/0B52890ypA1_beDZReldlWV9MLTA/edit?usp=sharing

A common pattern I have noticed is that there is one request that always takes more than 50 minutes to process with a return code of 502 leading up to the crash. The size of this request is very small (131 bytes). This request can be seen starting at 26-Sep-14 10:26:34, row number 26721 in the spreadsheet. The request completed eventually with a return code of 502 when the instance was killed and restarted. I hope this might give some extra details about the problem.

When the instance is healthy, the same request takes only 1-2 seconds.

Comment by oleksiys [ 27/Sep/14 ]

I think I found the problem, can you pls. try this patch (2 files).

https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/nucleus-grizzly-all.jar
https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/glassfish-grizzly-extra-all.jar

Pls. let me know if it works for you.

Thanks!

Comment by anshul.kumar [ 27/Sep/14 ]

This is great new Alexis. Do we need to enable any logging after this patch. I had switched off the loggers after some time as it creates a huge log backup within hours.

Comment by oleksiys [ 27/Sep/14 ]

No, you don't need any extra logging now, it should just work.

Comment by anshul.kumar [ 28/Sep/14 ]

so far so good. I patched one node with three instances on it and the instances have been very stable (never used more than 550 MB of heap) while the other instances that have not been patched are still crashing. I will update on Monday evening as we do not have comparable traffic over weekends as weekdays.

Comment by anshul.kumar [ 05/Oct/14 ]

Alexis - I have slowly rolled out this patch to our entire environment and the JVM instances have been very stable following the patch. The patch has resolved the problem permanently and this issue can be closed.

Very curious to know why we were not able to recreate the problem in our pre-prod application using load tests though.

Thank you for caring.

Comment by oleksiys [ 06/Oct/14 ]

I'm glad the fix works.
I assume your testing env. also has Apache and GF backend communicating via AJP protocol.
Try to reproduce the following usecase: send HTTP post message to apache, which contains entire HTTP header (with Content-Length) and a half of the POST payload, then close HTTP connection... Something like:

$ telnet localhost 80
POST /xxxxx HTTP/1.1
Host: localhost:80
Content-Length: 100
Content-Type: application/x-www-form-urlencoded

aaa=bbb&^C^C^C
Comment by smillidge-c2b2 [ 06/Oct/14 ]

Is there a Grizzly JIRA I can track for this if the fix is in Grizzly?

Comment by oleksiys [ 06/Oct/14 ]

I just created one here
https://java.net/jira/browse/GRIZZLY-1709

Comment by anshul.kumar [ 07/Oct/14 ]

Will this fix be included in any of the future Glassfish 4.1 release?

Comment by oleksiys [ 07/Oct/14 ]

Future - yes, but 4.1 has been released already. I'll prepare a patch for it separately.

Comment by kallie82 [ 09/Oct/14 ]

Has a patch been prepared perhaps? We are running into what seems to be the same issue on GlassFish Server Open Source Edition 4.1 (build 13).
Thanks!

Comment by anshul.kumar [ 20/Oct/14 ]

Alexis - Can you please help provide a patch for GF 4.1 as well for this issue. I have a few pending upgrades for the latest glassfish version

Comment by oleksiys [ 12/Dec/14 ]

here is the patch for 4.1
https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/glassfishv41-patch.zip





[GLASSFISH-6371] Be able to use EJB3 using a http tunnel Created: 30/Sep/08  Updated: 01/May/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: V3
Fix Version/s: future release

Type: New Feature Priority: Major
Reporter: batmat Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issuezilla Id: 6,371

 Description   

Hi,

After having considered migration to EJB3, we realize that GF doesn't have an
"http-invoker" feature.

In fact, we'd like to be able to tunnel all our call to the server through an
http one.

The better would be being able that the client doesn't need anything else than
http to (1) resolve the remote ejb reference, (2) call it. To sum up, would be
great if GF has the corresponding feature of JBoss (e.g. see
http://wiki.jboss.org/wiki/Accessing_EJB3s_over_HTTP_HTTPS or
http://wiki.jboss.org/wiki/EJBOverHTTPWithUnifiedInvoker).

At least, we'd be interested into knowing how we could inject say our transport
classes (integrating spring http-invoker), and we'd also be glad to know if
there's any chance that it gets into GF if we provide a patch for it.

Cheers.
Thanks for your work, GF is already a masterpiece.



 Comments   
Comment by km [ 01/Oct/08 ]

Looks like this is an interesting feature request. Thanks for filing it as such.
If I understand it correctly, you need some standard servlets that will tunnel
EJB calls via HTTP. This is useful for all the applications that have EJB
components, but don't have Web components that front end those EJB components.

IMO, JBoss makes you do too many configuration changes for this to be achieved.

Comment by batmat [ 01/Oct/08 ]

> IMO, JBoss makes you do too many configuration changes for this to be achieved.
I totally, fully, wholeheartedly agree. This configuration is a nightmare.

What I would dream of would be to be able to configure a glassfish out of the
box using say a particular setup-firewallconstrained[cluster].xml and be done
(and providing the right jndi property to access).

More generally, for me, this feature request is about being able to reduce the
number of needed ports, to simplify a firewall configuration. And more
importantly, to delete the dynamic ports/be able to specify fixed ones. Kind of
UnifiedInvoker as it exists in JBoss.

If you think it'd be a good thing to file another FR and link it to this one,
please just let me know.
Thanks.

Comment by Bill Shannon [ 01/Oct/08 ]

I know it's not exactly what you asked for, but given that you're separating
the client and server by a firewall, implying that they're more loosely
coupled, have you considered using JAX-WS to expose your EJBs as web services?
Yes, it's a somewhat different programming model, but it might be a better
match for your environment.

Comment by batmat [ 01/Oct/08 ]

Hi Shannon,
Well, at least, Sun coworkers are coherent. See the discussion with Ken Saks
and his proposal here:
https://glassfish.dev.java.net/servlets/ReadMsg?list=ejb&msgNo=527 .

To sum up, no. We're not considering WS programming for many reasons I describe
in the given thread.
Cheers.

Comment by Ken Cavanaugh [ 02/Oct/08 ]

One of the features we have been discussing for GFv3 is port unification for all protocols.
This would mean that HTTP, WS, and IIOP could all be handled over the same port.
Does this solve the requirement, or do you need more than just the same port?
We do occasionally see requests to allow tunneling of RMI-IIOP requests over HTTP,
but so far this has not been sufficiently common to consider implementing it.

Of course, you can also look at invoking EJBs directly from the HTTP path,
as others have discussed.

Comment by batmat [ 02/Oct/08 ]

> This would mean that HTTP, WS, and IIOP could all be handled over the same port.
Does this solve the requirement, or do you need more than just the same port?
That's not exactly the same thing, but it would definitely be a damned good
thing to convince people here about Java EE servers, and more precisely about GF
part.

So, I guess this could be sufficient.

> We do occasionally see requests to allow tunneling of RMI-IIOP requests over
HTTP, but so far this has not been sufficiently common to consider implementing it.
Do you see it as an interesting/acceptable request but don't have time to
implement it, or is it just not gonna be accepted even if we try to provide a
patch for it that'd suit you?

Cheers

Comment by batmat [ 27/Nov/08 ]

> One of the features we have been discussing for GFv3 is port unification for
all protocols. This would mean that HTTP, WS, and IIOP could all be handled over
the same port.

That feature would be really interesting. Isn't it this one:
https://glassfish.dev.java.net/issues/show_bug.cgi?id=4322.

Maybe the current issue should be put as related?

Cheers.

Comment by Tom Mueller [ 23/Jun/10 ]

Kedar no longer on project.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

Comment by jthoennes [ 26/Mar/12 ]

This is related to the port unification feature I am also looking for a long time now.
See GLASSFISH-4073: Move CORBA transport to Grizzly (https://java.net/jira/browse/GLASSFISH-4073).

If we could access the ORB for JNDI and EJB access using the Grizzly port unification framework
(port finder and port filter). See http://antwerkz.com/blog/port-unification-in-glassfish-3-part-1.

Batman, did you find any workaround so far?

Comment by Tom Mueller [ 04/Jan/13 ]

Moving to EJB component.

Comment by marina vatkina [ 04/Jan/13 ]

You can use EJBs as JAX-RS endpoints. Assigning to the JAX-RS for further evaluation.

Comment by kumara [ 24/Apr/13 ]

Well, programming in JAX-RS was already suggested as a solution and in that case there is nothing that the server has to do, the application needs to change.

The most concrete comments about a solution was use of port unification to ensure that IIOP traffic is on the same port as http. While this is not exactly a http tunnel, it might be enough for some scenarios.

I am going to request grizzly-kernel team to add a pointer to port unification documentation so that anyone referring to this issue has a handy reference on how to configure IIOP traffic on same port as http.

If port unification is not sufficient, given the availability of JAX-RS programming model and better integration between EJB and JAX-RS, the most prudent option is to change the application.

After adding the document pointer, please assign this back to orb sub-component.

Comment by oleksiys [ 24/Apr/13 ]

Is Corba (IIOP) working on top of Grizzly?
If not then port unification will not work.

Comment by jthoennes [ 25/Apr/13 ]

A related issue is also whether OpenMQ would work using port unification. I think the new OpenMQ 5.0 is based on Grizzly, isn't it?

Comment by oleksiys [ 01/May/13 ]

yes, MQ should work.





[GLASSFISH-11031] ContainerMapper should better handle the empty string root path Created: 13/Nov/09  Updated: 13/Feb/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1
Fix Version/s: future release

Type: Bug Priority: Major
Reporter: jfarcand Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: Macintosh


Issuezilla Id: 11,031
Tags: 3_1-exclude

 Description   

See https://glassfish.dev.java.net/issues/show_bug.cgi?id=10601

The new emptyString support in Servlet 3.0 breaks the ContainerMapper mapping
algorithm. We patched the Mapper to execute in compatible mode for the 3.0
release, but we must fix it properly in ContainerMapper for the 3.0 release.



 Comments   
Comment by jfarcand [ 13/Nov/09 ]

Exclude for that release.

Comment by jfarcand [ 01/Dec/09 ]

Alexey is now the owner

Comment by oleksiys [ 11/Oct/10 ]

target for ms7

Comment by oleksiys [ 11/Nov/10 ]

the fix may cause regression in admin gui, SCF is not the best time to make this change.
moving to 3.2





[GLASSFISH-17098] Move network-configuration CLI commands to nucleus under "grizzly-glue" module Created: 25/Jul/11  Updated: 02/Dec/11

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Task Priority: Major
Reporter: oleksiys Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_x-exclude, nucleus-cleanup

 Description   

subj.






[GLASSFISH-20618] Create source bundle for nucleus-grizzly-all Created: 10/Jun/13  Updated: 10/Jun/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: jjsnyder83 Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Per the summary.



 Comments   
Comment by jjsnyder83 [ 10/Jun/13 ]

Please create the source bundle for nucleus-grizzly-all.





[GLASSFISH-21136] Log filled with TimeoutException on TCPNIOTransportFilter.handleRead Created: 18/Jul/14  Updated: 21/Jul/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1_b08
Fix Version/s: None

Type: Bug Priority: Major
Reporter: gcruscoe Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, Java 8u11


Tags: fishcat

 Description   

Not sure what the cause is because it does not link to any of our code in the stack trace. Could be related to code running in a ejb timers, but not sure.

The stack trace I get is this:

2014-07-18 14:50:26.466 - :: ERROR [http-listener-2(6)] javax.enterprise.web.core:287 - An exception or error occurred in the container during the request processing
java.io.IOException: java.util.concurrent.TimeoutException
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:90) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:999) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:351) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:695) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.blockingRead(InputBuffer.java:1119) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead(ServerInputBuffer.java:95) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.fill(InputBuffer.java:1143) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.read(InputBuffer.java:353) ~[nucleus-grizzly-all.jar:na]
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:267) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:234) ~[web-core.jar:na]
at org.apache.catalina.authenticator.FormAuthenticator.saveRequest(FormAuthenticator.java:614) ~[web-core.jar:na]
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:250) ~[web-core.jar:na]
at com.sun.web.security.RealmAdapter.invokeAuthenticateDelegate(RealmAdapter.java:1524) ~[websecurity.jar:na]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:606) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:702) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) ~[web-core.jar:na]
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) ~[web-glue.jar:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) ~[web-core.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) [kernel.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) [kernel.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) [nucleus-grizzly-all.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.util.concurrent.TimeoutException: null
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:126) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:75) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.AbstractReader.read(AbstractReader.java:72) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:77) ~[nucleus-grizzly-all.jar:na]



 Comments   
Comment by oleksiys [ 18/Jul/14 ]

looks like it happens during the form authentication, when the server expects more information from the browser, but the browser doesn't send it. The read attempt fails after read-timeout expires.

Comment by gcruscoe [ 18/Jul/14 ]

This occurs while there is no user interaction on the web page, but the page does make ajax calls and is connected via atmosphere using websockets. This also occurs on startup at once (although less frequently I believe) when there are no web browsers pointing at the app server.

The application has several war applications as well as a main ear application. The wars do web service calls to the ejb level web services (all on the same app server). So when the user is using the application there is a web session to the war and the war makes web service calls.

I am wondering if a web page without a session was trying to do ajax calls – although this doesn't explain the same stack trace on startup. Anyway hope any of this helps.

I'll update if I can come up with more examples of what makes this happen.

Comment by oleksiys [ 18/Jul/14 ]

pls. share the exception you see during the startup, because the one you posted is related to form authentication.

Comment by gcruscoe [ 21/Jul/14 ]

Okay exception during startup was actually the same cause. It is the atmosphere connection from a web browser – I had not found and closed all tabs before restarting the app server. Now I do not get this on startup.

The session and SSO / realm are most certainly expired – I do not know if that would contribute this issue. I will work on figuring out more about this and post back when I have more. Hopefully it is benign, but I have not seen these in the logs on gf 3.1.2.2 so it is a bit troubling.





[GLASSFISH-21189] Frequent network dropouts Created: 10/Sep/14  Updated: 19/Sep/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Bug Priority: Major
Reporter: pranahata Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

JDK 8 u 5, 64 bit CentOS



 Description   

We have a GlassFish server hosted on a datacenter and a jee application with a desktop application deployed via java web start running on it.

When the JWS client downloads the jars, we get very frequent network dropouts, mainly with jar files > 3 Mb where the JWS client reports: "EOF: Unexpected end of ZLIB input Stream" and the server reports the following exception.

ava.io.IOException: java.lang.InterruptedException
at org.glassfish.grizzly.utils.Exceptions.makeIOException(Exceptions.java:81)
at org.glassfish.grizzly.http.io.OutputBuffer.blockAfterWriteIfNeeded(OutputBuffer.java:958)
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:682)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:355)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:342)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:161)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at jnlp.sample.servlet.DownloadResponse$FileDownloadResponse.sendRespond(DownloadResponse.java:257)
at jnlp.sample.servlet.JnlpDownloadServlet.handleRequest(JnlpDownloadServlet.java:187)
at jnlp.sample.servlet.JnlpDownloadServlet.doGet(JnlpDownloadServlet.java:120)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.StandardWrapper.service(

Clients connecting over a wireless internet connection get these errors much more frequently.

The client application talks to the server via http.

If i do a ping -t to the server, the number of lost packets is not that big and sometimes there is no packet loss at all when this exception happens.

Dropouts seem to last no longer than one second yet, the amount of failed downloads or calls seems to be too big and causes too many errors on the application.

If this is not a bug, i would appreciate if someone told me if there is any setting that we can configure on glassfish to enable http clients to continue reading after a drop out or to change the timeouts or retries or any other configuration to make the communication more robust?



 Comments   
Comment by oleksiys [ 19/Sep/14 ]

pls. try to disable request-timeout on http listener for example for http-listener-1:

$asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds=-1




[GLASSFISH-21061] keepAlive-timeout happened between "qin" and "qout", the connection will be broken Created: 05/May/14  Updated: 08/Aug/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: houtang Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 4 days
Time Spent: Not Specified
Original Estimate: 4 days
Environment:

all


Tags: NullPointerException, conn, disc, grizzly

 Description   

When I try to access a web application through ie6 on glassfish 3.1.2.2, the following phenomenon has been happened:
If there's a keepAlive-timeout happened between "qin" and "qout", the connection will be broken. You will found from ※1 on the printed log that the connection has been broken.
BTW, when we access through ie6 after the first connection was broken, there will be another connection created(※2) and return back as normal. However, when we access the web application through ie8 after the first connection was broken, the second connection can't be created, so the log about ※2 can't be print to the log under this situation, the connecton will be broken here and it can't be return the right result.

log of ie6
-----------------------------------
"23/Apr/2014:12:05:22.485" "20(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:2955"
"23/Apr/2014:12:05:25.090" "20(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:2955"
"23/Apr/2014:12:05:51.064" "20(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:2955" ※1
"23/Apr/2014:12:08:59.668" "81(http-thread-pool-28292(4))" "qout" "null"
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:4045" ※2
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "qout" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "recv" "GET /ServletSession/jsp/ServletSessionOut.jsp HTTP/1.1"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "call"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "rtn"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "send" "200" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:4045"
"23/Apr/2014:12:09:11.727" "20(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:4045"
-----------------------------------

log of ie8
-----------------------------------
"23/Apr/2014:12:38:49.004" "20(Grizzly-kernel-thread(1))" "conn" "10.167.157.142:1580"
"23/Apr/2014:12:38:57.194" "20(Grizzly-kernel-thread(1))" "qin" "10.167.157.142:1580"
"23/Apr/2014:12:39:12.654" "20(Grizzly-kernel-thread(1))" "disc" "10.167.157.142:1580"
"23/Apr/2014:12:39:24.572" "107(http-thread-pool-28292(13))" "qout" "null"
-----------------------------------

Above all, the question is whether this phenomenon is a bug about ie or it is a glassfish internal issue?

I think the connection shouldn't be broken when the keepAlive-timeout happened between "qin" and "qout".

In addition, When the above phenomenon happened on glassfish 2.1.1, the following log will be print to
the server.log, the content of the log is same to the issue of GLASSFISH-20622

server.log
-----------------------------------
java.lang.NullPointerException
at org.apache.coyote.tomcat5.OutputBuffer.addSessionCookieWithJvmRoute(OutputBuffer.java:689)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:370)
at org.apache.coyote.tomcat5.OutputBuffer.close(OutputBuffer.java:336)
at org.apache.coyote.tomcat5.CoyoteResponse.finishResponse(CoyoteResponse.java:594)
at org.apache.coyote.tomcat5.CoyoteAdapter.afterService(CoyoteAdapter.java:354)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.postResponse(DefaultProcessorTask.java:625)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:612)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:889)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:267)
at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:118)
-----------------------------------



 Comments   
Comment by oleksiys [ 23/May/14 ]

When do you print "qin" log? I see it happens in the kernel thread, but where exactly?

Thank you.

Comment by houtang [ 06/Jun/14 ]

==============Here is the background===============
Below is the logic trace when use glassfish3.1.2.
Basicly it is similar with glassfish2.1, as two thread.
Firstly Thread <Grizzly-kernel-thread> built connection "conn"(marked with ★1), then it went to "qin"(marked with ★2),
after that another Thread <http-thread-pool> was woke up, and excuted as below sequence "qout"-> "recv"-> "send" (marked with ▲1,▲2,▲3).
Then after 15 seconds(keepalivetimeout) the connection was disconnected "disc"(marked with ★3)

glassfish3.1.2
-------------------------------------------------
#=com.sun.grizzly
$=com.sun.enterprise.v3.services.impl.monitor
@=com.sun.grizzly.util
*=com.sun.grizzly.http
&=com.sun.enterprise.v3.services.impl

<Grizzly-kernel-thread>
#SelectorHandlerRunner.doSelect(SelectorHandler, NIOContext) : 201
#SelectorHandlerRunner.handleSelectedKey(SelectionKey, SelectorHandler, NIOContext)行: 302
$MonitorableSelectorHandler.acceptWithoutRegistration(SelectionKey) ★1 conn
#SelectorHandlerRunner.handleSelectedKey(SelectionKey, SelectorHandler, NIOContext)行: 381
#NIOContext.execute(ContextTask)
@ SyncThreadPool.execute(Runnable) : 189 ★2 qin
#SelectorHandlerRunner.doSelect(SelectorHandler, NIOContext) : 212
#TCPSelectorHandler.postSelect(Context) 行: 534
#SelectorThreadKeyHandler.cancel(SelectionKey) : 84
#BaseSelectionKeyHandler).doAfterKeyCancel(SelectionKey) 行: 234 ★3 disc

<http-thread-pool>
HttpWorkerThread(Thread).run() 行: 722
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).run() 行: 520
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).doWork() : 536 ▲1 qout
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).doWork() : 542
#ProtocolChainContextTask(ContextTask).run() : 71
*HttpProtocolChain(DefaultProtocolChain).execute(Context) : 90
*DefaultProtocolFilter.execute(Context) : 231
*ProcessorTask.process(InputStream, OutputStream) : 1118
*ProcessorTask.doProcess() : 752
*ProcessorTask.doProcess() : 793
*ProcessorTask.postResponse() : 817
*ProcessorTask.finishResponse() : 827
&ContainerMapper.afterService(Request, Response) : 487
*ProcessorTask.action(ActionCode, Object) : 1156
*ProcessorTask.prepareResponse() : 1833 ▲3 send
-------------------------------------------------

when i was debuging these programs, i found that if the keepalivetimeout(15 seconds) is up it can "disc"(★3) directly after "qout"(▲1),
and it also can "disc"(★3) after“recv”(▲2)
==============Background ends===============

My first question,is it correct that after "qout" or "recv" the connection can still be disconnected as keepalivetimeout(15 sec) is up?

And when i was reproducing the sitiation , after "recv" then the connection was disconnected(keeptimeout is up), IE6 can build a second connection and finish the request
and return "200", the strange thing is when i use IE8, there is no second connection!! I was so confused, My second question is: what makes the IE6 and IE8 different?
BTW, i got the same consequence in both glassfish3.1.2 and glassfish2.1

Below the is trace.log details
1)IE6
trace.log

"15/May/2014:19:22:47.193" "42(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:1086" // build the first connection
"15/May/2014:19:22:47.194" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1086"
"15/May/2014:19:22:47.194" "78(http-thread-pool-28292(4))" "qout" "193.168.154.174:1086"
"15/May/2014:19:23:14.646" "78(http-thread-pool-28292(4))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:23:17.667" "42(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:1086" // after "recv" connection is disconnected
"15/May/2014:19:23:23.715" "42(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:1087" // build the second connection
"15/May/2014:19:23:23.716" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1087"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "qout" "193.168.154.174:1087"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "send" "301" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "78(http-thread-pool-28292(4))" "send" "301" "null" // the first connection return "null"
"15/May/2014:19:23:23.717" "42(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "74(http-thread-pool-28292(3))" "qout" "193.168.154.174:1087"
"15/May/2014:19:23:23.718" "74(http-thread-pool-28292(3))" "recv" "GET /HelloServlet/ HTTP/1.1"
"15/May/2014:19:23:23.718" "74(http-thread-pool-28292(3))" "call"
"15/May/2014:19:23:23.719" "74(http-thread-pool-28292(3))" "rtn"
"15/May/2014:19:23:23.719" "74(http-thread-pool-28292(3))" "send" "200" "193.168.154.174:1087" // the second connection sucesses
"15/May/2014:19:23:23.719" "42(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:1087"

2)IE8
trace.log

"15/May/2014:19:34:03.339" "42(Grizzly-kernel-thread(1))" "conn" "10.167.157.75:51986" // build the first connection
"15/May/2014:19:34:03.339" "42(Grizzly-kernel-thread(1))" "qin" "10.167.157.75:51986"
"15/May/2014:19:34:03.339" "340(http-thread-pool-28292(9))" "qout" "10.167.157.75:51986"
"15/May/2014:19:34:44.796" "340(http-thread-pool-28292(9))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:34:47.472" "42(Grizzly-kernel-thread(1))" "disc" "10.167.157.75:51986" // after "recv" connection is disconnected
"15/May/2014:19:34:50.884" "340(http-thread-pool-28292(9))" "send" "301" "null" // the first connection return "null" and there is no second connection!!!

Thank you

Comment by oleksiys [ 11/Jun/14 ]

keep-alive timeout should be applied only in cases, when the connection is idle (waiting for the next request to come).
And 15 seconds starts counting once the request/response is processed.
Are you sure it's keep-alive timeout we're talking about?

Thank you.

Comment by houtang [ 11/Jun/14 ]

Thank you for your reply!

I have a query needs your confirmation:
Basically one requests/response processing has below steps(1~9):
1.conn
2.qin
3.qout
4.recv
5.call
6.rtn
7.send
8.k-wt
9.disc

10.conn
11.qin
12.qout
13.recv
14.call
15.rtn
16.send
17.k-wt
18.disc

do you mean the 15 seconds starts counting from 8 and lasts from 8 to 9
or starts counting from 9 and lasts from 9 to 10("conn" of the second request)?
My opinion is the 15 seconds starts counting from 1, and lasts from 1 to 9(from the source logic),
do you agree?

In this case I have below situation: when the request is processing and it is not
completed(for example in step"recv"), at this time connection is disconnected.

conn(15sec starts counting)>qin>qout->recv->disc(15sec ends)

Hope you can get my point
Pls let me know if any misunderstanding

Thank you

Comment by oleksiys [ 11/Jun/14 ]

Hi,

just to be clear, we're talking about HTTP keep-alive, it means HTTP (TCP) connection is not getting closed between requests. Probably it's what you meant in your description, but I just wanted to make sure we're on the same page... so at the step 9. and 18. no real HTTP (TCP) connection termination happens.

Coming back to the steps you listed, confirm that keep-alive timeout is ticking between 9. and 10.

If you observe the timeout during read/write - it must be different timeout then.
First of all I'd recommend to switch to Glassfish 3.1.2.2 and check if the issue is still there.

Thank you.

Comment by houtang [ 16/Jun/14 ]

Hi oleksiys,

>Coming back to the steps you listed, confirm that keep-alive timeout is ticking between 9. and 10."
>If you observe the timeout during read/write - it must be different timeout then.
Ok, seems we are talking about different timeout

Here is part of source code from glassfish(Grizzly), the keep-alive timeout I was talking about is
"idleLimit" below in "com.sun.grizzly.http.SelectorThreadKeyHandler.java".

com.sun.grizzly.http.SelectorThreadKeyHandler.java
------------------------
public void expire(Iterator<SelectionKey> iterator) {
if (timeout == SelectionKeyAttachment.UNLIMITED_TIMEOUT)

{ return; }

......
if (expire >= 0) {
long idleLimit = getIdleLimit(attachment);

if (idleLimit >= 0 && currentTime - expire >= idleLimit &&
(!(attachment instanceof SelectionKeyAttachment) ||
((SelectionKeyAttachment) attachment).timedOut(key)))

{ //preventing further idle timeout detection for same key //due to we dont directly cancel key anymore we cant //rely in key.isvalid detection addExpirationStamp(key, SelectionKeyAttachment.UNLIMITED_TIMEOUT); cancel(key);●1 }

}
}
}
}

------------------------

I hava debugged several times and I am pretty sure this timeout is between read/write,
Can you try to debug from your side?

And I also listed part of the source code of "qin","qout","recv" and "send",marked with ★ below
Here is the debug steps:
1. Put the breakpoints at ●1,●2,★recv
2. When a request comes,<http-thread-pool> Thread will stop at ●2,wait for 30 seconds(default timeout), then <Grizzly-kernel-thread> Thread will come up, stop at●1
3. Keep on debuging <http-thread-pool> and after step over ★recv, go to <Grizzly-kernel-thread> and step over ●1
4. disconnect the debug

<Grizzly-kernel-thread>:
com.sun.grizzly.util.SyncThreadPool#execute(Runnable)
------------------------
public void execute(Runnable task)
{
......
if (((maxQueuedTasks < 0) || (workerQueueSize < maxQueuedTasks)) && (workQueue.offer(task)))

{ onTaskQueued(task); ★qin }

......
}
------------------------

com.sun.grizzly.BaseSelectionKeyHandler.java
------------------------
protected void doAfterKeyCancel(SelectionKey key)
{
try

{ if (selectorHandler != null) selectorHandler.closeChannel(key.channel()); else closeChannel(key.channel()); }

finally

{ notifyLocallyClose(key); ★disc Object attachment = key.attach(null); if ((attachment instanceof SelectionKeyAttachment)) ((SelectionKeyAttachment)attachment).release(key); }

}
------------------------

<http-thread-pool>:
com.sun.grizzly.util.AbstractThreadPool.java
------------------------
protected void doWork()
{
......
AbstractThreadPool.onTaskDequeued(r); ★qout
Throwable error = null;
try

{ beforeExecute(t_, r);       r.run(); ●2 onTaskCompletedEvent(r); ...... }

------------------------

com.sun.grizzly.http.ProcessorTask.java
------------------------
public boolean parseRequest()
throws Exception
{
try

{    inputBuffer.parseRequestLine(); ★recv }

catch (BufferOverflowException boe) {
if (logger.isLoggable(Level.SEVERE))

{ logger.log(Level.SEVERE, LogMessages.SEVERE_GRIZZLY_HTTP_PROCESSOR_TASK_REQUEST_URI_TOO_LARGE_ERROR(), boe); }

......
}

protected void prepareResponse()

{ ...... sendHeaders(); ★send }

------------------------

Thank you

Comment by houtang [ 08/Aug/14 ]

Hi oleksiys,

How are you doing?
It's been almost 2 months since your last response.
Is there any progress? Or you still have misunderstandings?
I'm glad to receive your reply!

Thanks





[GLASSFISH-19582] Move AccessLog support on Grizzly level instead of WebContainer Created: 24/Jan/13  Updated: 24/Jan/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0_b70
Fix Version/s: future release

Type: New Feature Priority: Major
Reporter: oleksiys Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
blocks GLASSFISH-18244 Web Service calls are not logged on s... Open

 Description   

Some HTTP based services (WebServices/EJB, appclient), which are using Grizzly HttpServer directly also require access log support.






[GLASSFISH-19444] Glassfish grizzly thread consuming large amounts of CPU time in NIO poll operation Created: 14/Dec/12  Updated: 16/Nov/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: jfowler2 Assignee: oleksiys
Resolution: Unresolved Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

JDK version 1.6.0_37, Windows Server 2008 R2 Enterprise


Attachments: Java Archive File grizzly-framework-1.9.50.jar    

 Description   

We recently noticed an issue where our Glassifish server, after running successfully for several hours, would suddenly peg one of the CPUs at 100%. Our application becomes unresponsive during this time. After restarting, the problem will eventually happen again (usually after several hours).

I ran this command to see what the threads were doing:

asadmin generate-jvm-report --type=thread

In the resulting output, one thread looked highly suspicious (consuming orders of magnitude more CPU time than any other thread):

Thread Execution Information:

Thread "Grizzly-kernel-thread(1)" thread-id: 27 thread-state: RUNNABLE Running in native
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255)
at: sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at: java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at: java.lang.Thread.run(Thread.java:662)

Thread Synchronization Statistics:

Number of times this thread was blocked (to enter/reenter a Monitor): 4,520

Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0

Total CPU time for this thread: 2,753 seconds 703,125,000 nanoseconds.

User-level CPU time for this thread: 2,753 seconds 703,125,000 nanoseconds.

Object Monitors currently held or requested by this thread: []

Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []



 Comments   
Comment by oleksiys [ 17/Dec/12 ]

attaching a patch, which enables selector spin workaround for all operation systems.
Can you pls. apply this patch on GF 3.1.2.2 (rename this file to grizzly-framework.jar and copy it over existing file w/ the same name in glassfishv3/glassfish/modules folder).
Pls. let me know if you still see this issue w/ the patch applied, if yes - pls. check and attach the GF server.log file.

Thanks.

Comment by Stefan_Chossy [ 18/Mar/13 ]

Hi jfowler2,

did that patch fix your problems? We are facing pretty much the same issue but the patch didn't help any.

Comment by badbob.nsk [ 15/Nov/13 ]

Have an issue with the same symptoms.
Does this patch works?

Comment by oleksiys [ 16/Nov/13 ]

is it Glassfish 3.1.2.2 on Windows?

Comment by badbob.nsk [ 16/Nov/13 ]

It is glassfish 3.1.2.2 on SLES11.





[GLASSFISH-17934] Support SNI (server name indication) Created: 08/Dec/11  Updated: 11/Mar/14

Status: Reopened
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: foal Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Support SNI (server name indication) to allows multiple SSL Certificates on a Single IP/Port



 Comments   
Comment by Ryan Lubke [ 28/Feb/13 ]

If SNI support is desired, I would recommend switching to JDK 7 [1].

[1] http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements-7.html

Comment by oleksiys [ 11/Mar/14 ]

implemented in Grizzly [1], we can think how to promote it to Glassfish.

[1] https://java.net/jira/browse/GRIZZLY-1661





[GLASSFISH-21238] Response code 400 on a DELETE request with a body Created: 21/Oct/14  Updated: 21/Oct/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: ganichev Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Same behavior on linux x64, windows 7 x64. jdk1.8.0_20



 Description   

When a browser sends a request with a method DELETE and some body, glassfish replies with the response code 400. No corresponding filters or a servlet are called.

When a DELETE request contains no body, then servlet's methods are called as usual.

Glassfish 4.0 has no such problem.

Th sample request is:

DELETE /struct/StructureElement/3751?_dc=1413865817878 HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Content-Length: 11
Origin: http://localhost:8080
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.101 Safari/537.36
Content-Type: application/json
Accept: */*
Referer: http://localhost:8080/struct-client/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,ru;q=0.6
Cookie: BPMSESSIONID=8aMBU61U7xn1RFSObP1slBhkvMCC; JSESSIONID=d116d19d27d9a381eef12b5acacd; treeForm_tree-hi=treeForm:tree:resources:JDBC:connectionPoolResources:ypool

{"id":3751}





[GLASSFISH-21107] When uploading a large file using an async servlet, after 30 seconds it throws an InterruptedByTimeoutException Created: 26/Jun/14  Updated: 10/Dec/14

Status: Open
Project: glassfish
Component/s: configuration, grizzly-kernel, web_container
Affects Version/s: 4.1_b07
Fix Version/s: None

Type: Bug Priority: Major
Reporter: raulgd Assignee: Masoud Kalali
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7 64 bit, Java JDK 1.7.0u60 64 bit Glassfish 4.0.1b7 nightly build downloaded on 2014-06-25



 Description   

I created an async servlet for users to upload files, that way the inputstream is buffered without having to load the whole file in memory, as there can be very big files uploaded.

When trying out to upload a 4.6GB file, after 30 seconds, the servlet enters the onError(Throwable t) method with the exception:

Severe:   java.nio.channels.InterruptedByTimeoutException
	at org.apache.catalina.connector.InputBuffer.disableReadHandler(InputBuffer.java:324)
	at org.apache.catalina.connector.Request.asyncTimeout(Request.java:4418)
	at org.apache.catalina.connector.Request.processTimeout(Request.java:4469)
	at org.apache.catalina.connector.Request.access$000(Request.java:156)
	at org.apache.catalina.connector.Request$6.onTimeout(Request.java:4300)
	at org.glassfish.grizzly.http.server.Response$SuspendTimeout.onTimeout(Response.java:2131)
	at org.glassfish.grizzly.http.server.Response$DelayQueueWorker.doWork(Response.java:2180)
	at org.glassfish.grizzly.http.server.Response$DelayQueueWorker.doWork(Response.java:2175)
	at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Warning:   Context path from ServletContext:  differs from path from bundle: /
WARN:   WELD-000715: HttpContextLifecycle guard not set. The Servlet container is not fully compliant.

Now, I already configured Glassfish for longer timeouts on both Configurations-> server-config -> Network Config -> Network Listeners -> http-listener-1 -> HTTP So that timeout: -1, Connection upload timeout: 3600000 (1 hour), Request Timeout: -1, Max Post Size: -1

And on Configurations-> server-config -> Network Config -> Transports -> tcp so that Read Timeout: 3600000 (1 hour), Write Timeout: 3600000 (1 hour)

Now, AFAIK, because it's an NIO timeout exception, just by changing the tcp read and write timeout to 1 hour should avoid the issue, but still, the exception happens exactly 30 seconds later, as if the timeout configurations are being ignored.

I saw previous issues on grizzly ignoring the read and write timeouts, but they where closed as fixed in mid. 2013 so it shouldn't be an issue now.

For reference, here's my domain.xml:

<domain log-root="${com.sun.aas.instanceRoot}/logs" application-root="${com.sun.aas.instanceRoot}/applications" version="7">
  <security-configurations>
    <authentication-service default="true" name="adminAuth" use-password-credential="true">
      <security-provider name="spcrealm" type="LoginModule" provider-name="adminSpc">
        <login-module-config name="adminSpecialLM" control-flag="sufficient" module-class="com.sun.enterprise.admin.util.AdminLoginModule">
          <property name="config" value="server-config"></property>
          <property name="auth-realm" value="admin-realm"></property>
        </login-module-config>
      </security-provider>
      <security-provider name="filerealm" type="LoginModule" provider-name="adminFile">
        <login-module-config name="adminFileLM" control-flag="sufficient" module-class="com.sun.enterprise.security.auth.login.FileLoginModule">
          <property name="config" value="server-config"></property>
          <property name="auth-realm" value="admin-realm"></property>
        </login-module-config>
      </security-provider>
    </authentication-service>
    <authorization-service default="true" name="authorizationService">
      <security-provider name="simpleAuthorization" type="Simple" provider-name="simpleAuthorizationProvider">
        <authorization-provider-config support-policy-deploy="false" name="simpleAuthorizationProviderConfig"></authorization-provider-config>
      </security-provider>
    </authorization-service>
  </security-configurations>
  <managed-job-config></managed-job-config>
  <system-applications>
    <application context-root="" location="${com.sun.aas.installRootURI}/lib/install/applications/__admingui" directory-deployed="true" name="__admingui" object-type="system-admin">
      <module name="__admingui">
        <engine sniffer="web"></engine>
        <engine sniffer="security"></engine>
      </module>
    </application>
  </system-applications>
  <resources>
    <jdbc-resource pool-name="SamplePool" jndi-name="jdbc/sample"></jdbc-resource>
    <jdbc-resource pool-name="__TimerPool" jndi-name="jdbc/__TimerPool" object-type="system-admin"></jdbc-resource>
    <jdbc-resource pool-name="DerbyPool" jndi-name="jdbc/__default" object-type="system-all"></jdbc-resource>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource" res-type="javax.sql.XADataSource" name="__TimerPool">
      <property name="databaseName" value="${com.sun.aas.instanceRoot}/lib/databases/ejbtimer"></property>
      <property name="connectionAttributes" value=";create=true"></property>
    </jdbc-connection-pool>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.ClientDataSource" is-isolation-level-guaranteed="false" res-type="javax.sql.DataSource" name="DerbyPool">
      <property name="PortNumber" value="1527"></property>
      <property name="Password" value="APP"></property>
      <property name="User" value="APP"></property>
      <property name="serverName" value="localhost"></property>
      <property name="DatabaseName" value="sun-appserv-samples"></property>
      <property name="connectionAttributes" value=";create=true"></property>
    </jdbc-connection-pool>
    <connector-connection-pool max-pool-size="250" steady-pool-size="1" name="jms/__defaultConnectionFactory-Connection-Pool" resource-adapter-name="jmsra" connection-definition-name="javax.jms.ConnectionFactory"></connector-connection-pool>
    <connector-resource pool-name="jms/__defaultConnectionFactory-Connection-Pool" jndi-name="jms/__defaultConnectionFactory" object-type="system-all-req"></connector-resource>
    <context-service jndi-name="concurrent/__defaultContextService" object-type="system-all"></context-service>
    <managed-executor-service jndi-name="concurrent/__defaultManagedExecutorService" object-type="system-all"></managed-executor-service>
    <managed-scheduled-executor-service jndi-name="concurrent/__defaultManagedScheduledExecutorService" object-type="system-all"></managed-scheduled-executor-service>
    <managed-thread-factory jndi-name="concurrent/__defaultManagedThreadFactory" object-type="system-all"></managed-thread-factory>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.ClientDataSource" res-type="javax.sql.DataSource" name="SamplePool">
      <property name="DatabaseName" value="sample"></property>
      <property name="User" value="app"></property>
      <property name="Password" value="app"></property>
      <property name="URL" value="jdbc:derby://localhost:1527/sample"></property>
      <property name="PortNumber" value="1527"></property>
      <property name="serverName" value="localhost"></property>
    </jdbc-connection-pool>
  </resources>
  <servers>
    <server name="server" config-ref="server-config">
      <application-ref ref="__admingui" virtual-servers="__asadmin"></application-ref>
      <application-ref ref="VegosPortal" virtual-servers="server"></application-ref>
      <resource-ref ref="jdbc/__TimerPool"></resource-ref>
      <resource-ref ref="jdbc/__default"></resource-ref>
      <resource-ref ref="jms/__defaultConnectionFactory"></resource-ref>
      <resource-ref ref="concurrent/__defaultContextService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedExecutorService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedScheduledExecutorService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedThreadFactory"></resource-ref>
      <resource-ref ref="jdbc/sample"></resource-ref>
    </server>
  </servers>
  <nodes>
    <node node-host="localhost" name="localhost-domain1" type="CONFIG" install-dir="${com.sun.aas.productRoot}"></node>
  </nodes>
  <configs>
    <config name="server-config">
      <system-property description="Port Number that JMS Service will listen for remote clients connection." name="JMS_PROVIDER_PORT" value="7676"></system-property>
      <http-service>
        <access-log></access-log>
        <virtual-server id="server" network-listeners="http-listener-1,http-listener-2"></virtual-server>
        <virtual-server id="__asadmin" network-listeners="admin-listener"></virtual-server>
      </http-service>
      <iiop-service>
        <orb use-thread-pool-ids="thread-pool-1"></orb>
        <iiop-listener id="orb-listener-1" port="3700" address="0.0.0.0" lazy-init="true"></iiop-listener>
        <iiop-listener id="SSL" port="3820" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </iiop-listener>
        <iiop-listener id="SSL_MUTUALAUTH" port="3920" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as" client-auth-enabled="true"></ssl>
        </iiop-listener>
      </iiop-service>
      <admin-service system-jmx-connector-name="system" type="das-and-server">
        <jmx-connector port="8686" address="0.0.0.0" auth-realm-name="admin-realm" name="system">
          <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </jmx-connector>
        <property name="adminConsoleContextRoot" value="/admin"></property>
        <property name="adminConsoleDownloadLocation" value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
        <property name="ipsRoot" value="${com.sun.aas.installRoot}/.."></property>
        <das-config></das-config>
      </admin-service>
      <connector-service></connector-service>
      <transaction-service tx-log-dir="${com.sun.aas.instanceRoot}/logs"></transaction-service>
      <batch-runtime-configuration></batch-runtime-configuration>
      <jms-service default-jms-host="default_JMS_host" type="EMBEDDED">
        <jms-host port="${JMS_PROVIDER_PORT}" host="localhost" name="default_JMS_host"></jms-host>
      </jms-service>
      <ejb-container>
        <ejb-timer-service></ejb-timer-service>
      </ejb-container>
      <web-container>
        <session-config>
          <session-manager>
            <manager-properties></manager-properties>
            <store-properties></store-properties>
          </session-manager>
          <session-properties></session-properties>
        </session-config>
      </web-container>
      <rest-config></rest-config>
      <cdi-service></cdi-service>
      <diagnostic-service></diagnostic-service>
      <security-service>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="admin-realm">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="file">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm" name="certificate"></auth-realm>
        <jacc-provider policy-provider="com.sun.enterprise.security.provider.PolicyWrapper" name="default" policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
          <property name="repository" value="${com.sun.aas.instanceRoot}/generated/policy"></property>
        </jacc-provider>
        <jacc-provider policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider" name="simple" policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
        <audit-module classname="com.sun.enterprise.security.ee.Audit" name="default">
          <property name="auditOn" value="false"></property>
        </audit-module>
        <message-security-config auth-layer="SOAP">
          <provider-config provider-type="client" provider-id="XWS_ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="client" provider-id="ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="XWS_ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
        </message-security-config>
        <message-security-config auth-layer="HttpServlet">
          <provider-config provider-type="server" provider-id="GFConsoleAuthModule" class-name="org.glassfish.admingui.common.security.AdminConsoleAuthModule">
            <request-policy auth-source="sender"></request-policy>
            <response-policy></response-policy>
            <property name="loginPage" value="/login.jsf"></property>
            <property name="loginErrorPage" value="/loginError.jsf"></property>
          </provider-config>
        </message-security-config>
        <property name="default-digest-algorithm" value="SHA-256"></property>
      </security-service>
      <java-config debug-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009" system-classpath="" classpath-suffix="">
        <jvm-options>-XX:MaxPermSize=256m</jvm-options>
        <jvm-options>-client</jvm-options>
        <jvm-options>-Djava.awt.headless=true</jvm-options>
        <jvm-options>-Djdk.corba.allowOutputStreamSubclass=true</jvm-options>
        <jvm-options>-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder</jvm-options>
        <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
        <jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
        <jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
        <jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
        <jvm-options>-Xmx1024m</jvm-options>
        <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
        <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
        <jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
        <jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
        <jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
        <jvm-options>-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.port=6666</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
        <jvm-options>-Dgosh.args=--nointeractive</jvm-options>
        <jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
        <jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
        <jvm-options>-Dfelix.fileinstall.log.level=2</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
        <jvm-options>-XX:NewRatio=2</jvm-options>
      </java-config>
      <network-config>
        <protocols>
          <protocol name="http-listener-1">
            <http request-timeout-seconds="-1" timeout-seconds="-1" connection-upload-timeout-millis="3600000" default-virtual-server="server" max-connections="1500">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="http-listener-2">
            <http default-virtual-server="server" max-connections="250">
              <file-cache></file-cache>
            </http>
            <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" ssl3-enabled="false" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-listener">
            <http default-virtual-server="__asadmin" max-connections="250" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="sec-admin-listener">
            <http default-virtual-server="__asadmin" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
            <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-http-redirect">
            <http-redirect secure="true"></http-redirect>
          </protocol>
          <protocol name="pu-protocol">
            <port-unification>
              <protocol-finder protocol="sec-admin-listener" name="http-finder" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder"></protocol-finder>
              <protocol-finder protocol="admin-http-redirect" name="admin-http-redirect" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder"></protocol-finder>
            </port-unification>
          </protocol>
        </protocols>
        <network-listeners>
          <network-listener port="80" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="8181" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="4848" protocol="pu-protocol" transport="tcp" name="admin-listener" thread-pool="admin-thread-pool"></network-listener>
        </network-listeners>
        <transports>
          <transport write-timeout-millis="3600000" byte-buffer-type="HEAP" read-timeout-millis="3600000" display-configuration="true" name="tcp"></transport>
        </transports>
      </network-config>
      <thread-pools>
        <thread-pool max-thread-pool-size="50" name="admin-thread-pool" max-queue-size="256"></thread-pool>
        <thread-pool name="http-thread-pool"></thread-pool>
        <thread-pool max-thread-pool-size="200" name="thread-pool-1"></thread-pool>
      </thread-pools>
      <monitoring-service>
        <module-monitoring-levels></module-monitoring-levels>
      </monitoring-service>
      <group-management-service>
        <failure-detection></failure-detection>
      </group-management-service>
      <availability-service></availability-service>
    </config>
    <config name="default-config">
      <http-service>
        <access-log></access-log>
        <virtual-server id="server" network-listeners="http-listener-1, http-listener-2">
          <property name="default-web-xml" value="${com.sun.aas.instanceRoot}/config/default-web.xml"></property>
        </virtual-server>
        <virtual-server id="__asadmin" network-listeners="admin-listener"></virtual-server>
      </http-service>
      <iiop-service>
        <orb use-thread-pool-ids="thread-pool-1"></orb>
        <iiop-listener id="orb-listener-1" port="${IIOP_LISTENER_PORT}" address="0.0.0.0"></iiop-listener>
        <iiop-listener id="SSL" port="${IIOP_SSL_LISTENER_PORT}" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </iiop-listener>
        <iiop-listener id="SSL_MUTUALAUTH" port="${IIOP_SSL_MUTUALAUTH_PORT}" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as" client-auth-enabled="true"></ssl>
        </iiop-listener>
      </iiop-service>
      <admin-service system-jmx-connector-name="system">
        <jmx-connector port="${JMX_SYSTEM_CONNECTOR_PORT}" address="0.0.0.0" auth-realm-name="admin-realm" name="system">
          <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="glassfish-instance"></ssl>
        </jmx-connector>
        <property name="adminConsoleDownloadLocation" value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
        <das-config></das-config>
      </admin-service>
      <web-container>
        <session-config>
          <session-manager>
            <manager-properties></manager-properties>
            <store-properties></store-properties>
          </session-manager>
          <session-properties></session-properties>
        </session-config>
      </web-container>
      <ejb-container>
        <ejb-timer-service></ejb-timer-service>
      </ejb-container>
      <mdb-container></mdb-container>
      <jms-service addresslist-behavior="priority" default-jms-host="default_JMS_host" type="EMBEDDED">
        <jms-host port="${JMS_PROVIDER_PORT}" host="localhost" name="default_JMS_host"></jms-host>
      </jms-service>
      <log-service log-rotation-limit-in-bytes="2000000" file="${com.sun.aas.instanceRoot}/logs/server.log">
        <module-log-levels></module-log-levels>
      </log-service>
      <security-service>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="admin-realm">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="file">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm" name="certificate"></auth-realm>
        <jacc-provider policy-provider="com.sun.enterprise.security.provider.PolicyWrapper" name="default" policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
          <property name="repository" value="${com.sun.aas.instanceRoot}/generated/policy"></property>
        </jacc-provider>
        <jacc-provider policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider" name="simple" policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
        <audit-module classname="com.sun.enterprise.security.ee.Audit" name="default">
          <property name="auditOn" value="false"></property>
        </audit-module>
        <message-security-config auth-layer="SOAP">
          <provider-config provider-type="client" provider-id="XWS_ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="client" provider-id="ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="XWS_ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
        </message-security-config>
      </security-service>
      <transaction-service tx-log-dir="${com.sun.aas.instanceRoot}/logs" automatic-recovery="true"></transaction-service>
      <diagnostic-service></diagnostic-service>
      <java-config debug-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${JAVA_DEBUGGER_PORT}" system-classpath="" classpath-suffix="">
        <jvm-options>-XX:MaxPermSize=256m</jvm-options>
        <jvm-options>-server</jvm-options>
        <jvm-options>-Djava.awt.headless=true</jvm-options>
        <jvm-options>-Djdk.corba.allowOutputStreamSubclass=true</jvm-options>
        <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
        <jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
        <jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
        <jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
        <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
        <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
        <jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
        <jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
        <jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
        <jvm-options>-XX:NewRatio=2</jvm-options>
        <jvm-options>-Xmx1024m</jvm-options>
        <jvm-options>-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.fileinstall</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.port=${OSGI_SHELL_TELNET_PORT}</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
        <jvm-options>-Dgosh.args=--noshutdown -c noop=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
        <jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
        <jvm-options>-Dfelix.fileinstall.log.level=3</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
      </java-config>
      <availability-service>
        <web-container-availability></web-container-availability>
        <ejb-container-availability sfsb-store-pool-name="jdbc/hastore"></ejb-container-availability>
        <jms-availability></jms-availability>
      </availability-service>
      <network-config>
        <protocols>
          <protocol name="http-listener-1">
            <http default-virtual-server="server">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="http-listener-2">
            <http default-virtual-server="server">
              <file-cache></file-cache>
            </http>
            <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" ssl3-enabled="false" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-listener">
            <http default-virtual-server="__asadmin" max-connections="250">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="sec-admin-listener">
            <http default-virtual-server="__asadmin" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
            <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="glassfish-instance"></ssl>
          </protocol>
          <protocol name="admin-http-redirect">
            <http-redirect secure="true"></http-redirect>
          </protocol>
          <protocol name="pu-protocol">
            <port-unification>
              <protocol-finder protocol="sec-admin-listener" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder" name="http-finder"></protocol-finder>
              <protocol-finder protocol="admin-http-redirect" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder" name="admin-http-redirect"></protocol-finder>
            </port-unification>
          </protocol>
        </protocols>
        <network-listeners>
          <network-listener port="${HTTP_LISTENER_PORT}" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="${HTTP_SSL_LISTENER_PORT}" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="${ASADMIN_LISTENER_PORT}" protocol="pu-protocol" transport="tcp" name="admin-listener" thread-pool="http-thread-pool"></network-listener>
        </network-listeners>
        <transports>
          <transport name="tcp"></transport>
        </transports>
      </network-config>
      <thread-pools>
        <thread-pool name="http-thread-pool"></thread-pool>
        <thread-pool max-thread-pool-size="200" name="thread-pool-1"></thread-pool>
        <thread-pool max-thread-pool-size="50" name="admin-thread-pool" max-queue-size="256"></thread-pool>
      </thread-pools>
      <group-management-service>
        <failure-detection></failure-detection>
      </group-management-service>
      <system-property description="Port Number that JMS Service will listen for remote clients connection." name="JMS_PROVIDER_PORT" value="27676"></system-property>
      <system-property name="ASADMIN_LISTENER_PORT" value="24848"></system-property>
      <system-property name="HTTP_LISTENER_PORT" value="28080"></system-property>
      <system-property name="HTTP_SSL_LISTENER_PORT" value="28181"></system-property>
      <system-property name="IIOP_LISTENER_PORT" value="23700"></system-property>
      <system-property name="IIOP_SSL_LISTENER_PORT" value="23820"></system-property>
      <system-property name="IIOP_SSL_MUTUALAUTH_PORT" value="23920"></system-property>
      <system-property name="JMX_SYSTEM_CONNECTOR_PORT" value="28686"></system-property>
      <system-property name="OSGI_SHELL_TELNET_PORT" value="26666"></system-property>
      <system-property name="JAVA_DEBUGGER_PORT" value="29009"></system-property>
      <monitoring-service>
        <module-monitoring-levels></module-monitoring-levels>
      </monitoring-service>
    </config>
  </configs>
  <property name="administrative.domain.name" value="domain1"></property>
  <secure-admin enabled="true" special-admin-indicator="8666d952-1f41-4ba9-a0c0-ec7396d974b7">
    <secure-admin-principal dn="CN=localhost,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
    <secure-admin-principal dn="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
  </secure-admin>
  <clusters></clusters>
  <applications>
    <application context-root="/vegos" location="file:/D:/NetbeansProjects/VegosPortal/target/VegosPortal-1.0/" name="VegosPortal" directory-deployed="true" object-type="user">
      <property name="archiveType" value="war"></property>
      <property name="appLocation" value="file:/D:/NetbeansProjects/VegosPortal/target/VegosPortal-1.0/"></property>
      <property name="org.glassfish.ejb.container.application_unique_id" value="91995825334714368"></property>
      <property name="defaultAppName" value="VegosPortal-1.0"></property>
      <module name="VegosPortal">
        <engine sniffer="ejb"></engine>
        <engine sniffer="security"></engine>
        <engine sniffer="weld"></engine>
        <engine sniffer="web"></engine>
      </module>
    </application>
  </applications>
</domain>

I tried the exact same code in WildFly 8.1.0 and works perfectly, so I know it's grizzly's NIO configuration.



 Comments   
Comment by raulgd [ 10/Dec/14 ]

This problem doesn't happen in Glassfish 4.1b13 with JDK 8u25 anymore.

Only there are 2 things to specify, first, the http-thread-pool timeout also has to be set to the same timeout as the tcp read timeout and connection upload timeout.

The other one, on the async servlet, right after initiating the async context with:

AsyncContext context = request.startAsync(); 

You have to set the timeout for the context with:

context.setTimeout(3600000);

Just replace the 3600000 with whatever timeout you set.

It would be ideal to just have the servlet get the timeout specified from the domain configuration, but at least is working now setting it by hand.





[GLASSFISH-21269] Cancelled POST via AJP triggers busy wait by http-listener-1 thread Created: 11/Dec/14  Updated: 16/Dec/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: fuzzyBSc2 Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04.4, Chrome browser with SPDY, apache 2.2.22-1ubuntu1.4, mod_jk 1:1.2.32-1, JAX-RS-based servlet with async-supported set to true



 Description   

A http worker thread goes into a busy conversation with Apache's mod_jk. Both the mod_jk thread and http-listener-1 pool thread are sitting at high CPU (together at around 100% for my single CPU test setup) and the listener appears never to return to the thread pool.

The conversation is as follows (captured using strace on the high CPU glassfish thread):
glassfish -> mod_jk: 0x41 0x42 0x00 0x03 0x06 0x1f 0xfa (Send me up to 8186 bytes of the body)
mod_jk -> glassfish: 0x12 0x34 0x00 0x00 (end of body)
This sequence of messages is repeated very very rapidly, suggesting that the glassfish side is not handling the end of body message from mod_jk.

The request is to an asynchronous JAX-RS method via the @Suspended annotation on an AsyncResponse parameter. The body of the message is JSON and is being deserialised in the container into another method parameter.

I think the following is occurring:

  • The browser sends a post request with a (in this case) 39260 byte body, but cancels it. This closes the spdy stream and causes mod_spdy to report an end of file to mod_jk internally within apache
  • Meanwhile glassfish begins to process the request. It has not read the whole request at this time. It invokes jackson to read the request which in turn reads from the InputStream that AjpHandlerFilter provides (see AjpHandlerFilter.handleRead in the stack trace).
  • The filter chain then appears to mishandle the AJP end of file message and instead continues to request more data. I didn't see AjpHandlerFilter in the read stack trace I captured so maybe it isn't intercepting correctly.

Restarting either apache or glassfish resolves the problem. This bug may be related to https://java.net/jira/browse/GLASSFISH-21202 which seems to have a similar trigger but results in memory growth rather than the high cpu and thread exhaustion I am seeing.

I have included stack traces for the write and read phases of the conversation as captured by jstack. I have also included strace output for the beginning of the problem, and to show how it recovers once Apache is restarted:
Thread 12324: (state = IN_NATIVE)

  • sun.nio.ch.FileDispatcherImpl.write0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
  • sun.nio.ch.SocketDispatcher.write(java.io.FileDescriptor, long, int) @bci=4, line=47 (Compiled frame)
  • sun.nio.ch.IOUtil.writeFromNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=93 (Compiled frame)
  • sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=12, line=51 (Compiled frame)
  • sun.nio.ch.SocketChannelImpl.write(java.nio.ByteBuffer) @bci=206, line=487 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(java.nio.channels.SocketChannel, java.nio.ByteBuffer) @bci=2, line=149 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeSimpleBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection, org.glassfish.grizzly.Buffer) @bci=130, line=133 (Compil
    ed frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.write(org.glassfish.grizzly.nio.transport.TCPNIOConnection, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.
    grizzly.WriteResult) @bci=53, line=680 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTemporarySelectorWriter.writeNow0(org.glassfish.grizzly.nio.NIOConnection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.WriteResult) @bci=14, line=65 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write0(org.glassfish.grizzly.nio.NIOConnection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.WriteResult, long, java.util.concurrent.TimeUnit) @bci=50, line=202 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.PushBackHandler, long, java.util.concurrent.TimeUnit) @bci=71, line=153 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.MessageCloner) @bci=19, line=78 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.lang.Object, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.MessageCloner) @bci=11, line=58 (Compiled frame)
  • org.glassfish.grizzly.AbstractWriter.write(org.glassfish.grizzly.Connection, java.lang.Object, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler) @bci=10, line=105 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=104, line=129 (Compiled frame)
  • org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=20, line=191 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=111 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Compiled frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.write(java.lang.Object, java.lang.Object, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.PushBackHandler, org.glassfish.grizzly.asyncqueue.MessageCloner, boolean) @bci=135, line=848 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.write(java.lang.Object) @bci=13, line=702 (Compiled frame)
  • org.glassfish.grizzly.http.ajp.AjpHandlerFilter.sendMoreDataRequestIfNeeded(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=114, line=504 (Compiled frame)
  • org.glassfish.grizzly.http.ajp.AjpHandlerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=18, line=197 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.read(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=83, line=351 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.read() @bci=65, line=695 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.blockingRead() @bci=4, line=1119 (Compiled frame)
  • org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead() @bci=18, line=95 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.fill(int) @bci=23, line=1143 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.read(byte[], int, int) @bci=74, line=353 (Interpreted frame)
  • org.apache.catalina.connector.InputBuffer.read(byte[], int, int) @bci=33, line=267 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteInputStream.read(byte[], int, int) @bci=97, line=270 (Interpreted frame)
  • org.glassfish.jersey.message.internal.EntityInputStream.read(byte[], int, int) @bci=7, line=101 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore() @bci=48, line=178 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(int[], int, int, int, int) @bci=268, line=1749 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName() @bci=64, line=1654 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(int) @bci=78, line=1499 (Compiled frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken() @bci=255, line=700 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.util.Map) @bci=133, line=416 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=143, line=312 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=3, line=26 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=59, line=525 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.lang.Object) @bci=5, line=106 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, com.fasterxml.jackson.core.JsonToken) @bci=53, line=242 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=26, line=118 (Compiled frame)
  • com.fasterxml.jackson.databind.ObjectReader._bind(com.fasterxml.jackson.core.JsonParser, java.lang.Object) @bci=128, line=1233 (Interpreted frame)
  • com.fasterxml.jackson.databind.ObjectReader.readValue(com.fasterxml.jackson.core.JsonParser) @bci=6, line=677 (Interpreted frame)
  • com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, java.io.InputStream) @bci=204, line=777 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(javax.ws.rs.ext.ReaderInterceptorContext, javax.ws.rs.ext.MessageBodyReader, org.glassfish.jersey.message.internal.EntityInputStream) @bci=61, line=251 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=292, line=229 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=1, line=73 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, org.glassfish.jersey.internal.PropertiesDelegate, java.io.InputStream, java.lang.Iterable, boolean) @bci=48, line=1124 (Interpreted frame)
  • org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], org.glassfish.jersey.internal.PropertiesDelegate) @bci=116, line=851 (Interpreted frame)
  • org.glassfish.jersey.server.ContainerRequest.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[]) @bci=8, line=270 (Interpreted frame)
  • org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide() @bci=60, line=96 (Interpreted frame)
  • org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(java.util.List) @bci=46, line=81 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues() @bci=4, line=121 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(java.lang.Object, javax.ws.rs.core.Request) @bci=3, line=136 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(java.lang.Object, org.glassfish.jersey.server.ContainerRequest) @bci=5, line=104 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(org.glassfish.jersey.server.internal.process.RequestProcessingContext, java.lang.Object) @bci=28, line=387 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(org.glassfish.jersey.server.internal.process.RequestProcessingContext) @bci=97, line=331 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(java.lang.Object) @bci=5, line=103 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime$1.run() @bci=57, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=4, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=1, line=267 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.util.concurrent.Callable, boolean) @bci=36, line=315 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(org.glassfish.jersey.internal.util.Producer, boolean) @bci=2, line=297 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.lang.Runnable) @bci=9, line=267 (Interpreted frame)
  • org.glassfish.jersey.process.internal.RequestScope.runInScope(org.glassfish.jersey.process.internal.RequestScope$Instance, java.lang.Runnable) @bci=23, line=297 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime.process(org.glassfish.jersey.server.ContainerRequest) @bci=164, line=254 (Interpreted frame)
  • org.glassfish.jersey.server.ApplicationHandler.handle(org.glassfish.jersey.server.ContainerRequest) @bci=5, line=1028 (Interpreted frame)
  • org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=119, line=372 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=381 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=362, line=344 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=39, line=221 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapper.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.Servlet) @bci=122, line=1682 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=694, line=318 (Interpreted frame)
  • org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=74, line=160 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.doInvoke(org.apache.catalina.Request, org.apache.catalina.Response, boolean) @bci=168, line=734 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=4, line=673 (Interpreted frame)
  • com.sun.enterprise.web.WebPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=99, line=99 (Interpreted frame)
  • org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=44, line=174 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.doService(org.glassfish.grizzly.http.server.Request, org.apache.catalina.connector.Request, org.glassfish.grizzly.http.server.Response, org.apache.catalina.connector.Response, boolean) @bci=425, line=415 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=183, line=282 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call() @bci=12, line=459 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=15, line=167 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.runService(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=48, line=201 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.doHandle(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=131, line=175 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=348, line=235 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Interpreted frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Interpreted frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=69, line=561 (Interpreted frame)
  • org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener, java.util.logging.Logger) @bci=9, line=112 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=6, line=117 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=3, line=56 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run() @bci=12, line=137 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() @bci=47, line=565 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() @bci=9, line=545 (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

The read phase of the cycle looks like this:
Thread 12324: (state = IN_NATIVE)

  • sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
  • sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=39 (Compiled frame)
  • sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223 (Compiled frame)
  • sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=29, line=192 (Compiled frame)
  • sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.readSimpleByteBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection, java.nio.ByteBuffer) @bci=10, line=344 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.allocateAndReadBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection) @bci=55, line=237 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer) @bci=22, line=618 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTemporarySelectorReader.readNow0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.ReadResult) @bci=25, line=65 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.ReadResult, org.glassfish.grizzly.Buffer, long, java.util.concurrent.TimeUnit) @bci=39, line=171 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.Interceptor, org.glassfish.grizzly.ReadResult, org.glassfish.grizzly.Buffer, long, java.util.concurrent.TimeUnit) @bci=20, line=141 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.Interceptor, long, java.util.concurrent.TimeUnit) @bci=52, line=113 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.Interceptor) @bci=18, line=75 (Compiled frame)
  • org.glassfish.grizzly.AbstractReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer) @bci=12, line=72 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=57, line=77 (Compiled frame)
  • org.glassfish.grizzly.filterchain.TransportFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=20, line=173 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.read(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=83, line=351 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.read() @bci=65, line=695 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.blockingRead() @bci=4, line=1119 (Compiled frame)
  • org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead() @bci=18, line=95 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.fill(int) @bci=23, line=1143 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.read(byte[], int, int) @bci=74, line=353 (Interpreted frame)
  • org.apache.catalina.connector.InputBuffer.read(byte[], int, int) @bci=33, line=267 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteInputStream.read(byte[], int, int) @bci=97, line=270 (Interpreted frame)
  • org.glassfish.jersey.message.internal.EntityInputStream.read(byte[], int, int) @bci=7, line=101 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore() @bci=48, line=178 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(int[], int, int, int, int) @bci=268, line=1749 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName() @bci=64, line=1654 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(int) @bci=78, line=1499 (Compiled frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken() @bci=255, line=700 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.util.Map) @bci=133, line=416 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=143, line=312 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=3, line=26 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=59, line=525 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.lan
    g.Object) @bci=5, line=106 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, com.fasterxml.jackson.core.JsonToken) @bci=53, line=242 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=26, line=118 (Compiled frame)
  • com.fasterxml.jackson.databind.ObjectReader._bind(com.fasterxml.jackson.core.JsonParser, java.lang.Object) @bci=128, line=1233 (Interpreted frame)
  • com.fasterxml.jackson.databind.ObjectReader.readValue(com.fasterxml.jackson.core.JsonParser) @bci=6, line=677 (Interpreted frame)
  • com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, java.io.InputStream) @bci=204, line=777 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(javax.ws.rs.ext.ReaderInterceptorContext, javax.ws.rs.ext.MessageBodyReader, org.glassfish.jersey.message.internal.EntityInputStream) @bci=61, line=251 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=292, line=229 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=1, line=73 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, org.glassfish.jersey.internal.PropertiesDelegate, java.io.InputStream, java.lang.Iterable, boolean) @bci=48, line=1124 (Interpreted frame)
  • org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], org.glassfish.jersey.internal.PropertiesDelegate) @bci=116, line=851 (Interpreted frame)
  • org.glassfish.jersey.server.ContainerRequest.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[]) @bci=8, line=270 (Interpreted frame)
  • org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide() @bci=60, line=96 (Interpreted frame)
  • org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(java.util.List) @bci=46, line=81 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues() @bci=4, line=121 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(java.lang.Object, javax.ws.rs.core.Request) @bci=3, line=136 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(java.lang.Object, org.glassfish.jersey.server.ContainerRequest) @bci=5, line=104 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(org.glassfish.jersey.server.internal.process.RequestProcessingContext, java.lang.Object) @bci=28, line=387 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(org.glassfish.jersey.server.internal.process.RequestProcessingContext) @bci=97, line=331 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(java.lang.Object) @bci=5, line=103 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime$1.run() @bci=57, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=4, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=1, line=267 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.util.concurrent.Callable, boolean) @bci=36, line=315 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(org.glassfish.jersey.internal.util.Producer, boolean) @bci=2, line=297 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.lang.Runnable) @bci=9, line=267 (Interpreted frame)
  • org.glassfish.jersey.process.internal.RequestScope.runInScope(org.glassfish.jersey.process.internal.RequestScope$Instance, java.lang.Runnable) @bci=23, line=297 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime.process(org.glassfish.jersey.server.ContainerRequest) @bci=164, line=254 (Interpreted frame)
  • org.glassfish.jersey.server.ApplicationHandler.handle(org.glassfish.jersey.server.ContainerRequest) @bci=5, line=1028 (Interpreted frame)
  • org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=119, line=372 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=381 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=362, line=344 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=39, line=221 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapper.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.Servlet) @bci=122, line=1682 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=694, line=318 (Interpreted frame)
  • org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=74, line=160 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.doInvoke(org.apache.catalina.Request, org.apache.catalina.Response, boolean) @bci=168, line=734 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=4, line=673 (Interpreted frame)
  • com.sun.enterprise.web.WebPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=99, line=99 (Interpreted frame)
  • org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=44, line=174 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.doService(org.glassfish.grizzly.http.server.Request, org.apache.catalina.connector.Request, org.glassfish.grizzly.http.server.Response, org.apache.catalina.connector.Response, boolean) @bci=425, line=415 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=183, line=282 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call() @bci=12, line=459 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=15, line=167 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.runService(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=48, line=201 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.doHandle(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=131, line=175 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=348, line=235 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Interpreted frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Interpreted frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=69, line=561 (Interpreted frame)
  • org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener, java.util.logging.Logger) @bci=9, line=112 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=6, line=117 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=3, line=56 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run() @bci=12, line=137 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() @bci=47, line=565 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() @bci=9, line=545 (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

strace output on the glassfish http-listener-1 thread:
futex(0x7fc2b8219454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2b8219450,

{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fc2b8219428, FUTEX_WAKE_PRIVATE, 1) = 1
read(480, "\0224\4a\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 8027
read(484, "\0224\4[\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 9311
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
.... many more repeats until I restart apache ...
futex(0x7fc2b8219454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2b8219450, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}

) = 1
futex(0x7fc2b8219428, FUTEX_WAKE_PRIVATE, 1) = 1
read(480, "\0224\4a\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 8027
read(484, "\0224\4[\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 9311
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4



 Comments   
Comment by fuzzyBSc2 [ 11/Dec/14 ]

Oops, sorry. The recovery portion of the strace when apache is restarted should be this:
read(484, 0x7fc261e29ff0, 242337) = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(488, EPOLL_CTL_ADD, 484, {EPOLLIN, {u32=484, u64=9290625383554613732}}) = 0
epoll_ctl(488, EPOLL_CTL_MOD, 484, {EPOLLIN, {u32=484, u64=9290625383554613732}}) = 0
epoll_wait(488, {{EPOLLIN|EPOLLERR|EPOLLHUP,

{u32=484, u64=9290625383554613732}

}}, 4096, 30000) = 1
read(484, 0x7fc261e29ff0, 242337) = -1 ECONNRESET (Connection reset by peer)
write(441, "\1", 1) = 1
epoll_ctl(488, EPOLL_CTL_DEL, 484, {0, {u32=484, u64=484}}) = -1 ENOENT (No such file or directory)
close(484) = 0
epoll_wait(488, {}, 4096, 0) = 0
write(2, "[#|2014-12-09T15:42:00.719+1000|"..., 8192) = -1 EPIPE (Broken pipe)
— SIGPIPE (Broken pipe) @ 0 (0) —
write(2, "(DefaultFilterChain.java:133)\n\ta"..., 955) = -1 EPIPE (Broken pipe)
— SIGPIPE (Broken pipe) @ 0 (0) —

Comment by fuzzyBSc2 [ 11/Dec/14 ]

Note: I have tried applying the patches from GLASSFISH-21202 to see if they resolve this problem, but currently they appear to have been prepared for glassfish 4.0 only, not 4.1.

Comment by oleksiys [ 12/Dec/14 ]

pls. try this patch
https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/glassfishv41-patch.zip

Comment by fuzzyBSc2 [ 15/Dec/14 ]

Thanks,

I have applied this patch and retested. I have not been able to reproduce the problem with this patch in place

Comment by fuzzyBSc2 [ 16/Dec/14 ]

If you don't mind, just a procedural question: Is there any way I could trace this binary patch to a change in the source tree? Part of the open source policy of the company I work for is that I need to show how I would build the software from source if required.

Comment by oleksiys [ 16/Dec/14 ]

)

you can track it on github
https://github.com/GrizzlyNIO/grizzly-mirror/tree/glassfish41





[GLASSFISH-7109] [JDK ISSUE] GlassFish performance degradation caused by invoking setSoLinger/setReuseAddress Created: 30/Jan/09  Updated: 06/Mar/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: v2.1
Fix Version/s: not determined

Type: Bug Priority: Minor
Reporter: jfarcand Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All
URL: http://bugs.sun.com/view_bug.do?bug_id=6799574


Issuezilla Id: 7,109
Status Whiteboard:

V2.1.1_exclude


 Description   

[Umbrella bug for JDK 5/6/7 issue 6799574)

Grizzly is suffering performance degradation when setSoLinger and setReuseAddess
starts throwing the following exception:

[#|2009-01-26T00:33:56.325-0800|WARNING|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=SelectorReaderThread-8084;_RequestID=11ae0030-c392-4217-8408-cfa7efe0a879;|setSoLinger
exception
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:261)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:166)
at sun.nio.ch.SocketAdaptor.setIntOption(SocketAdaptor.java:296)
at sun.nio.ch.SocketAdaptor.setSoLinger(SocketAdaptor.java:331)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketOptions(SelectorThread.java:1893)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.registerNewChannels(SelectorReadThread.java:93)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.startEndpoint(SelectorReadThread.java:121)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)

#]

[#|2009-01-26T00:33:56.327-0800|WARNING|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=SelectorReaderThread-8084;_RequestID=11ae0030-c392-4217-8408-cfa7efe0a879;|setReuseAddress
exception
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:261)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:166)
at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:286)
at sun.nio.ch.SocketAdaptor.setReuseAddress(SocketAdaptor.java:399)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketOptions(SelectorThread.java:1910)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.registerNewChannels(SelectorReadThread.java:93)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.startEndpoint(SelectorReadThread.java:121)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)

#]

This has been discussed here:

https://glassfish.dev.java.net/servlets/ReadMsg?listName=users&msgNo=26597

One user reported: ..that these errors are harmless, however, as you can see
below, the throughput of my application was reduced by 50% in the minutes
surrounding the spurt of errors:

minute requests setSoLinger/setReuseAddress exceptions
----- ----- -
14:23 7620 0
14:24 10063 0
14:25 9714 0
14:26 8847 0
14:28 7370 0
14:29 9787 0
14:30 9104 0
14:31 8171 0
14:32 4066 15 errors in two groups: 8 @ 14:32:33 and 7 @ 14:32:58
14:33 6908 0
14:34 10463 0
14:35 9870 0
14:36 8236 0
14:37 8685 0
14:38 8098 0

My application constantly serves 200-350 requests per second - and has been
doing so for 1 week now. This is the only incident that any errors have been
thrown since the application began operation.



 Comments   
Comment by jfarcand [ 30/Jan/09 ]

Trying to find a workaround. Disabling setSoLinger (setting it to false) cause
some performance issue, but at least the exception is not reproducible.

Comment by jfarcand [ 15/Apr/09 ]

JDK issue

http://monaco.sfbay.sun.com/detail.jsf?cr=6799574

Re-assign to docs.

Comment by Paul Davies [ 13/May/09 ]

Description of this issue added to the Release Notes. Review pending.

Comment by kumara [ 01/Sep/09 ]

Changing version from 9.1.1 to v2.1 to reflect new name/version.

Comment by Paul Davies [ 30/Sep/09 ]

To be added to the v3 Release Notes

Comment by jagadesh [ 15/Oct/09 ]

Will not be fixed for V2.1.1

Comment by Paul Davies [ 22/Oct/09 ]

Reassigned to grisdal.

Comment by Gail Risdal [ 08/Dec/09 ]

Added to v3 Release Notes.

Comment by Gail Risdal [ 09/Dec/09 ]

Closed prematurely. Reopened to enable code fix.

Comment by Gail Risdal [ 09/Dec/09 ]

Reassigned to enable code fix.

Comment by Paul Davies [ 26/May/11 ]

This issue was noted as required in the Release Notes. I have changed the subcomponent to enable engineering to resolve or close this bug as necessary.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.





[GLASSFISH-15581] ssl2-enabled attribute on the ssl config element should be deprecated or removed Created: 14/Jan/11  Updated: 02/Dec/11

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: 4.0

Type: Bug Priority: Minor
Reporter: Ryan Lubke Assignee: Ryan Lubke
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_x-exclude

 Description   

Currently the ssl element in the domain.xml exposes an attribute called ssl2-enabled. This attribute has no real impact on the runtime (at least with the Sun and Apple JDKs) as ssl2 isn't supported by JSSE.

This has two main points of impact:
1) grizzly-config
2) documentation

We need to come to a final decision on deprecation or removal and take the approriate actions on each item.






[GLASSFISH-15803] GUI console and instance are not available after http-listener-1.port is changed into 4848 Created: 02/Feb/11  Updated: 02/Dec/11

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1_b38
Fix Version/s: 4.0

Type: Bug Priority: Minor
Reporter: tetyanac Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows



 Description   

I changed setting configs.config.server-config.network-config.network-listeners.network-listener.http-listener-1.port=4848 via GUI admin console and after that GUI console is not available and GF instance as well. Why is it possible to change the port to already occupied with process one?



 Comments   
Comment by Anissa Lam [ 02/Feb/11 ]

The enforcement of not assigning conflicting port # will need to happen at the backend.
Assigning to 'admin'.

Comment by Tom Mueller [ 18/Feb/11 ]

Validation of network listener config changes happens within Grizzly.
Assigning to the grizzly category.

Comment by oleksiys [ 18/Feb/11 ]

reassigning to Justin.





[GLASSFISH-20955] Building grizzly-config with JDK 1.8 fails Created: 15/Jan/14  Updated: 19/Sep/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Ryan Lubke Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: jdk8

 Description   

Compiling nucleus/grizzly/config using JDK 1.8.0b121 fails:

Failed to execute goal org.glassfish.build:command-security-maven-plugin:1.0.6:check (default-check) on project grizzly-config: Error loading config beans: java.lang.RuntimeException: org.apache.maven.plugin.MojoExecutionException: Error analyzing java/lang/Object: IllegalArgumentException -> [Help 1]






[GLASSFISH-19372] ClientAbort Exception Default Servlet Created: 27/Nov/12  Updated: 27/Nov/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b05
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: peppeme Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux centos


Tags: brokenpipe, clientabortexception, grizzly

 Description   

I can't reproduce the issue but I get many and many Client Abort Exception:
It seems a client side communication interruption but
The server.log is filled by these becoming unreadable..

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:430)
at com.sun.grizzly.util.buf.ByteChunk.flushBuffer(ByteChunk.java:458)
at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:380)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:455)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:442)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:160)
at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:2010)
at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1040)
at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:466)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at filter.AdwFilter.doFilter(AdwFilter.java:206)
...
...
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
at






[GLASSFISH-18257] On URI decode exception the access log is not used Created: 26/Jan/12  Updated: 26/Jan/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1_b43
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: benjamin_m Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux x86_64


Tags: grizzly, logging

 Description   

When Grizzly throws an "Invalid URI character encoding" exception, the URI is part of the stack trace but the HTTP request info isn't saved on the access log.
This is a problem if the request URI makes it obvious that the requester is trying an exploit/vulnerability.
Without the access log used, there is no way of seeing the IP/hostname of the requester to identify the source of this attack attempt.



 Comments   
Comment by oleksiys [ 26/Jan/12 ]

Can you pls. check if it's still the case in the latest promoted Glassfish 3.1.2
http://dlc.sun.com.edgesuite.net/glassfish/3.1.2/promoted/glassfish-3.1.2-b19.zip

Thanks.

Comment by benjamin_m [ 26/Jan/12 ]

After trying to replicate in a VM with the suggested build, a similar error is not thrown.
To specify, here is the stack trace of the URI decoding issue which is not being logged in the access log.

[#|2012-01-26T07:50:40.472+0100|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=23;_ThreadName=Thread-1;|Internal Server error: /../../../../../../../../boot.ini
java.io.IOException: Invalid URI character encoding
at com.sun.grizzly.util.http.HttpRequestURIDecoder.decode(HttpRequestURIDecoder.java:101)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:185)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)

Comment by oleksiys [ 26/Jan/12 ]

Just to make sure, you didn't see the record in the access.log related to the corrupted request?

Comment by benjamin_m [ 26/Jan/12 ]

Exactly.
Because the exception is thrown on URI decode, Grizzly gives up there and nothing is written to the access log.
Which becomes very problematic when you have a rogue client trying some exploit like it's the case here.

Comment by oleksiys [ 26/Jan/12 ]

I agree, just wanted to make sure this is true for the latest 3.1.2 build.





Generated at Tue Mar 03 23:26:41 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.