[GLASSFISH-21005] GlassFish stops responding to http connections if client aborts downloads (grizzly or http service i suppose) Created: 08/Mar/14  Updated: 13/Mar/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Bug Priority: Blocker
Reporter: pranahata Assignee: oleksiys
Resolution: Unresolved Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CenOs 64 bit, Java 7 u 51



 Description   

To reproduce the issue

Create a Serlvet that serves a file.

From a different box, do this a few hundred times.

HttpURLConnection connection = null;

URL url = new URL("http://lala:8080/myapp/myfileservingservlet");
connection = (HttpURLConnection)url.openConnection();

connection.setDoInput(true);
connection.setRequestMethod("GET");
InputStream is = connection.getInputStream();
connection.getHeaderField("Content-Type);
connection.disconnect();

GlassFish logs shows a fair few of these:

Caused by: java.io.IOException: Connection closed
at org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:298) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:613) ~[nucleus-grizzly-all.jar:na]

And also a fair few of these

java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.flushByteBuffer(TCPNIOTransport.java:1252)

Trying to get anything else from the server shows:

Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

Note that it is easier to reproduce if the request goes over the internet but doesn't happen so frequently if using localhost, no sure about another box in the local network.

I have also seen the same errors in the logs when launching jws applications via JnlpDownloadServlet. To my undertanding the jws container does a similar thing by requesting the http headers of a jar but then aborting the download if the jar is in the jws cache and hasn't been modified.

Setting as blocker as it can make the server unusable and there one can not control how http clients interact with the server.



 Comments   
Comment by oleksiys [ 12/Mar/14 ]

Can you please try this patch [1] and check if it's still reproducible?

Thanks.

[1] https://dl.dropboxusercontent.com/u/7319744/nucleus-grizzly-all.jar

Comment by pranahata [ 13/Mar/14 ]

Olexsys,

I am quite busy at the moment, the test case is pretty simple. Where you able to reproduce the error without the patch?





[GLASSFISH-18803] no response for a chunked package Created: 13/Jun/12  Updated: 14/Dec/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b23
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: yavuzs Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 1 day
Time Spent: Not Specified
Original Estimate: 1 day
Environment:

SunOs, x86, 64bit


Attachments: File seac808012.cap    
Tags: http, http-listener

 Description   

there is a server that sends us chunked requests (by POST), our application on glassfish takes these packages and process them. but somehow, some of the packages are not processed by the application, also there is no log activity about these packages. but when we capture the network activity we can see that the request has come to the server.
we tried "-Dcom.sun.enterprise.web.connector.grizzly.enableSnoop=true" parameter. with this parameter, for the lost request there is no log.
here is a lost request:
Host: 192.168.100.10:49205
Transfer-Encoding: chunked

185
<?xml version='1.0' encoding='UTF-8'?>
<env:Envelope xmlns:env='http://www.w3.org/2003/05/soap-envelope' xmlns:seadac='http://www.schange.com/2004/SeaDAC/triggers'>
<env:Header><seadac:transaction transaction-id='

{20321632-B758-442a-ADFE-6E4607343AE3}

' origination-time='2012-06-12T08:29:04Z' env:role='http://www.w3.org/2003/05/soap-envelope/ultimateReceiver'/></env:Header>
<env:Body>
80
<seadac:subscription-notification subscription-id='

{1942B6B8-158F-4279-B9E9-0429671BEC67}

' current-request='3' next-request='4'>
c7
<seadac:hierarchy-changed hierarchy-uid='539401'><seadac:node-added hierarchy-uid='539710' asset-uid='528673'/></seadac:hierarchy-changed></seadac:subscription-notification></env:Body></env:Envelope>
0

you can find the capture attached.
frame 7824 and 7825 are the same request that the server sends to all subscribed clients. for this example there are two client subscribed. the gf application and the tomcat application. as you see, at the frame 7824 tomcat responded the request but at the frame 7825 fg did not respond the request.
regards,



 Comments   
Comment by yavuzs [ 19/Jun/12 ]

any response?

Comment by oleksiys [ 14/Dec/12 ]

sorry for the late response.
is it still the case w/ Glassfish 3.1.2.2?

thanks.

Comment by yavuzs [ 14/Dec/12 ]

the version is : GlassFish Server Open Source Edition 3.1.2 (build 23)

Comment by oleksiys [ 14/Dec/12 ]

yes, I understand, but there is a newer version (patch release) where this issue might have been fixed.
are you able to reproduce the issue consistently, or it happens sporadically?

Comment by yavuzs [ 14/Dec/12 ]

it happens sporadically. we didn't try new patch yet.





[GLASSFISH-21202] Glassfish/Grizzly Memory Leak in Glassfish 4? Created: 16/Sep/14  Updated: 18/Sep/15

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: talk2umakanth Assignee: oleksiys
Resolution: Unresolved Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

GlassFish Server Open Source Edition 4.0 (build 89)
Red Hat Enterprise Linux Server release 5.9 Beta (Tikanga)
Linux 2.6.18-339.el5 #1 SMP Mon Aug 27 15:42:17 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux



 Description   

We're facing very frequent out of Memory problems in our Glassfish Installation of an enterprise web application.

The application is running in a cluster of 18 JVMs. Initial heap size was 1 GB which resulted in the JVMs going down almost every hour and we have increased the heap size to 1.5 GB which makes the JVMs to crash every 3-4 hours.

The Heap dump does not show any application objects at all whenever the JVMs crash and shows the same objects from Glassfish/Grizzly consuming majority of the memory all the time. Below object is found to consume more than 65% of heap in most of the heap dumps.

762,277,600 bytes (52.71 %) of Java heap is used by 5,632,379 instances of org/glassfish/grizzly/memory/HeapMemoryManager$TrimmableHeapBuffer

Following error is also seen in the logs frequently (which is due a malformed URL having && in the request query string??). Can this be a potential cause of the problem? Running a 12 hours load test using similar URL does not produce out of memory in non production systems.

GRIZZLY0155: Invalid chunk starting at byte [387] and ending at byte [387] with a value of [null] ignored]]

The error is not reproducible in non-production environment but heap dumps from production can be provided if required.

  • Are there any know memory leaks in Glassfish 4.0 (Build 89)/Grizzly 2.3.1?
  • What does the object HeapMemoryManager$TrimmableHeapBuffer constitute of? Is there any way this can be avoided if there are no known/feasible fixes for Glassfish or Grizzly?

Additional Information (may not be relevant)

  • Each JVM is fronted by an Apache 2.4.9 HTTP Server using mod_jk to communicate with glassfish which are in turn balanced by Netscale load balancer.


 Comments   
Comment by oleksiys [ 18/Sep/14 ]

Heap dump tree could help us understand the problem.

Thank you.

Comment by talk2umakanth [ 18/Sep/14 ]

You can see the heap tree in the image: http://i.stack.imgur.com/tJEQV.png

All the heapdumps show the same objects leaking as is being shown in the image.

Comment by oleksiys [ 18/Sep/14 ]

Can you pls. apply the following patch (2 jars) [1] and [2] and enable FINER logging level for org.glassfish.grizzly.http.io.InputBuffer
When/if it fails pls. attach the thread dump and logs.

Thank you.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.0/nucleus-grizzly-all.jar
[2] https://dl.dropboxusercontent.com/u/7319744/glassfish-4.0/glassfish-grizzly-extra-all.jar

Comment by anshul.kumar [ 18/Sep/14 ]

Thanks oleksiys. Does this patch contain any fixes as well or it is just to gather additional logging information? We just want to make sure before promoting it in the production system. It has tested successfully in non-production environment.

Comment by oleksiys [ 19/Sep/14 ]

It does have different glassfish4 fixes, not just logging.

Comment by anshul.kumar [ 19/Sep/14 ]

We applied this patch on one node containing 3 JVM instances of the application and the instances are still crashing after applying the patch.

The crashdumps including JVM Log files, thread dumps and the images of heap and permgen space leading up to out of memory of the instances can be downloaded from this link: https://drive.google.com/file/d/0B52890ypA1_bYlR4Snh5bUxLMzg/edit?usp=sharing

Note: The JVM Time Zone is one hour behind the timestamps that are visible in the images of heap size and permgen size. Another observation that we are seeing is that the JVM crashes just when it tries to resize the permgen space. Max Permgen space for JVM is set to 512 MB. Can this be a part of problem and should we try setting min and max perm size to be the same value to avoid permgen resize?

Comment by oleksiys [ 22/Sep/14 ]

Thank you for the info.
Can you pls. apply this patch [1] and also enable FINEST logging for
org.glassfish.grizzly.http.util.Parameters?

Thank you!

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/nucleus-grizzly-all.jar

Comment by anshul.kumar [ 22/Sep/14 ]

Thank you for the information.

Do you want to keep the logging level of org.glassfish.grizzly.http.io.InputBuffer at FINER as well or we should switch that logger off now?

Comment by oleksiys [ 23/Sep/14 ]

please keep it as well.

thank you.

Comment by anshul.kumar [ 23/Sep/14 ]

The crash dumps of two instance crashes from today after applying the updated jar file can be downloaded from here:
https://drive.google.com/file/d/0B52890ypA1_bbEFpLS1zZkxpbzg/edit?usp=sharing

(Thread dumps are also included in the zip file). If you need more data, please let me know.

Thanks

Comment by anshul.kumar [ 25/Sep/14 ]

Is there anything that could be found in the latest crash dumps? Should we collect any other data as the instances are still crashing every 2-3 hours

Comment by oleksiys [ 25/Sep/14 ]

The logs look fine the only piece of information missed (from the very latest log) is thread http-listener-3(44), from the thread dump I see this thread is constantly reading data, but I don't see any logging activity from this thread in the server.log* files.
For now it looks like the problem might be caused by a huge HTTP POST request, which represents form parameters.

Can you pls. share the raw heap dump file so I can learn it locally?

Thank you!

Comment by anshul.kumar [ 26/Sep/14 ]

Thank you! A crashdump including the server.log* files, thread dumps and heapdump from the latest crash can be downloaded from here: https://drive.google.com/file/d/0B52890ypA1_bVzBvaGtjTHNYTE0/edit?usp=sharing

Please let me know if you need any additional data. I am going to reduce the maximum post size for the glassfish instances to 1-2 MB and will enable the option to print the request size in HTTP Server (apache) to see if there was a big POST request leading up to the crash.

Comment by anshul.kumar [ 26/Sep/14 ]

I enabled the request size directive in HTTP Server. The maximum size of GET or POST data is 171 KB leading up to the latest crash. Is that POST size big enough for the JVM to consume 1.5 GB memory and should I trey reducing this size?

You can see the log of requests for the latest crash by downloading the spreadsheet from here: https://drive.google.com/file/d/0B52890ypA1_beDZReldlWV9MLTA/edit?usp=sharing

A common pattern I have noticed is that there is one request that always takes more than 50 minutes to process with a return code of 502 leading up to the crash. The size of this request is very small (131 bytes). This request can be seen starting at 26-Sep-14 10:26:34, row number 26721 in the spreadsheet. The request completed eventually with a return code of 502 when the instance was killed and restarted. I hope this might give some extra details about the problem.

When the instance is healthy, the same request takes only 1-2 seconds.

Comment by oleksiys [ 27/Sep/14 ]

I think I found the problem, can you pls. try this patch (2 files).

https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/nucleus-grizzly-all.jar
https://dl.dropboxusercontent.com/u/7319744/glassfish-21202/glassfish-grizzly-extra-all.jar

Pls. let me know if it works for you.

Thanks!

Comment by anshul.kumar [ 27/Sep/14 ]

This is great new Alexis. Do we need to enable any logging after this patch. I had switched off the loggers after some time as it creates a huge log backup within hours.

Comment by oleksiys [ 27/Sep/14 ]

No, you don't need any extra logging now, it should just work.

Comment by anshul.kumar [ 28/Sep/14 ]

so far so good. I patched one node with three instances on it and the instances have been very stable (never used more than 550 MB of heap) while the other instances that have not been patched are still crashing. I will update on Monday evening as we do not have comparable traffic over weekends as weekdays.

Comment by anshul.kumar [ 05/Oct/14 ]

Alexis - I have slowly rolled out this patch to our entire environment and the JVM instances have been very stable following the patch. The patch has resolved the problem permanently and this issue can be closed.

Very curious to know why we were not able to recreate the problem in our pre-prod application using load tests though.

Thank you for caring.

Comment by oleksiys [ 06/Oct/14 ]

I'm glad the fix works.
I assume your testing env. also has Apache and GF backend communicating via AJP protocol.
Try to reproduce the following usecase: send HTTP post message to apache, which contains entire HTTP header (with Content-Length) and a half of the POST payload, then close HTTP connection... Something like:

$ telnet localhost 80
POST /xxxxx HTTP/1.1
Host: localhost:80
Content-Length: 100
Content-Type: application/x-www-form-urlencoded

aaa=bbb&^C^C^C
Comment by smillidge-c2b2 [ 06/Oct/14 ]

Is there a Grizzly JIRA I can track for this if the fix is in Grizzly?

Comment by oleksiys [ 06/Oct/14 ]

I just created one here
https://java.net/jira/browse/GRIZZLY-1709

Comment by anshul.kumar [ 07/Oct/14 ]

Will this fix be included in any of the future Glassfish 4.1 release?

Comment by oleksiys [ 07/Oct/14 ]

Future - yes, but 4.1 has been released already. I'll prepare a patch for it separately.

Comment by kallie82 [ 09/Oct/14 ]

Has a patch been prepared perhaps? We are running into what seems to be the same issue on GlassFish Server Open Source Edition 4.1 (build 13).
Thanks!

Comment by anshul.kumar [ 20/Oct/14 ]

Alexis - Can you please help provide a patch for GF 4.1 as well for this issue. I have a few pending upgrades for the latest glassfish version

Comment by oleksiys [ 12/Dec/14 ]

here is the patch for 4.1
https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/glassfishv41-patch.zip

Comment by anshul.kumar [ 18/Sep/15 ]

Just wanted to add a note here regarding this fix. While after applying the patch on glassfish that was provided, the JVMs behind Apache (using AJP) have been super stable, the same glassfish JVMs will crash about every 12 hours if we do not use an Apache Frontend in front of the JVMs (We use BIGIP F5 load balancer instead of Apache for some JVMs and they crash every 12 hours if this patch is used).

The crash has the same symptoms as I saw in the crashes when it occurred for JVMs behind Apache.

The raw heapdump for one of such crashes is present here: https://drive.google.com/open?id=0B52890ypA1_bdEEyMXNoYWUxSjg

When I revert the patch from this ticket and put the two jars that came with original glassfish (version 4 Build 89), the crash does not occur if I am not using an Apache frontend.

So, in a nutshell, if using original build 89, the crashes occur with Apache frontend but do not occur if apache is not used. If I use the patch, the crashes occur without apache but do not occur with apache.





[GLASSFISH-21246] Grizzly thread pool waiting on LinkedTransferQueue hangs application Created: 03/Nov/14  Updated: 12/Nov/15

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1_b10
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: afcarv Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 4.1_b13, JDK 1.7u67, Linux CentOS 3.10.48 x64



 Description   

Elaborating on the issue I posted on GLASSFISH-21211.

After some time running, http-listener-1 thread pool stops responding. Checking the logs, I see multiple messages like the one below:

[2014-11-03T18:50:47.114+0000] [glassfish 4.1] [SEVERE] [] [org.glassfish.grizzly.nio.SelectorRunner] [tid: _ThreadID=60 _ThreadName=http-listener-1-kernel(1) SelectorRunner] [timeMillis: 1415040647114] [levelValue: 1000] [[
  doSelect exception
java.util.concurrent.RejectedExecutionException: The thread pool's task queue is full, limit: 4096
        at org.glassfish.grizzly.threadpool.AbstractThreadPool.onTaskQueueOverflow(AbstractThreadPool.java:464)
        at org.glassfish.grizzly.threadpool.QueueLimitedThreadPool.execute(QueueLimitedThreadPool.java:81)
        at org.glassfish.grizzly.threadpool.GrizzlyExecutorService.execute(GrizzlyExecutorService.java:161)
        at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.executeIoEvent(WorkerThreadIOStrategy.java:100)
        at org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89)
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:414)
        at org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:383)
        at org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:347)
        at org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:278)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:744)
]]

I then took a thread dump which shows all threads in http-listener-1 in a state like the below:

"http-listener-1(31)" daemon prio=10 tid=0x00007ff19422b000 nid=0x6c61 waiting on condition [0x00007ff1f6eed000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006036fd360> (a java.util.concurrent.LinkedTransferQueue)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
        at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
        at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
        at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:557)
        at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545)
        at java.lang.Thread.run(Thread.java:744)

They seem to be waiting on LinkedTransferQueue take() method, which appears to be not responding (waiting for some item to arrive in the queue?). I also took a memory dump that I can look into if someone points me in the right direction.

Thought this could be related to GRIZZLY-1711 but after applying the latest version (2.3.17-SNAPSHOT) this behavior is still happening.

Let me know if I should get more data - thanks!



 Comments   
Comment by davidwinters1980 [ 03/Nov/14 ]

This appears to be occurring as a result of the http thread pool reaching its limit of 4096.

As per the default domain.xml setting:

<thread-pools>
<thread-pool name="admin-thread-pool" max-thread-pool-size="50" max-queue-size="256"></thread-pool>
<thread-pool name="http-thread-pool" max-queue-size="4096"></thread-pool>
<thread-pool name="thread-pool-1" max-thread-pool-size="200"/>
</thread-pools>

You could configure this to -1 or unlimited but this will likely lead to other issues such as out of memory etc

It would appear as though tasks are being added to this queue faster than the tasks can be processed. I would investigate why this occurring? If you could take a number of thread dumps over a period of time whereby we capture this issue occurring that would be useful.

Furthermore, it maybe worth increasing the value of your http request processing threads as this will allow more tasks to be handled concurrently off the http thread task queue and so it may reach the 4096 limit so quickly. What is the current value of the number of http request processing threads??

The issue you have here is very dependent on the number of http requests being pushed to the http task pool and how quickly these requests are being handled so appropriate tuning of the http parameters may resolve this issue. If this fails then looking at the state of all threads running over a period of time should give you a hint as to why requests are taking so long to process.

Comment by afcarv [ 03/Nov/14 ]

I don't think it's a configuration issue as the environment is able to handle much heavier loads - it may be due to specific and temporary environmental reasons (network latency, database slowdown, etc) but you'd think it would resume work as soon as this is normalized. This does not happen (waited for some hours to be sure), even if the consumer is stopped and there's no inbound traffic anymore - the only way to bring it back online is with a server/cluster restart. I did notice a lot of connections in the CLOSE_WAIT state in the server, though.

Haven't tried to raise the queue limit or remove it yet, but I suspect it would only delay the occurrence or eventually reach an OOM error.

I see no obvious reason for this looking at the thread pool dump, but will post it later for analysis. Thanks!

Comment by afcarv [ 04/Nov/14 ]

Please find attached the full thread dump and the server log file. The dump was taken about 30 minutes after the issue manifested itself - you can see all threads in the http-listener-1-kernel pool reach the queue limit and stop responding. http-thread-pool contains 32 threads; there are 8 acceptor threads.

Interestingly, http-listener-2 (SSL) continues to respond normally. http-listener-1 hanged and required a server restart. The configuration is a 2 instance cluster with a nginx load balancer in front; there are Jersey applications deployed serving RESTful webservices. The configuration can handle about 5x the average load on the server - no increased throughput was observed leading to the issue, which persisted after stopping all inbound traffic.

I will schedule a cron job to take a thread dump every 5 minutes to check on any odd behavior leading to the issue - thanks!

Thread dump: https://www.dropbox.com/s/qejdzhwejlv1zzp/threadump.txt?dl=0
Server log: https://www.dropbox.com/s/rbu24cep9y25aol/server.log?dl=0

Comment by afcarv [ 04/Nov/14 ]

Some more info I got from the memory dump -

I see lots of instances of TCPNIOConnection (1500+) in what appears to be a closing state; as the latest snapshot was applied I can see these with a CloseReason of "LOCALLY". May explain the connections in the CLOSE_WAIT state I saw? Is it possible that nginx closed the socket before a FIN packet could be sent from the server, and now it is not able to end the connection properly? Not sure if this could be just a consequence of some other issue, though.

Comment by oleksiys [ 06/Nov/14 ]

Which Grizzly version is used in 4.1_b10?

Thank you.

Comment by afcarv [ 06/Nov/14 ]

We're using b13 (couldn't find the tag for it so I selected the closest), but I believe it's the same version - 2.3.15.

Currently, 2.3.17-SNAPSHOT is applied.

By the way - we found out that what triggers this behavior is a DB performance degradation, causing the requests to queue and eventually reach the limit. No additional info on why queue isn't being reduced, though.

Comment by davidwinters1980 [ 06/Nov/14 ]

Could you take a number of thread dumps say every 5 minutes so that we can compare the different thread dump files leading up to the hang? Also could you send us on the contents of the <transports> which should contain the different tcp parameters used by Grizzly.

<transports>
<transport byte-buffer-type="HEAP" display-configuration="true" name="tcp" .....></transport>
</transports>

Useful to know db degradation issues are causing requests in the queue to fill up. It might be useful to get some debug grizzly logging on here: org.glassfish.grizzly.level=FINEST

Thanks.

Comment by afcarv [ 10/Nov/14 ]

I had some monitoring in place but we've been focusing on fixing the db degradation so the issue won't be triggered - so far we've been successful, so it looks unlikely I'll be able to collect more data in the production system.

What I'm going to try next is to create a sample application and configuration that will be able to recreate the issue - doesn't look too hard; just a sample RESTful webservice that performs a db query that has an artificial delay in response, small db connection pool and a sample JMeter test script with http requests that will timeout before the app has a chance to respond. Should replicate the conditions pretty closely (don't know if nginx in front plays a role in this or not, so will try without it first).

Will report on any news, thanks!

Comment by oleksiys [ 10/Nov/14 ]

pls. try to patch GFv4.1 with this patch [1].
I need server.log output message(s) like:
"The queue is overflowed. queuePermits=......."

Thank you.

[1] https://dl.dropboxusercontent.com/u/7319744/glassfish-21246/nucleus-grizzly-all.jar

Comment by afcarv [ 22/Oct/15 ]

Hi - I've been looking into this again. We still get a non-responsive cluster from time to time, and it's always due to some external resource that cause the task queue to overfill; I've seen it happen due to a DB slowdown (as I've reported before) and also due to an issue with ActiveMQ (slow response).

Recreating this in a test environment has been proving to be quite tricky, but it appears I have devised a somewhat reliable way to do it using the scenario I've described before and using a test thread pool with at least twice the number of the task queue maximum capacity to generate requests to the application server.

I've successfully triggered this issue in GFv4.0, GFv4.1 and GFv4.1.1.

I will apply your patch next and report back if I get any results - thanks!

Comment by afcarv [ 22/Oct/15 ]

Please find server.log file below:

https://dl.dropboxusercontent.com/u/106573849/GF-21246/server.log

Method "testDAOMethod()" of class "TestDAOBean" calls a stored procedure that waits for 1 minute before giving a response. Ran a test script that kept calling a RESTful endpoint which invoked this method; massive amount of requests caused the http-listener-1 task queue to quickly overfill and stop responding. At this point, I stopped the test script by killing the process.

About 1 hour later, http-listener-1 is still hung. I then made a test call to the SSL endpoint which is handled by http-listener-2; this request went through cleanly, as can also be seen in the log file.

Let me know if you want me to run additional tests or fetch additional information. Thanks!

Comment by ruVooDoo [ 12/Nov/15 ]

Hello, our prod env show the same pattern...
"http-listener-1(119)" daemon prio=10 tid=0x00007f6409038000 nid=0x27e2 waiting on condition [0x00007f62f62e1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x000000067d016e20> (a java.util.concurrent.LinkedTransferQueue)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:735)
    at java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:644)
    at java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1137)
    at org.glassfish.grizzly.threadpool.FixedThreadPool$BasicWorker.getTask(FixedThreadPool.java:105)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:556)
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
    at java.lang.Thread.run(Thread.java:724)

Thats very wierd, becauseour app working perfect for month`s on GF 4.0, but now it`s hanging... We did not change anything in the app.

Could you please provide a fix?

Comment by oleksiys [ 12/Nov/15 ]

@ruVooDoo the stacktrace you provided doesn't mean GF hangs, it's just a thread waiting for a task to be assigned.
The problem can be somewhere else.
Please provide full stacktrace (all threads) snapshot at the time you see the problem.

Comment by ruVooDoo [ 12/Nov/15 ]

https://www.dropbox.com/s/9ur7963sf3tb2ja/gso_hung_27757.txt?dl=0
We are using GlassFish Server Open Source Edition 4.0 (build 89)





[GLASSFISH-11031] ContainerMapper should better handle the empty string root path Created: 13/Nov/09  Updated: 13/Feb/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1
Fix Version/s: future release

Type: Bug Priority: Major
Reporter: jfarcand Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: Macintosh


Issuezilla Id: 11,031
Tags: 3_1-exclude

 Description   

See https://glassfish.dev.java.net/issues/show_bug.cgi?id=10601

The new emptyString support in Servlet 3.0 breaks the ContainerMapper mapping
algorithm. We patched the Mapper to execute in compatible mode for the 3.0
release, but we must fix it properly in ContainerMapper for the 3.0 release.



 Comments   
Comment by jfarcand [ 13/Nov/09 ]

Exclude for that release.

Comment by jfarcand [ 01/Dec/09 ]

Alexey is now the owner

Comment by oleksiys [ 11/Oct/10 ]

target for ms7

Comment by oleksiys [ 11/Nov/10 ]

the fix may cause regression in admin gui, SCF is not the best time to make this change.
moving to 3.2





[GLASSFISH-6371] Be able to use EJB3 using a http tunnel Created: 30/Sep/08  Updated: 01/May/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: V3
Fix Version/s: future release

Type: New Feature Priority: Major
Reporter: batmat Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issuezilla Id: 6,371

 Description   

Hi,

After having considered migration to EJB3, we realize that GF doesn't have an
"http-invoker" feature.

In fact, we'd like to be able to tunnel all our call to the server through an
http one.

The better would be being able that the client doesn't need anything else than
http to (1) resolve the remote ejb reference, (2) call it. To sum up, would be
great if GF has the corresponding feature of JBoss (e.g. see
http://wiki.jboss.org/wiki/Accessing_EJB3s_over_HTTP_HTTPS or
http://wiki.jboss.org/wiki/EJBOverHTTPWithUnifiedInvoker).

At least, we'd be interested into knowing how we could inject say our transport
classes (integrating spring http-invoker), and we'd also be glad to know if
there's any chance that it gets into GF if we provide a patch for it.

Cheers.
Thanks for your work, GF is already a masterpiece.



 Comments   
Comment by km [ 01/Oct/08 ]

Looks like this is an interesting feature request. Thanks for filing it as such.
If I understand it correctly, you need some standard servlets that will tunnel
EJB calls via HTTP. This is useful for all the applications that have EJB
components, but don't have Web components that front end those EJB components.

IMO, JBoss makes you do too many configuration changes for this to be achieved.

Comment by batmat [ 01/Oct/08 ]

> IMO, JBoss makes you do too many configuration changes for this to be achieved.
I totally, fully, wholeheartedly agree. This configuration is a nightmare.

What I would dream of would be to be able to configure a glassfish out of the
box using say a particular setup-firewallconstrained[cluster].xml and be done
(and providing the right jndi property to access).

More generally, for me, this feature request is about being able to reduce the
number of needed ports, to simplify a firewall configuration. And more
importantly, to delete the dynamic ports/be able to specify fixed ones. Kind of
UnifiedInvoker as it exists in JBoss.

If you think it'd be a good thing to file another FR and link it to this one,
please just let me know.
Thanks.

Comment by Bill Shannon [ 01/Oct/08 ]

I know it's not exactly what you asked for, but given that you're separating
the client and server by a firewall, implying that they're more loosely
coupled, have you considered using JAX-WS to expose your EJBs as web services?
Yes, it's a somewhat different programming model, but it might be a better
match for your environment.

Comment by batmat [ 01/Oct/08 ]

Hi Shannon,
Well, at least, Sun coworkers are coherent. See the discussion with Ken Saks
and his proposal here:
https://glassfish.dev.java.net/servlets/ReadMsg?list=ejb&msgNo=527 .

To sum up, no. We're not considering WS programming for many reasons I describe
in the given thread.
Cheers.

Comment by Ken Cavanaugh [ 02/Oct/08 ]

One of the features we have been discussing for GFv3 is port unification for all protocols.
This would mean that HTTP, WS, and IIOP could all be handled over the same port.
Does this solve the requirement, or do you need more than just the same port?
We do occasionally see requests to allow tunneling of RMI-IIOP requests over HTTP,
but so far this has not been sufficiently common to consider implementing it.

Of course, you can also look at invoking EJBs directly from the HTTP path,
as others have discussed.

Comment by batmat [ 02/Oct/08 ]

> This would mean that HTTP, WS, and IIOP could all be handled over the same port.
Does this solve the requirement, or do you need more than just the same port?
That's not exactly the same thing, but it would definitely be a damned good
thing to convince people here about Java EE servers, and more precisely about GF
part.

So, I guess this could be sufficient.

> We do occasionally see requests to allow tunneling of RMI-IIOP requests over
HTTP, but so far this has not been sufficiently common to consider implementing it.
Do you see it as an interesting/acceptable request but don't have time to
implement it, or is it just not gonna be accepted even if we try to provide a
patch for it that'd suit you?

Cheers

Comment by batmat [ 27/Nov/08 ]

> One of the features we have been discussing for GFv3 is port unification for
all protocols. This would mean that HTTP, WS, and IIOP could all be handled over
the same port.

That feature would be really interesting. Isn't it this one:
https://glassfish.dev.java.net/issues/show_bug.cgi?id=4322.

Maybe the current issue should be put as related?

Cheers.

Comment by Tom Mueller [ 23/Jun/10 ]

Kedar no longer on project.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

Comment by jthoennes [ 26/Mar/12 ]

This is related to the port unification feature I am also looking for a long time now.
See GLASSFISH-4073: Move CORBA transport to Grizzly (https://java.net/jira/browse/GLASSFISH-4073).

If we could access the ORB for JNDI and EJB access using the Grizzly port unification framework
(port finder and port filter). See http://antwerkz.com/blog/port-unification-in-glassfish-3-part-1.

Batman, did you find any workaround so far?

Comment by Tom Mueller [ 04/Jan/13 ]

Moving to EJB component.

Comment by marina vatkina [ 04/Jan/13 ]

You can use EJBs as JAX-RS endpoints. Assigning to the JAX-RS for further evaluation.

Comment by kumara [ 24/Apr/13 ]

Well, programming in JAX-RS was already suggested as a solution and in that case there is nothing that the server has to do, the application needs to change.

The most concrete comments about a solution was use of port unification to ensure that IIOP traffic is on the same port as http. While this is not exactly a http tunnel, it might be enough for some scenarios.

I am going to request grizzly-kernel team to add a pointer to port unification documentation so that anyone referring to this issue has a handy reference on how to configure IIOP traffic on same port as http.

If port unification is not sufficient, given the availability of JAX-RS programming model and better integration between EJB and JAX-RS, the most prudent option is to change the application.

After adding the document pointer, please assign this back to orb sub-component.

Comment by oleksiys [ 24/Apr/13 ]

Is Corba (IIOP) working on top of Grizzly?
If not then port unification will not work.

Comment by jthoennes [ 25/Apr/13 ]

A related issue is also whether OpenMQ would work using port unification. I think the new OpenMQ 5.0 is based on Grizzly, isn't it?

Comment by oleksiys [ 01/May/13 ]

yes, MQ should work.





[GLASSFISH-17934] Support SNI (server name indication) Created: 08/Dec/11  Updated: 11/Mar/14

Status: Reopened
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: foal Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Support SNI (server name indication) to allows multiple SSL Certificates on a Single IP/Port



 Comments   
Comment by Ryan Lubke [ 28/Feb/13 ]

If SNI support is desired, I would recommend switching to JDK 7 [1].

[1] http://docs.oracle.com/javase/7/docs/technotes/guides/security/enhancements-7.html

Comment by oleksiys [ 11/Mar/14 ]

implemented in Grizzly [1], we can think how to promote it to Glassfish.

[1] https://java.net/jira/browse/GRIZZLY-1661





[GLASSFISH-17098] Move network-configuration CLI commands to nucleus under "grizzly-glue" module Created: 25/Jul/11  Updated: 02/Dec/11

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Task Priority: Major
Reporter: oleksiys Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: 3_1_x-exclude, nucleus-cleanup

 Description   

subj.






[GLASSFISH-21136] Log filled with TimeoutException on TCPNIOTransportFilter.handleRead Created: 18/Jul/14  Updated: 21/Jul/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1_b08
Fix Version/s: None

Type: Bug Priority: Major
Reporter: gcruscoe Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, Java 8u11


Tags: fishcat

 Description   

Not sure what the cause is because it does not link to any of our code in the stack trace. Could be related to code running in a ejb timers, but not sure.

The stack trace I get is this:

2014-07-18 14:50:26.466 - :: ERROR [http-listener-2(6)] javax.enterprise.web.core:287 - An exception or error occurred in the container during the request processing
java.io.IOException: java.util.concurrent.TimeoutException
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:90) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.TransportFilter.handleRead(TransportFilter.java:173) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ssl.SSLBaseFilter$SSLTransportFilterWrapper.handleRead(SSLBaseFilter.java:999) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:351) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:695) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.blockingRead(InputBuffer.java:1119) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead(ServerInputBuffer.java:95) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.fill(InputBuffer.java:1143) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.io.InputBuffer.read(InputBuffer.java:353) ~[nucleus-grizzly-all.jar:na]
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:267) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:234) ~[web-core.jar:na]
at org.apache.catalina.authenticator.FormAuthenticator.saveRequest(FormAuthenticator.java:614) ~[web-core.jar:na]
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:250) ~[web-core.jar:na]
at com.sun.web.security.RealmAdapter.invokeAuthenticateDelegate(RealmAdapter.java:1524) ~[websecurity.jar:na]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:606) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:702) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) ~[web-core.jar:na]
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) ~[web-glue.jar:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:174) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:734) ~[web-core.jar:na]
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:673) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:412) ~[web-core.jar:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:282) ~[web-core.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:459) [kernel.jar:na]
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:167) [kernel.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:175) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:235) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:284) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:201) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:133) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:561) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:117) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:56) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:137) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:565) [nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:545) [nucleus-grizzly-all.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.util.concurrent.TimeoutException: null
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:126) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(TemporarySelectorReader.java:75) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.AbstractReader.read(AbstractReader.java:72) ~[nucleus-grizzly-all.jar:na]
at org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(TCPNIOTransportFilter.java:77) ~[nucleus-grizzly-all.jar:na]



 Comments   
Comment by oleksiys [ 18/Jul/14 ]

looks like it happens during the form authentication, when the server expects more information from the browser, but the browser doesn't send it. The read attempt fails after read-timeout expires.

Comment by gcruscoe [ 18/Jul/14 ]

This occurs while there is no user interaction on the web page, but the page does make ajax calls and is connected via atmosphere using websockets. This also occurs on startup at once (although less frequently I believe) when there are no web browsers pointing at the app server.

The application has several war applications as well as a main ear application. The wars do web service calls to the ejb level web services (all on the same app server). So when the user is using the application there is a web session to the war and the war makes web service calls.

I am wondering if a web page without a session was trying to do ajax calls – although this doesn't explain the same stack trace on startup. Anyway hope any of this helps.

I'll update if I can come up with more examples of what makes this happen.

Comment by oleksiys [ 18/Jul/14 ]

pls. share the exception you see during the startup, because the one you posted is related to form authentication.

Comment by gcruscoe [ 21/Jul/14 ]

Okay exception during startup was actually the same cause. It is the atmosphere connection from a web browser – I had not found and closed all tabs before restarting the app server. Now I do not get this on startup.

The session and SSO / realm are most certainly expired – I do not know if that would contribute this issue. I will work on figuring out more about this and post back when I have more. Hopefully it is benign, but I have not seen these in the logs on gf 3.1.2.2 so it is a bit troubling.





[GLASSFISH-21189] Frequent network dropouts Created: 10/Sep/14  Updated: 19/Sep/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0
Fix Version/s: None

Type: Bug Priority: Major
Reporter: pranahata Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

JDK 8 u 5, 64 bit CentOS



 Description   

We have a GlassFish server hosted on a datacenter and a jee application with a desktop application deployed via java web start running on it.

When the JWS client downloads the jars, we get very frequent network dropouts, mainly with jar files > 3 Mb where the JWS client reports: "EOF: Unexpected end of ZLIB input Stream" and the server reports the following exception.

ava.io.IOException: java.lang.InterruptedException
at org.glassfish.grizzly.utils.Exceptions.makeIOException(Exceptions.java:81)
at org.glassfish.grizzly.http.io.OutputBuffer.blockAfterWriteIfNeeded(OutputBuffer.java:958)
at org.glassfish.grizzly.http.io.OutputBuffer.write(OutputBuffer.java:682)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:355)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:342)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:161)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
at jnlp.sample.servlet.DownloadResponse$FileDownloadResponse.sendRespond(DownloadResponse.java:257)
at jnlp.sample.servlet.JnlpDownloadServlet.handleRequest(JnlpDownloadServlet.java:187)
at jnlp.sample.servlet.JnlpDownloadServlet.doGet(JnlpDownloadServlet.java:120)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.apache.catalina.core.StandardWrapper.service(

Clients connecting over a wireless internet connection get these errors much more frequently.

The client application talks to the server via http.

If i do a ping -t to the server, the number of lost packets is not that big and sometimes there is no packet loss at all when this exception happens.

Dropouts seem to last no longer than one second yet, the amount of failed downloads or calls seems to be too big and causes too many errors on the application.

If this is not a bug, i would appreciate if someone told me if there is any setting that we can configure on glassfish to enable http clients to continue reading after a drop out or to change the timeouts or retries or any other configuration to make the communication more robust?



 Comments   
Comment by oleksiys [ 19/Sep/14 ]

pls. try to disable request-timeout on http listener for example for http-listener-1:

$asadmin set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds=-1




[GLASSFISH-21061] keepAlive-timeout happened between "qin" and "qout", the connection will be broken Created: 05/May/14  Updated: 08/Aug/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: houtang Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: 4 days
Time Spent: Not Specified
Original Estimate: 4 days
Environment:

all


Tags: NullPointerException, conn, disc, grizzly

 Description   

When I try to access a web application through ie6 on glassfish 3.1.2.2, the following phenomenon has been happened:
If there's a keepAlive-timeout happened between "qin" and "qout", the connection will be broken. You will found from ※1 on the printed log that the connection has been broken.
BTW, when we access through ie6 after the first connection was broken, there will be another connection created(※2) and return back as normal. However, when we access the web application through ie8 after the first connection was broken, the second connection can't be created, so the log about ※2 can't be print to the log under this situation, the connecton will be broken here and it can't be return the right result.

log of ie6
-----------------------------------
"23/Apr/2014:12:05:22.485" "20(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:2955"
"23/Apr/2014:12:05:25.090" "20(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:2955"
"23/Apr/2014:12:05:51.064" "20(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:2955" ※1
"23/Apr/2014:12:08:59.668" "81(http-thread-pool-28292(4))" "qout" "null"
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:4045" ※2
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "qout" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "recv" "GET /ServletSession/jsp/ServletSessionOut.jsp HTTP/1.1"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "call"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "rtn"
"23/Apr/2014:12:09:01.587" "83(http-thread-pool-28292(5))" "send" "200" "193.168.154.174:4045"
"23/Apr/2014:12:09:01.587" "20(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:4045"
"23/Apr/2014:12:09:11.727" "20(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:4045"
-----------------------------------

log of ie8
-----------------------------------
"23/Apr/2014:12:38:49.004" "20(Grizzly-kernel-thread(1))" "conn" "10.167.157.142:1580"
"23/Apr/2014:12:38:57.194" "20(Grizzly-kernel-thread(1))" "qin" "10.167.157.142:1580"
"23/Apr/2014:12:39:12.654" "20(Grizzly-kernel-thread(1))" "disc" "10.167.157.142:1580"
"23/Apr/2014:12:39:24.572" "107(http-thread-pool-28292(13))" "qout" "null"
-----------------------------------

Above all, the question is whether this phenomenon is a bug about ie or it is a glassfish internal issue?

I think the connection shouldn't be broken when the keepAlive-timeout happened between "qin" and "qout".

In addition, When the above phenomenon happened on glassfish 2.1.1, the following log will be print to
the server.log, the content of the log is same to the issue of GLASSFISH-20622

server.log
-----------------------------------
java.lang.NullPointerException
at org.apache.coyote.tomcat5.OutputBuffer.addSessionCookieWithJvmRoute(OutputBuffer.java:689)
at org.apache.coyote.tomcat5.OutputBuffer.doFlush(OutputBuffer.java:370)
at org.apache.coyote.tomcat5.OutputBuffer.close(OutputBuffer.java:336)
at org.apache.coyote.tomcat5.CoyoteResponse.finishResponse(CoyoteResponse.java:594)
at org.apache.coyote.tomcat5.CoyoteAdapter.afterService(CoyoteAdapter.java:354)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.postResponse(DefaultProcessorTask.java:625)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:612)
at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:889)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:267)
at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:118)
-----------------------------------



 Comments   
Comment by oleksiys [ 23/May/14 ]

When do you print "qin" log? I see it happens in the kernel thread, but where exactly?

Thank you.

Comment by houtang [ 06/Jun/14 ]

==============Here is the background===============
Below is the logic trace when use glassfish3.1.2.
Basicly it is similar with glassfish2.1, as two thread.
Firstly Thread <Grizzly-kernel-thread> built connection "conn"(marked with ★1), then it went to "qin"(marked with ★2),
after that another Thread <http-thread-pool> was woke up, and excuted as below sequence "qout"-> "recv"-> "send" (marked with ▲1,▲2,▲3).
Then after 15 seconds(keepalivetimeout) the connection was disconnected "disc"(marked with ★3)

glassfish3.1.2
-------------------------------------------------
#=com.sun.grizzly
$=com.sun.enterprise.v3.services.impl.monitor
@=com.sun.grizzly.util
*=com.sun.grizzly.http
&=com.sun.enterprise.v3.services.impl

<Grizzly-kernel-thread>
#SelectorHandlerRunner.doSelect(SelectorHandler, NIOContext) : 201
#SelectorHandlerRunner.handleSelectedKey(SelectionKey, SelectorHandler, NIOContext)行: 302
$MonitorableSelectorHandler.acceptWithoutRegistration(SelectionKey) ★1 conn
#SelectorHandlerRunner.handleSelectedKey(SelectionKey, SelectorHandler, NIOContext)行: 381
#NIOContext.execute(ContextTask)
@ SyncThreadPool.execute(Runnable) : 189 ★2 qin
#SelectorHandlerRunner.doSelect(SelectorHandler, NIOContext) : 212
#TCPSelectorHandler.postSelect(Context) 行: 534
#SelectorThreadKeyHandler.cancel(SelectionKey) : 84
#BaseSelectionKeyHandler).doAfterKeyCancel(SelectionKey) 行: 234 ★3 disc

<http-thread-pool>
HttpWorkerThread(Thread).run() 行: 722
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).run() 行: 520
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).doWork() : 536 ▲1 qout
@SyncThreadPool$SyncThreadWorker(AbstractThreadPool$Worker).doWork() : 542
#ProtocolChainContextTask(ContextTask).run() : 71
*HttpProtocolChain(DefaultProtocolChain).execute(Context) : 90
*DefaultProtocolFilter.execute(Context) : 231
*ProcessorTask.process(InputStream, OutputStream) : 1118
*ProcessorTask.doProcess() : 752
*ProcessorTask.doProcess() : 793
*ProcessorTask.postResponse() : 817
*ProcessorTask.finishResponse() : 827
&ContainerMapper.afterService(Request, Response) : 487
*ProcessorTask.action(ActionCode, Object) : 1156
*ProcessorTask.prepareResponse() : 1833 ▲3 send
-------------------------------------------------

when i was debuging these programs, i found that if the keepalivetimeout(15 seconds) is up it can "disc"(★3) directly after "qout"(▲1),
and it also can "disc"(★3) after“recv”(▲2)
==============Background ends===============

My first question,is it correct that after "qout" or "recv" the connection can still be disconnected as keepalivetimeout(15 sec) is up?

And when i was reproducing the sitiation , after "recv" then the connection was disconnected(keeptimeout is up), IE6 can build a second connection and finish the request
and return "200", the strange thing is when i use IE8, there is no second connection!! I was so confused, My second question is: what makes the IE6 and IE8 different?
BTW, i got the same consequence in both glassfish3.1.2 and glassfish2.1

Below the is trace.log details
1)IE6
trace.log

"15/May/2014:19:22:47.193" "42(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:1086" // build the first connection
"15/May/2014:19:22:47.194" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1086"
"15/May/2014:19:22:47.194" "78(http-thread-pool-28292(4))" "qout" "193.168.154.174:1086"
"15/May/2014:19:23:14.646" "78(http-thread-pool-28292(4))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:23:17.667" "42(Grizzly-kernel-thread(1))" "disc" "193.168.154.174:1086" // after "recv" connection is disconnected
"15/May/2014:19:23:23.715" "42(Grizzly-kernel-thread(1))" "conn" "193.168.154.174:1087" // build the second connection
"15/May/2014:19:23:23.716" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1087"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "qout" "193.168.154.174:1087"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:23:23.716" "349(http-thread-pool-28292(11))" "send" "301" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "78(http-thread-pool-28292(4))" "send" "301" "null" // the first connection return "null"
"15/May/2014:19:23:23.717" "42(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "42(Grizzly-kernel-thread(1))" "qin" "193.168.154.174:1087"
"15/May/2014:19:23:23.717" "74(http-thread-pool-28292(3))" "qout" "193.168.154.174:1087"
"15/May/2014:19:23:23.718" "74(http-thread-pool-28292(3))" "recv" "GET /HelloServlet/ HTTP/1.1"
"15/May/2014:19:23:23.718" "74(http-thread-pool-28292(3))" "call"
"15/May/2014:19:23:23.719" "74(http-thread-pool-28292(3))" "rtn"
"15/May/2014:19:23:23.719" "74(http-thread-pool-28292(3))" "send" "200" "193.168.154.174:1087" // the second connection sucesses
"15/May/2014:19:23:23.719" "42(Grizzly-kernel-thread(1))" "k-wt" "193.168.154.174:1087"

2)IE8
trace.log

"15/May/2014:19:34:03.339" "42(Grizzly-kernel-thread(1))" "conn" "10.167.157.75:51986" // build the first connection
"15/May/2014:19:34:03.339" "42(Grizzly-kernel-thread(1))" "qin" "10.167.157.75:51986"
"15/May/2014:19:34:03.339" "340(http-thread-pool-28292(9))" "qout" "10.167.157.75:51986"
"15/May/2014:19:34:44.796" "340(http-thread-pool-28292(9))" "recv" "GET /HelloServlet HTTP/1.1"
"15/May/2014:19:34:47.472" "42(Grizzly-kernel-thread(1))" "disc" "10.167.157.75:51986" // after "recv" connection is disconnected
"15/May/2014:19:34:50.884" "340(http-thread-pool-28292(9))" "send" "301" "null" // the first connection return "null" and there is no second connection!!!

Thank you

Comment by oleksiys [ 11/Jun/14 ]

keep-alive timeout should be applied only in cases, when the connection is idle (waiting for the next request to come).
And 15 seconds starts counting once the request/response is processed.
Are you sure it's keep-alive timeout we're talking about?

Thank you.

Comment by houtang [ 11/Jun/14 ]

Thank you for your reply!

I have a query needs your confirmation:
Basically one requests/response processing has below steps(1~9):
1.conn
2.qin
3.qout
4.recv
5.call
6.rtn
7.send
8.k-wt
9.disc

10.conn
11.qin
12.qout
13.recv
14.call
15.rtn
16.send
17.k-wt
18.disc

do you mean the 15 seconds starts counting from 8 and lasts from 8 to 9
or starts counting from 9 and lasts from 9 to 10("conn" of the second request)?
My opinion is the 15 seconds starts counting from 1, and lasts from 1 to 9(from the source logic),
do you agree?

In this case I have below situation: when the request is processing and it is not
completed(for example in step"recv"), at this time connection is disconnected.

conn(15sec starts counting)>qin>qout->recv->disc(15sec ends)

Hope you can get my point
Pls let me know if any misunderstanding

Thank you

Comment by oleksiys [ 11/Jun/14 ]

Hi,

just to be clear, we're talking about HTTP keep-alive, it means HTTP (TCP) connection is not getting closed between requests. Probably it's what you meant in your description, but I just wanted to make sure we're on the same page... so at the step 9. and 18. no real HTTP (TCP) connection termination happens.

Coming back to the steps you listed, confirm that keep-alive timeout is ticking between 9. and 10.

If you observe the timeout during read/write - it must be different timeout then.
First of all I'd recommend to switch to Glassfish 3.1.2.2 and check if the issue is still there.

Thank you.

Comment by houtang [ 16/Jun/14 ]

Hi oleksiys,

>Coming back to the steps you listed, confirm that keep-alive timeout is ticking between 9. and 10."
>If you observe the timeout during read/write - it must be different timeout then.
Ok, seems we are talking about different timeout

Here is part of source code from glassfish(Grizzly), the keep-alive timeout I was talking about is
"idleLimit" below in "com.sun.grizzly.http.SelectorThreadKeyHandler.java".

com.sun.grizzly.http.SelectorThreadKeyHandler.java
------------------------
public void expire(Iterator<SelectionKey> iterator) {
if (timeout == SelectionKeyAttachment.UNLIMITED_TIMEOUT)

{ return; }

......
if (expire >= 0) {
long idleLimit = getIdleLimit(attachment);

if (idleLimit >= 0 && currentTime - expire >= idleLimit &&
(!(attachment instanceof SelectionKeyAttachment) ||
((SelectionKeyAttachment) attachment).timedOut(key)))

{ //preventing further idle timeout detection for same key //due to we dont directly cancel key anymore we cant //rely in key.isvalid detection addExpirationStamp(key, SelectionKeyAttachment.UNLIMITED_TIMEOUT); cancel(key);●1 }

}
}
}
}

------------------------

I hava debugged several times and I am pretty sure this timeout is between read/write,
Can you try to debug from your side?

And I also listed part of the source code of "qin","qout","recv" and "send",marked with ★ below
Here is the debug steps:
1. Put the breakpoints at ●1,●2,★recv
2. When a request comes,<http-thread-pool> Thread will stop at ●2,wait for 30 seconds(default timeout), then <Grizzly-kernel-thread> Thread will come up, stop at●1
3. Keep on debuging <http-thread-pool> and after step over ★recv, go to <Grizzly-kernel-thread> and step over ●1
4. disconnect the debug

<Grizzly-kernel-thread>:
com.sun.grizzly.util.SyncThreadPool#execute(Runnable)
------------------------
public void execute(Runnable task)
{
......
if (((maxQueuedTasks < 0) || (workerQueueSize < maxQueuedTasks)) && (workQueue.offer(task)))

{ onTaskQueued(task); ★qin }

......
}
------------------------

com.sun.grizzly.BaseSelectionKeyHandler.java
------------------------
protected void doAfterKeyCancel(SelectionKey key)
{
try

{ if (selectorHandler != null) selectorHandler.closeChannel(key.channel()); else closeChannel(key.channel()); }

finally

{ notifyLocallyClose(key); ★disc Object attachment = key.attach(null); if ((attachment instanceof SelectionKeyAttachment)) ((SelectionKeyAttachment)attachment).release(key); }

}
------------------------

<http-thread-pool>:
com.sun.grizzly.util.AbstractThreadPool.java
------------------------
protected void doWork()
{
......
AbstractThreadPool.onTaskDequeued(r); ★qout
Throwable error = null;
try

{ beforeExecute(t_, r);       r.run(); ●2 onTaskCompletedEvent(r); ...... }

------------------------

com.sun.grizzly.http.ProcessorTask.java
------------------------
public boolean parseRequest()
throws Exception
{
try

{    inputBuffer.parseRequestLine(); ★recv }

catch (BufferOverflowException boe) {
if (logger.isLoggable(Level.SEVERE))

{ logger.log(Level.SEVERE, LogMessages.SEVERE_GRIZZLY_HTTP_PROCESSOR_TASK_REQUEST_URI_TOO_LARGE_ERROR(), boe); }

......
}

protected void prepareResponse()

{ ...... sendHeaders(); ★send }

------------------------

Thank you

Comment by houtang [ 08/Aug/14 ]

Hi oleksiys,

How are you doing?
It's been almost 2 months since your last response.
Is there any progress? Or you still have misunderstandings?
I'm glad to receive your reply!

Thanks





[GLASSFISH-21410] Invalidating Session using POST via JAX-RS creates IOException Created: 09/Aug/15  Updated: 09/Aug/15

Status: Open
Project: glassfish
Component/s: grizzly-kernel, jax-rs, web_container
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: atrajano Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Given the following sample code

@Path("session")
@Stateless
public class Resource2 {

@GET
@Path("create")
public Response create(@Context HttpServletRequest req)

{ req.getSession().setAttribute("foo", "bar"); return Response.ok(req.getSession().getAttribute("foo")).build(); }

@POST
@Path("invalidate")
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
public Response invalidate(@Context HttpServletRequest req,
@FormParam("foo") String foo)

{ HttpSession session = req.getSession(false); Object foox = session.getAttribute(foo); session.invalidate(); return Response.temporaryRedirect(URI.create("http://slashdot.org/")).build(); }

}

I get the following stack trace

java.io.IOException
at org.glassfish.grizzly.http.io.InputBuffer.skip(InputBuffer.java:721)
at org.glassfish.grizzly.http.server.Request.skipPostBody(Request.java:2078)
at org.glassfish.grizzly.http.server.Request.parseRequestParameters(Request.java:2032)
at org.glassfish.grizzly.http.server.Request.getParameter(Request.java:1066)
at org.apache.catalina.connector.Request.getParameter(Request.java:1552)
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:448)
at org.jboss.weld.servlet.ConversationContextActivator.determineConversationId(ConversationContextActivator.java:182)
at org.jboss.weld.context.http.LazyHttpConversationContextImpl.checkContextInitialized(LazyHttpConversationContextImpl.java:93)
at org.jboss.weld.context.http.LazyHttpConversationContextImpl.destroy(LazyHttpConversationContextImpl.java:85)
at org.jboss.weld.context.http.HttpSessionContextImpl.destroy(HttpSessionContextImpl.java:59)
at org.jboss.weld.servlet.HttpContextLifecycle.sessionDestroyed(HttpContextLifecycle.java:152)
at org.jboss.weld.servlet.WeldInitialListener.sessionDestroyed(WeldInitialListener.java:133)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:910)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:854)
at org.apache.catalina.session.StandardSession.expire(StandardSession.java:842)
at org.apache.catalina.session.StandardSession.invalidate(StandardSession.java:1603)
at org.apache.catalina.session.StandardSessionFacade.invalidate(StandardSessionFacade.java:204)
at net.trajano.test.Resource2.invalidate(Resource2.java:44)

The invalidation seems to be asynchronous since the redirect still works. And it seems that the POST content stream was already closed by the time it reaches InputBuffer.skip.

Perhaps it shouldn't throw an IOException and instead if it is closed just silently not read any more data?






[GLASSFISH-21400] HTTP request hangs after blocking N number of requests Created: 31/Jul/15  Updated: 31/Jul/15

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: rahmanusta Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:
  • Windows 7 X64
  • JDK 1.8.0_60-ea

Tags: adoptajsr

 Description   

When blocking N number of HTTP requests on a Web component, many more requests doesn't response, request hangs.

I'm faced with this problem firstly in a JMS application. And, recorded a short video for that issue. In this case, N is 5.

Video URL: https://www.youtube.com/watch?v=j3Yss72c8Ko
Demo app: https://github.com/rahmanusta/jmstopicapp/archive/master.zip

I assumed it is about JMS firstly , but after tests we detected that does not seems about JMS. Just using the Servlet component I've seen similar issue.

For example if you block 10 HTTP GET request with Thread.sleep method, + requests to another Servlet won't give you response, request will hang. In this case N is not certain. But I've seen N generally between 7 and 10.



 Comments   
Comment by oleksiys [ 31/Jul/15 ]

Hi,

from what I see it's not a bug.
By default Glassfish has max 5 worker threads to process servlet requests. You block all of them, so no more HTTP request can be processed, that's why it just hangs.

You can increase the max worker threads count via admin gui or asadmin [1]

WBR.

[1]

asadmin set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=newvalue
Comment by rahmanusta [ 31/Jul/15 ]

Thanks oleksiys;

Do you think 5 is enough by default? I don't think that is enough.

Comment by oleksiys [ 31/Jul/15 ]

well, it depends on your requirements.
For sure it's not enough for production, but for testing IMO it should be ok, except usecases like yours, where threads are permanently blocked waiting for some event to arrive.

What do you think could be a good default value for the max thread pool size?

Comment by reza_rahman [ 31/Jul/15 ]

Thanks so much for promptly responding to this. I agree it's hard to really pin down a number for all situations.

Comment by rahmanusta [ 31/Jul/15 ]

I'm not sure but we can look at other application server's default value where the GF stands. What about 20?





[GLASSFISH-21269] Cancelled POST via AJP triggers busy wait by http-listener-1 thread Created: 11/Dec/14  Updated: 16/Dec/14

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: fuzzyBSc2 Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 12.04.4, Chrome browser with SPDY, apache 2.2.22-1ubuntu1.4, mod_jk 1:1.2.32-1, JAX-RS-based servlet with async-supported set to true



 Description   

A http worker thread goes into a busy conversation with Apache's mod_jk. Both the mod_jk thread and http-listener-1 pool thread are sitting at high CPU (together at around 100% for my single CPU test setup) and the listener appears never to return to the thread pool.

The conversation is as follows (captured using strace on the high CPU glassfish thread):
glassfish -> mod_jk: 0x41 0x42 0x00 0x03 0x06 0x1f 0xfa (Send me up to 8186 bytes of the body)
mod_jk -> glassfish: 0x12 0x34 0x00 0x00 (end of body)
This sequence of messages is repeated very very rapidly, suggesting that the glassfish side is not handling the end of body message from mod_jk.

The request is to an asynchronous JAX-RS method via the @Suspended annotation on an AsyncResponse parameter. The body of the message is JSON and is being deserialised in the container into another method parameter.

I think the following is occurring:

  • The browser sends a post request with a (in this case) 39260 byte body, but cancels it. This closes the spdy stream and causes mod_spdy to report an end of file to mod_jk internally within apache
  • Meanwhile glassfish begins to process the request. It has not read the whole request at this time. It invokes jackson to read the request which in turn reads from the InputStream that AjpHandlerFilter provides (see AjpHandlerFilter.handleRead in the stack trace).
  • The filter chain then appears to mishandle the AJP end of file message and instead continues to request more data. I didn't see AjpHandlerFilter in the read stack trace I captured so maybe it isn't intercepting correctly.

Restarting either apache or glassfish resolves the problem. This bug may be related to https://java.net/jira/browse/GLASSFISH-21202 which seems to have a similar trigger but results in memory growth rather than the high cpu and thread exhaustion I am seeing.

I have included stack traces for the write and read phases of the conversation as captured by jstack. I have also included strace output for the beginning of the problem, and to show how it recovers once Apache is restarted:
Thread 12324: (state = IN_NATIVE)

  • sun.nio.ch.FileDispatcherImpl.write0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
  • sun.nio.ch.SocketDispatcher.write(java.io.FileDescriptor, long, int) @bci=4, line=47 (Compiled frame)
  • sun.nio.ch.IOUtil.writeFromNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=93 (Compiled frame)
  • sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=12, line=51 (Compiled frame)
  • sun.nio.ch.SocketChannelImpl.write(java.nio.ByteBuffer) @bci=206, line=487 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(java.nio.channels.SocketChannel, java.nio.ByteBuffer) @bci=2, line=149 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.writeSimpleBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection, org.glassfish.grizzly.Buffer) @bci=130, line=133 (Compil
    ed frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.write(org.glassfish.grizzly.nio.transport.TCPNIOConnection, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.
    grizzly.WriteResult) @bci=53, line=680 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTemporarySelectorWriter.writeNow0(org.glassfish.grizzly.nio.NIOConnection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.WriteResult) @bci=14, line=65 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write0(org.glassfish.grizzly.nio.NIOConnection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.WriteResult, long, java.util.concurrent.TimeUnit) @bci=50, line=202 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.PushBackHandler, long, java.util.concurrent.TimeUnit) @bci=71, line=153 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.net.SocketAddress, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.MessageCloner) @bci=19, line=78 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorWriter.write(org.glassfish.grizzly.Connection, java.lang.Object, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.MessageCloner) @bci=11, line=58 (Compiled frame)
  • org.glassfish.grizzly.AbstractWriter.write(org.glassfish.grizzly.Connection, java.lang.Object, org.glassfish.grizzly.asyncqueue.WritableMessage, org.glassfish.grizzly.CompletionHandler) @bci=10, line=105 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleWrite(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=104, line=129 (Compiled frame)
  • org.glassfish.grizzly.filterchain.TransportFilter.handleWrite(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=20, line=191 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=111 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Compiled frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.write(java.lang.Object, java.lang.Object, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.asyncqueue.PushBackHandler, org.glassfish.grizzly.asyncqueue.MessageCloner, boolean) @bci=135, line=848 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.write(java.lang.Object) @bci=13, line=702 (Compiled frame)
  • org.glassfish.grizzly.http.ajp.AjpHandlerFilter.sendMoreDataRequestIfNeeded(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=114, line=504 (Compiled frame)
  • org.glassfish.grizzly.http.ajp.AjpHandlerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=18, line=197 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.read(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=83, line=351 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.read() @bci=65, line=695 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.blockingRead() @bci=4, line=1119 (Compiled frame)
  • org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead() @bci=18, line=95 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.fill(int) @bci=23, line=1143 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.read(byte[], int, int) @bci=74, line=353 (Interpreted frame)
  • org.apache.catalina.connector.InputBuffer.read(byte[], int, int) @bci=33, line=267 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteInputStream.read(byte[], int, int) @bci=97, line=270 (Interpreted frame)
  • org.glassfish.jersey.message.internal.EntityInputStream.read(byte[], int, int) @bci=7, line=101 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore() @bci=48, line=178 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(int[], int, int, int, int) @bci=268, line=1749 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName() @bci=64, line=1654 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(int) @bci=78, line=1499 (Compiled frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken() @bci=255, line=700 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.util.Map) @bci=133, line=416 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=143, line=312 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=3, line=26 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=59, line=525 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.lang.Object) @bci=5, line=106 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, com.fasterxml.jackson.core.JsonToken) @bci=53, line=242 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=26, line=118 (Compiled frame)
  • com.fasterxml.jackson.databind.ObjectReader._bind(com.fasterxml.jackson.core.JsonParser, java.lang.Object) @bci=128, line=1233 (Interpreted frame)
  • com.fasterxml.jackson.databind.ObjectReader.readValue(com.fasterxml.jackson.core.JsonParser) @bci=6, line=677 (Interpreted frame)
  • com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, java.io.InputStream) @bci=204, line=777 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(javax.ws.rs.ext.ReaderInterceptorContext, javax.ws.rs.ext.MessageBodyReader, org.glassfish.jersey.message.internal.EntityInputStream) @bci=61, line=251 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=292, line=229 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=1, line=73 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, org.glassfish.jersey.internal.PropertiesDelegate, java.io.InputStream, java.lang.Iterable, boolean) @bci=48, line=1124 (Interpreted frame)
  • org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], org.glassfish.jersey.internal.PropertiesDelegate) @bci=116, line=851 (Interpreted frame)
  • org.glassfish.jersey.server.ContainerRequest.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[]) @bci=8, line=270 (Interpreted frame)
  • org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide() @bci=60, line=96 (Interpreted frame)
  • org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(java.util.List) @bci=46, line=81 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues() @bci=4, line=121 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(java.lang.Object, javax.ws.rs.core.Request) @bci=3, line=136 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(java.lang.Object, org.glassfish.jersey.server.ContainerRequest) @bci=5, line=104 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(org.glassfish.jersey.server.internal.process.RequestProcessingContext, java.lang.Object) @bci=28, line=387 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(org.glassfish.jersey.server.internal.process.RequestProcessingContext) @bci=97, line=331 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(java.lang.Object) @bci=5, line=103 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime$1.run() @bci=57, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=4, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=1, line=267 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.util.concurrent.Callable, boolean) @bci=36, line=315 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(org.glassfish.jersey.internal.util.Producer, boolean) @bci=2, line=297 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.lang.Runnable) @bci=9, line=267 (Interpreted frame)
  • org.glassfish.jersey.process.internal.RequestScope.runInScope(org.glassfish.jersey.process.internal.RequestScope$Instance, java.lang.Runnable) @bci=23, line=297 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime.process(org.glassfish.jersey.server.ContainerRequest) @bci=164, line=254 (Interpreted frame)
  • org.glassfish.jersey.server.ApplicationHandler.handle(org.glassfish.jersey.server.ContainerRequest) @bci=5, line=1028 (Interpreted frame)
  • org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=119, line=372 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=381 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=362, line=344 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=39, line=221 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapper.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.Servlet) @bci=122, line=1682 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=694, line=318 (Interpreted frame)
  • org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=74, line=160 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.doInvoke(org.apache.catalina.Request, org.apache.catalina.Response, boolean) @bci=168, line=734 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=4, line=673 (Interpreted frame)
  • com.sun.enterprise.web.WebPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=99, line=99 (Interpreted frame)
  • org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=44, line=174 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.doService(org.glassfish.grizzly.http.server.Request, org.apache.catalina.connector.Request, org.glassfish.grizzly.http.server.Response, org.apache.catalina.connector.Response, boolean) @bci=425, line=415 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=183, line=282 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call() @bci=12, line=459 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=15, line=167 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.runService(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=48, line=201 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.doHandle(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=131, line=175 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=348, line=235 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Interpreted frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Interpreted frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=69, line=561 (Interpreted frame)
  • org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener, java.util.logging.Logger) @bci=9, line=112 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=6, line=117 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=3, line=56 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run() @bci=12, line=137 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() @bci=47, line=565 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() @bci=9, line=545 (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

The read phase of the cycle looks like this:
Thread 12324: (state = IN_NATIVE)

  • sun.nio.ch.FileDispatcherImpl.read0(java.io.FileDescriptor, long, int) @bci=0 (Compiled frame; information may be imprecise)
  • sun.nio.ch.SocketDispatcher.read(java.io.FileDescriptor, long, int) @bci=4, line=39 (Compiled frame)
  • sun.nio.ch.IOUtil.readIntoNativeBuffer(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=114, line=223 (Compiled frame)
  • sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher) @bci=29, line=192 (Compiled frame)
  • sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer) @bci=234, line=379 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.readSimpleByteBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection, java.nio.ByteBuffer) @bci=10, line=344 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOUtils.allocateAndReadBuffer(org.glassfish.grizzly.nio.transport.TCPNIOConnection) @bci=55, line=237 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer) @bci=22, line=618 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTemporarySelectorReader.readNow0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.ReadResult) @bci=25, line=65 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.ReadResult, org.glassfish.grizzly.Buffer, long, java.util.concurrent.TimeUnit) @bci=39, line=171 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read0(org.glassfish.grizzly.nio.NIOConnection, org.glassfish.grizzly.Interceptor, org.glassfish.grizzly.ReadResult, org.glassfish.grizzly.Buffer, long, java.util.concurrent.TimeUnit) @bci=20, line=141 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.Interceptor, long, java.util.concurrent.TimeUnit) @bci=52, line=113 (Compiled frame)
  • org.glassfish.grizzly.nio.tmpselectors.TemporarySelectorReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer, org.glassfish.grizzly.CompletionHandler, org.glassfish.grizzly.Interceptor) @bci=18, line=75 (Compiled frame)
  • org.glassfish.grizzly.AbstractReader.read(org.glassfish.grizzly.Connection, org.glassfish.grizzly.Buffer) @bci=12, line=72 (Compiled frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransportFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=57, line=77 (Compiled frame)
  • org.glassfish.grizzly.filterchain.TransportFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=20, line=173 (Compiled frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.read(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=83, line=351 (Compiled frame)
  • org.glassfish.grizzly.filterchain.FilterChainContext.read() @bci=65, line=695 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.blockingRead() @bci=4, line=1119 (Compiled frame)
  • org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead() @bci=18, line=95 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.fill(int) @bci=23, line=1143 (Compiled frame)
  • org.glassfish.grizzly.http.io.InputBuffer.read(byte[], int, int) @bci=74, line=353 (Interpreted frame)
  • org.apache.catalina.connector.InputBuffer.read(byte[], int, int) @bci=33, line=267 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteInputStream.read(byte[], int, int) @bci=97, line=270 (Interpreted frame)
  • org.glassfish.jersey.message.internal.EntityInputStream.read(byte[], int, int) @bci=7, line=101 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.loadMore() @bci=48, line=178 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.parseEscapedName(int[], int, int, int, int) @bci=268, line=1749 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.slowParseName() @bci=64, line=1654 (Interpreted frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parseName(int) @bci=78, line=1499 (Compiled frame)
  • com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken() @bci=255, line=700 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.util.Map) @bci=133, line=416 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=143, line=312 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=3, line=26 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=59, line=525 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, java.lan
    g.Object) @bci=5, line=106 (Interpreted frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext, com.fasterxml.jackson.core.JsonToken) @bci=53, line=242 (Compiled frame)
  • com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(com.fasterxml.jackson.core.JsonParser, com.fasterxml.jackson.databind.DeserializationContext) @bci=26, line=118 (Compiled frame)
  • com.fasterxml.jackson.databind.ObjectReader._bind(com.fasterxml.jackson.core.JsonParser, java.lang.Object) @bci=128, line=1233 (Interpreted frame)
  • com.fasterxml.jackson.databind.ObjectReader.readValue(com.fasterxml.jackson.core.JsonParser) @bci=6, line=677 (Interpreted frame)
  • com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, java.io.InputStream) @bci=204, line=777 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.invokeReadFrom(javax.ws.rs.ext.ReaderInterceptorContext, javax.ws.rs.ext.MessageBodyReader, org.glassfish.jersey.message.internal.EntityInputStream) @bci=61, line=251 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$TerminalReaderInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=292, line=229 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(javax.ws.rs.ext.ReaderInterceptorContext) @bci=1, line=73 (Interpreted frame)
  • org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed() @bci=46, line=149 (Interpreted frame)
  • org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], javax.ws.rs.core.MediaType, javax.ws.rs.core.MultivaluedMap, org.glassfish.jersey.internal.PropertiesDelegate, java.io.InputStream, java.lang.Iterable, boolean) @bci=48, line=1124 (Interpreted frame)
  • org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[], org.glassfish.jersey.internal.PropertiesDelegate) @bci=116, line=851 (Interpreted frame)
  • org.glassfish.jersey.server.ContainerRequest.readEntity(java.lang.Class, java.lang.reflect.Type, java.lang.annotation.Annotation[]) @bci=8, line=270 (Interpreted frame)
  • org.glassfish.jersey.server.internal.inject.EntityParamValueFactoryProvider$EntityValueFactory.provide() @bci=60, line=96 (Interpreted frame)
  • org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(java.util.List) @bci=46, line=81 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues() @bci=4, line=121 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(java.lang.Object, javax.ws.rs.core.Request) @bci=3, line=136 (Interpreted frame)
  • org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(java.lang.Object, org.glassfish.jersey.server.ContainerRequest) @bci=5, line=104 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(org.glassfish.jersey.server.internal.process.RequestProcessingContext, java.lang.Object) @bci=28, line=387 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(org.glassfish.jersey.server.internal.process.RequestProcessingContext) @bci=97, line=331 (Interpreted frame)
  • org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(java.lang.Object) @bci=5, line=103 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime$1.run() @bci=57, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=4, line=271 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors$1.call() @bci=1, line=267 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.util.concurrent.Callable, boolean) @bci=36, line=315 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(org.glassfish.jersey.internal.util.Producer, boolean) @bci=2, line=297 (Interpreted frame)
  • org.glassfish.jersey.internal.Errors.process(java.lang.Runnable) @bci=9, line=267 (Interpreted frame)
  • org.glassfish.jersey.process.internal.RequestScope.runInScope(org.glassfish.jersey.process.internal.RequestScope$Instance, java.lang.Runnable) @bci=23, line=297 (Interpreted frame)
  • org.glassfish.jersey.server.ServerRuntime.process(org.glassfish.jersey.server.ContainerRequest) @bci=164, line=254 (Interpreted frame)
  • org.glassfish.jersey.server.ApplicationHandler.handle(org.glassfish.jersey.server.ContainerRequest) @bci=5, line=1028 (Interpreted frame)
  • org.glassfish.jersey.servlet.WebComponent.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=119, line=372 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(java.net.URI, java.net.URI, javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=9, line=381 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) @bci=362, line=344 (Interpreted frame)
  • org.glassfish.jersey.servlet.ServletContainer.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=39, line=221 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapper.service(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.Servlet) @bci=122, line=1682 (Interpreted frame)
  • org.apache.catalina.core.StandardWrapperValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=694, line=318 (Interpreted frame)
  • org.apache.catalina.core.StandardContextValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=74, line=160 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.doInvoke(org.apache.catalina.Request, org.apache.catalina.Response, boolean) @bci=168, line=734 (Interpreted frame)
  • org.apache.catalina.core.StandardPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=4, line=673 (Interpreted frame)
  • com.sun.enterprise.web.WebPipeline.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=99, line=99 (Interpreted frame)
  • org.apache.catalina.core.StandardHostValve.invoke(org.apache.catalina.Request, org.apache.catalina.Response) @bci=44, line=174 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.doService(org.glassfish.grizzly.http.server.Request, org.apache.catalina.connector.Request, org.glassfish.grizzly.http.server.Response, org.apache.catalina.connector.Response, boolean) @bci=425, line=415 (Interpreted frame)
  • org.apache.catalina.connector.CoyoteAdapter.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=183, line=282 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call() @bci=12, line=459 (Interpreted frame)
  • com.sun.enterprise.v3.services.impl.ContainerMapper.service(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=15, line=167 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.runService(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=48, line=201 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpHandler.doHandle(org.glassfish.grizzly.http.server.Request, org.glassfish.grizzly.http.server.Response) @bci=131, line=175 (Interpreted frame)
  • org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=348, line=235 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=2, line=119 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(org.glassfish.grizzly.filterchain.FilterExecutor, org.glassfish.grizzly.filterchain.Filter, org.glassfish.grizzly.filterchain.FilterChainContext) @bci=38, line=284 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(org.glassfish.grizzly.filterchain.FilterChainContext, org.glassfish.grizzly.filterchain.FilterExecutor, int, int, org.glassfish.grizzly.filterchain.DefaultFilterChain$FiltersState) @bci=48, line=201 (Compiled frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(org.glassfish.grizzly.filterchain.FilterChainContext) @bci=50, line=133 (Interpreted frame)
  • org.glassfish.grizzly.filterchain.DefaultFilterChain.process(org.glassfish.grizzly.Context) @bci=103, line=112 (Interpreted frame)
  • org.glassfish.grizzly.ProcessorExecutor.execute(org.glassfish.grizzly.Context) @bci=53, line=77 (Interpreted frame)
  • org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=69, line=561 (Interpreted frame)
  • org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener, java.util.logging.Logger) @bci=9, line=112 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=6, line=117 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(org.glassfish.grizzly.Connection, org.glassfish.grizzly.IOEvent, org.glassfish.grizzly.IOEventLifeCycleListener) @bci=3, line=56 (Interpreted frame)
  • org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run() @bci=12, line=137 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork() @bci=47, line=565 (Interpreted frame)
  • org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run() @bci=9, line=545 (Interpreted frame)
  • java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

strace output on the glassfish http-listener-1 thread:
futex(0x7fc2b8219454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2b8219450,

{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x7fc2b8219428, FUTEX_WAKE_PRIVATE, 1) = 1
read(480, "\0224\4a\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 8027
read(484, "\0224\4[\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 9311
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
.... many more repeats until I restart apache ...
futex(0x7fc2b8219454, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fc2b8219450, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}

) = 1
futex(0x7fc2b8219428, FUTEX_WAKE_PRIVATE, 1) = 1
read(480, "\0224\4a\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 8027
read(484, "\0224\4[\2\4\0\10HTTP/1.1\0\0\34/base/resourc"..., 242337) = 9311
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4
write(484, "AB\0\3\6\37\372", 7) = 7
read(484, "\0224\0\0", 242337) = 4



 Comments   
Comment by fuzzyBSc2 [ 11/Dec/14 ]

Oops, sorry. The recovery portion of the strace when apache is restarted should be this:
read(484, 0x7fc261e29ff0, 242337) = -1 EAGAIN (Resource temporarily unavailable)
epoll_ctl(488, EPOLL_CTL_ADD, 484, {EPOLLIN, {u32=484, u64=9290625383554613732}}) = 0
epoll_ctl(488, EPOLL_CTL_MOD, 484, {EPOLLIN, {u32=484, u64=9290625383554613732}}) = 0
epoll_wait(488, {{EPOLLIN|EPOLLERR|EPOLLHUP,

{u32=484, u64=9290625383554613732}

}}, 4096, 30000) = 1
read(484, 0x7fc261e29ff0, 242337) = -1 ECONNRESET (Connection reset by peer)
write(441, "\1", 1) = 1
epoll_ctl(488, EPOLL_CTL_DEL, 484, {0, {u32=484, u64=484}}) = -1 ENOENT (No such file or directory)
close(484) = 0
epoll_wait(488, {}, 4096, 0) = 0
write(2, "[#|2014-12-09T15:42:00.719+1000|"..., 8192) = -1 EPIPE (Broken pipe)
— SIGPIPE (Broken pipe) @ 0 (0) —
write(2, "(DefaultFilterChain.java:133)\n\ta"..., 955) = -1 EPIPE (Broken pipe)
— SIGPIPE (Broken pipe) @ 0 (0) —

Comment by fuzzyBSc2 [ 11/Dec/14 ]

Note: I have tried applying the patches from GLASSFISH-21202 to see if they resolve this problem, but currently they appear to have been prepared for glassfish 4.0 only, not 4.1.

Comment by oleksiys [ 12/Dec/14 ]

pls. try this patch
https://dl.dropboxusercontent.com/u/7319744/glassfish-4.1/glassfishv41-patch.zip

Comment by fuzzyBSc2 [ 15/Dec/14 ]

Thanks,

I have applied this patch and retested. I have not been able to reproduce the problem with this patch in place

Comment by fuzzyBSc2 [ 16/Dec/14 ]

If you don't mind, just a procedural question: Is there any way I could trace this binary patch to a change in the source tree? Part of the open source policy of the company I work for is that I need to show how I would build the software from source if required.

Comment by oleksiys [ 16/Dec/14 ]

)

you can track it on github
https://github.com/GrizzlyNIO/grizzly-mirror/tree/glassfish41





[GLASSFISH-19582] Move AccessLog support on Grizzly level instead of WebContainer Created: 24/Jan/13  Updated: 24/Jan/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.0_b70
Fix Version/s: future release

Type: New Feature Priority: Major
Reporter: oleksiys Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Dependency
blocks GLASSFISH-18244 Web Service calls are not logged on s... Open

 Description   

Some HTTP based services (WebServices/EJB, appclient), which are using Grizzly HttpServer directly also require access log support.






[GLASSFISH-20618] Create source bundle for nucleus-grizzly-all Created: 10/Jun/13  Updated: 10/Jun/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: jjsnyder83 Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Per the summary.



 Comments   
Comment by jjsnyder83 [ 10/Jun/13 ]

Please create the source bundle for nucleus-grizzly-all.





[GLASSFISH-19444] Glassfish grizzly thread consuming large amounts of CPU time in NIO poll operation Created: 14/Dec/12  Updated: 16/Nov/13

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2.2
Fix Version/s: None

Type: Bug Priority: Major
Reporter: jfowler2 Assignee: oleksiys
Resolution: Unresolved Votes: 3
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

JDK version 1.6.0_37, Windows Server 2008 R2 Enterprise


Attachments: Java Archive File grizzly-framework-1.9.50.jar    

 Description   

We recently noticed an issue where our Glassifish server, after running successfully for several hours, would suddenly peg one of the CPUs at 100%. Our application becomes unresponsive during this time. After restarting, the problem will eventually happen again (usually after several hours).

I ran this command to see what the threads were doing:

asadmin generate-jvm-report --type=thread

In the resulting output, one thread looked highly suspicious (consuming orders of magnitude more CPU time than any other thread):

Thread Execution Information:

Thread "Grizzly-kernel-thread(1)" thread-id: 27 thread-state: RUNNABLE Running in native
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273)
at: sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255)
at: sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136)
at: sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
at: sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at: com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
at: com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
at: com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
at: java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at: java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at: java.lang.Thread.run(Thread.java:662)

Thread Synchronization Statistics:

Number of times this thread was blocked (to enter/reenter a Monitor): 4,520

Number of times this thread waited for a notification (i.e. it was in WAITING or TIMED_WAITING state): 0

Total CPU time for this thread: 2,753 seconds 703,125,000 nanoseconds.

User-level CPU time for this thread: 2,753 seconds 703,125,000 nanoseconds.

Object Monitors currently held or requested by this thread: []

Ownable Synchronizers (e.g. ReentrantLock and ReentrantReadWriteLock) held by this thread: []



 Comments   
Comment by oleksiys [ 17/Dec/12 ]

attaching a patch, which enables selector spin workaround for all operation systems.
Can you pls. apply this patch on GF 3.1.2.2 (rename this file to grizzly-framework.jar and copy it over existing file w/ the same name in glassfishv3/glassfish/modules folder).
Pls. let me know if you still see this issue w/ the patch applied, if yes - pls. check and attach the GF server.log file.

Thanks.

Comment by Stefan_Chossy [ 18/Mar/13 ]

Hi jfowler2,

did that patch fix your problems? We are facing pretty much the same issue but the patch didn't help any.

Comment by badbob.nsk [ 15/Nov/13 ]

Have an issue with the same symptoms.
Does this patch works?

Comment by oleksiys [ 16/Nov/13 ]

is it Glassfish 3.1.2.2 on Windows?

Comment by badbob.nsk [ 16/Nov/13 ]

It is glassfish 3.1.2.2 on SLES11.





[GLASSFISH-21107] When uploading a large file using an async servlet, after 30 seconds it throws an InterruptedByTimeoutException Created: 26/Jun/14  Updated: 07/Dec/15

Status: Open
Project: glassfish
Component/s: configuration, grizzly-kernel, web_container
Affects Version/s: 4.1_b07
Fix Version/s: None

Type: Bug Priority: Major
Reporter: raulgd Assignee: Chris Kasso
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7 64 bit, Java JDK 1.7.0u60 64 bit Glassfish 4.0.1b7 nightly build downloaded on 2014-06-25



 Description   

I created an async servlet for users to upload files, that way the inputstream is buffered without having to load the whole file in memory, as there can be very big files uploaded.

When trying out to upload a 4.6GB file, after 30 seconds, the servlet enters the onError(Throwable t) method with the exception:

Severe:   java.nio.channels.InterruptedByTimeoutException
	at org.apache.catalina.connector.InputBuffer.disableReadHandler(InputBuffer.java:324)
	at org.apache.catalina.connector.Request.asyncTimeout(Request.java:4418)
	at org.apache.catalina.connector.Request.processTimeout(Request.java:4469)
	at org.apache.catalina.connector.Request.access$000(Request.java:156)
	at org.apache.catalina.connector.Request$6.onTimeout(Request.java:4300)
	at org.glassfish.grizzly.http.server.Response$SuspendTimeout.onTimeout(Response.java:2131)
	at org.glassfish.grizzly.http.server.Response$DelayQueueWorker.doWork(Response.java:2180)
	at org.glassfish.grizzly.http.server.Response$DelayQueueWorker.doWork(Response.java:2175)
	at org.glassfish.grizzly.utils.DelayedExecutor$DelayedRunnable.run(DelayedExecutor.java:158)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Warning:   Context path from ServletContext:  differs from path from bundle: /
WARN:   WELD-000715: HttpContextLifecycle guard not set. The Servlet container is not fully compliant.

Now, I already configured Glassfish for longer timeouts on both Configurations-> server-config -> Network Config -> Network Listeners -> http-listener-1 -> HTTP So that timeout: -1, Connection upload timeout: 3600000 (1 hour), Request Timeout: -1, Max Post Size: -1

And on Configurations-> server-config -> Network Config -> Transports -> tcp so that Read Timeout: 3600000 (1 hour), Write Timeout: 3600000 (1 hour)

Now, AFAIK, because it's an NIO timeout exception, just by changing the tcp read and write timeout to 1 hour should avoid the issue, but still, the exception happens exactly 30 seconds later, as if the timeout configurations are being ignored.

I saw previous issues on grizzly ignoring the read and write timeouts, but they where closed as fixed in mid. 2013 so it shouldn't be an issue now.

For reference, here's my domain.xml:

<domain log-root="${com.sun.aas.instanceRoot}/logs" application-root="${com.sun.aas.instanceRoot}/applications" version="7">
  <security-configurations>
    <authentication-service default="true" name="adminAuth" use-password-credential="true">
      <security-provider name="spcrealm" type="LoginModule" provider-name="adminSpc">
        <login-module-config name="adminSpecialLM" control-flag="sufficient" module-class="com.sun.enterprise.admin.util.AdminLoginModule">
          <property name="config" value="server-config"></property>
          <property name="auth-realm" value="admin-realm"></property>
        </login-module-config>
      </security-provider>
      <security-provider name="filerealm" type="LoginModule" provider-name="adminFile">
        <login-module-config name="adminFileLM" control-flag="sufficient" module-class="com.sun.enterprise.security.auth.login.FileLoginModule">
          <property name="config" value="server-config"></property>
          <property name="auth-realm" value="admin-realm"></property>
        </login-module-config>
      </security-provider>
    </authentication-service>
    <authorization-service default="true" name="authorizationService">
      <security-provider name="simpleAuthorization" type="Simple" provider-name="simpleAuthorizationProvider">
        <authorization-provider-config support-policy-deploy="false" name="simpleAuthorizationProviderConfig"></authorization-provider-config>
      </security-provider>
    </authorization-service>
  </security-configurations>
  <managed-job-config></managed-job-config>
  <system-applications>
    <application context-root="" location="${com.sun.aas.installRootURI}/lib/install/applications/__admingui" directory-deployed="true" name="__admingui" object-type="system-admin">
      <module name="__admingui">
        <engine sniffer="web"></engine>
        <engine sniffer="security"></engine>
      </module>
    </application>
  </system-applications>
  <resources>
    <jdbc-resource pool-name="SamplePool" jndi-name="jdbc/sample"></jdbc-resource>
    <jdbc-resource pool-name="__TimerPool" jndi-name="jdbc/__TimerPool" object-type="system-admin"></jdbc-resource>
    <jdbc-resource pool-name="DerbyPool" jndi-name="jdbc/__default" object-type="system-all"></jdbc-resource>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.EmbeddedXADataSource" res-type="javax.sql.XADataSource" name="__TimerPool">
      <property name="databaseName" value="${com.sun.aas.instanceRoot}/lib/databases/ejbtimer"></property>
      <property name="connectionAttributes" value=";create=true"></property>
    </jdbc-connection-pool>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.ClientDataSource" is-isolation-level-guaranteed="false" res-type="javax.sql.DataSource" name="DerbyPool">
      <property name="PortNumber" value="1527"></property>
      <property name="Password" value="APP"></property>
      <property name="User" value="APP"></property>
      <property name="serverName" value="localhost"></property>
      <property name="DatabaseName" value="sun-appserv-samples"></property>
      <property name="connectionAttributes" value=";create=true"></property>
    </jdbc-connection-pool>
    <connector-connection-pool max-pool-size="250" steady-pool-size="1" name="jms/__defaultConnectionFactory-Connection-Pool" resource-adapter-name="jmsra" connection-definition-name="javax.jms.ConnectionFactory"></connector-connection-pool>
    <connector-resource pool-name="jms/__defaultConnectionFactory-Connection-Pool" jndi-name="jms/__defaultConnectionFactory" object-type="system-all-req"></connector-resource>
    <context-service jndi-name="concurrent/__defaultContextService" object-type="system-all"></context-service>
    <managed-executor-service jndi-name="concurrent/__defaultManagedExecutorService" object-type="system-all"></managed-executor-service>
    <managed-scheduled-executor-service jndi-name="concurrent/__defaultManagedScheduledExecutorService" object-type="system-all"></managed-scheduled-executor-service>
    <managed-thread-factory jndi-name="concurrent/__defaultManagedThreadFactory" object-type="system-all"></managed-thread-factory>
    <jdbc-connection-pool datasource-classname="org.apache.derby.jdbc.ClientDataSource" res-type="javax.sql.DataSource" name="SamplePool">
      <property name="DatabaseName" value="sample"></property>
      <property name="User" value="app"></property>
      <property name="Password" value="app"></property>
      <property name="URL" value="jdbc:derby://localhost:1527/sample"></property>
      <property name="PortNumber" value="1527"></property>
      <property name="serverName" value="localhost"></property>
    </jdbc-connection-pool>
  </resources>
  <servers>
    <server name="server" config-ref="server-config">
      <application-ref ref="__admingui" virtual-servers="__asadmin"></application-ref>
      <application-ref ref="VegosPortal" virtual-servers="server"></application-ref>
      <resource-ref ref="jdbc/__TimerPool"></resource-ref>
      <resource-ref ref="jdbc/__default"></resource-ref>
      <resource-ref ref="jms/__defaultConnectionFactory"></resource-ref>
      <resource-ref ref="concurrent/__defaultContextService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedExecutorService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedScheduledExecutorService"></resource-ref>
      <resource-ref ref="concurrent/__defaultManagedThreadFactory"></resource-ref>
      <resource-ref ref="jdbc/sample"></resource-ref>
    </server>
  </servers>
  <nodes>
    <node node-host="localhost" name="localhost-domain1" type="CONFIG" install-dir="${com.sun.aas.productRoot}"></node>
  </nodes>
  <configs>
    <config name="server-config">
      <system-property description="Port Number that JMS Service will listen for remote clients connection." name="JMS_PROVIDER_PORT" value="7676"></system-property>
      <http-service>
        <access-log></access-log>
        <virtual-server id="server" network-listeners="http-listener-1,http-listener-2"></virtual-server>
        <virtual-server id="__asadmin" network-listeners="admin-listener"></virtual-server>
      </http-service>
      <iiop-service>
        <orb use-thread-pool-ids="thread-pool-1"></orb>
        <iiop-listener id="orb-listener-1" port="3700" address="0.0.0.0" lazy-init="true"></iiop-listener>
        <iiop-listener id="SSL" port="3820" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </iiop-listener>
        <iiop-listener id="SSL_MUTUALAUTH" port="3920" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as" client-auth-enabled="true"></ssl>
        </iiop-listener>
      </iiop-service>
      <admin-service system-jmx-connector-name="system" type="das-and-server">
        <jmx-connector port="8686" address="0.0.0.0" auth-realm-name="admin-realm" name="system">
          <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </jmx-connector>
        <property name="adminConsoleContextRoot" value="/admin"></property>
        <property name="adminConsoleDownloadLocation" value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
        <property name="ipsRoot" value="${com.sun.aas.installRoot}/.."></property>
        <das-config></das-config>
      </admin-service>
      <connector-service></connector-service>
      <transaction-service tx-log-dir="${com.sun.aas.instanceRoot}/logs"></transaction-service>
      <batch-runtime-configuration></batch-runtime-configuration>
      <jms-service default-jms-host="default_JMS_host" type="EMBEDDED">
        <jms-host port="${JMS_PROVIDER_PORT}" host="localhost" name="default_JMS_host"></jms-host>
      </jms-service>
      <ejb-container>
        <ejb-timer-service></ejb-timer-service>
      </ejb-container>
      <web-container>
        <session-config>
          <session-manager>
            <manager-properties></manager-properties>
            <store-properties></store-properties>
          </session-manager>
          <session-properties></session-properties>
        </session-config>
      </web-container>
      <rest-config></rest-config>
      <cdi-service></cdi-service>
      <diagnostic-service></diagnostic-service>
      <security-service>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="admin-realm">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="file">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm" name="certificate"></auth-realm>
        <jacc-provider policy-provider="com.sun.enterprise.security.provider.PolicyWrapper" name="default" policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
          <property name="repository" value="${com.sun.aas.instanceRoot}/generated/policy"></property>
        </jacc-provider>
        <jacc-provider policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider" name="simple" policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
        <audit-module classname="com.sun.enterprise.security.ee.Audit" name="default">
          <property name="auditOn" value="false"></property>
        </audit-module>
        <message-security-config auth-layer="SOAP">
          <provider-config provider-type="client" provider-id="XWS_ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="client" provider-id="ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="XWS_ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
        </message-security-config>
        <message-security-config auth-layer="HttpServlet">
          <provider-config provider-type="server" provider-id="GFConsoleAuthModule" class-name="org.glassfish.admingui.common.security.AdminConsoleAuthModule">
            <request-policy auth-source="sender"></request-policy>
            <response-policy></response-policy>
            <property name="loginPage" value="/login.jsf"></property>
            <property name="loginErrorPage" value="/loginError.jsf"></property>
          </provider-config>
        </message-security-config>
        <property name="default-digest-algorithm" value="SHA-256"></property>
      </security-service>
      <java-config debug-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9009" system-classpath="" classpath-suffix="">
        <jvm-options>-XX:MaxPermSize=256m</jvm-options>
        <jvm-options>-client</jvm-options>
        <jvm-options>-Djava.awt.headless=true</jvm-options>
        <jvm-options>-Djdk.corba.allowOutputStreamSubclass=true</jvm-options>
        <jvm-options>-Djavax.management.builder.initial=com.sun.enterprise.v3.admin.AppServerMBeanServerBuilder</jvm-options>
        <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
        <jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
        <jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
        <jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
        <jvm-options>-Xmx1024m</jvm-options>
        <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
        <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
        <jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
        <jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
        <jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
        <jvm-options>-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.shell.remote,org.apache.felix.fileinstall</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.port=6666</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
        <jvm-options>-Dgosh.args=--nointeractive</jvm-options>
        <jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
        <jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
        <jvm-options>-Dfelix.fileinstall.log.level=2</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
        <jvm-options>-XX:NewRatio=2</jvm-options>
      </java-config>
      <network-config>
        <protocols>
          <protocol name="http-listener-1">
            <http request-timeout-seconds="-1" timeout-seconds="-1" connection-upload-timeout-millis="3600000" default-virtual-server="server" max-connections="1500">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="http-listener-2">
            <http default-virtual-server="server" max-connections="250">
              <file-cache></file-cache>
            </http>
            <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" ssl3-enabled="false" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-listener">
            <http default-virtual-server="__asadmin" max-connections="250" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="sec-admin-listener">
            <http default-virtual-server="__asadmin" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
            <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-http-redirect">
            <http-redirect secure="true"></http-redirect>
          </protocol>
          <protocol name="pu-protocol">
            <port-unification>
              <protocol-finder protocol="sec-admin-listener" name="http-finder" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder"></protocol-finder>
              <protocol-finder protocol="admin-http-redirect" name="admin-http-redirect" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder"></protocol-finder>
            </port-unification>
          </protocol>
        </protocols>
        <network-listeners>
          <network-listener port="80" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="8181" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="4848" protocol="pu-protocol" transport="tcp" name="admin-listener" thread-pool="admin-thread-pool"></network-listener>
        </network-listeners>
        <transports>
          <transport write-timeout-millis="3600000" byte-buffer-type="HEAP" read-timeout-millis="3600000" display-configuration="true" name="tcp"></transport>
        </transports>
      </network-config>
      <thread-pools>
        <thread-pool max-thread-pool-size="50" name="admin-thread-pool" max-queue-size="256"></thread-pool>
        <thread-pool name="http-thread-pool"></thread-pool>
        <thread-pool max-thread-pool-size="200" name="thread-pool-1"></thread-pool>
      </thread-pools>
      <monitoring-service>
        <module-monitoring-levels></module-monitoring-levels>
      </monitoring-service>
      <group-management-service>
        <failure-detection></failure-detection>
      </group-management-service>
      <availability-service></availability-service>
    </config>
    <config name="default-config">
      <http-service>
        <access-log></access-log>
        <virtual-server id="server" network-listeners="http-listener-1, http-listener-2">
          <property name="default-web-xml" value="${com.sun.aas.instanceRoot}/config/default-web.xml"></property>
        </virtual-server>
        <virtual-server id="__asadmin" network-listeners="admin-listener"></virtual-server>
      </http-service>
      <iiop-service>
        <orb use-thread-pool-ids="thread-pool-1"></orb>
        <iiop-listener id="orb-listener-1" port="${IIOP_LISTENER_PORT}" address="0.0.0.0"></iiop-listener>
        <iiop-listener id="SSL" port="${IIOP_SSL_LISTENER_PORT}" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as"></ssl>
        </iiop-listener>
        <iiop-listener id="SSL_MUTUALAUTH" port="${IIOP_SSL_MUTUALAUTH_PORT}" address="0.0.0.0" security-enabled="true">
          <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="s1as" client-auth-enabled="true"></ssl>
        </iiop-listener>
      </iiop-service>
      <admin-service system-jmx-connector-name="system">
        <jmx-connector port="${JMX_SYSTEM_CONNECTOR_PORT}" address="0.0.0.0" auth-realm-name="admin-realm" name="system">
          <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="glassfish-instance"></ssl>
        </jmx-connector>
        <property name="adminConsoleDownloadLocation" value="${com.sun.aas.installRoot}/lib/install/applications/admingui.war"></property>
        <das-config></das-config>
      </admin-service>
      <web-container>
        <session-config>
          <session-manager>
            <manager-properties></manager-properties>
            <store-properties></store-properties>
          </session-manager>
          <session-properties></session-properties>
        </session-config>
      </web-container>
      <ejb-container>
        <ejb-timer-service></ejb-timer-service>
      </ejb-container>
      <mdb-container></mdb-container>
      <jms-service addresslist-behavior="priority" default-jms-host="default_JMS_host" type="EMBEDDED">
        <jms-host port="${JMS_PROVIDER_PORT}" host="localhost" name="default_JMS_host"></jms-host>
      </jms-service>
      <log-service log-rotation-limit-in-bytes="2000000" file="${com.sun.aas.instanceRoot}/logs/server.log">
        <module-log-levels></module-log-levels>
      </log-service>
      <security-service>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="admin-realm">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/admin-keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.file.FileRealm" name="file">
          <property name="file" value="${com.sun.aas.instanceRoot}/config/keyfile"></property>
          <property name="jaas-context" value="fileRealm"></property>
        </auth-realm>
        <auth-realm classname="com.sun.enterprise.security.auth.realm.certificate.CertificateRealm" name="certificate"></auth-realm>
        <jacc-provider policy-provider="com.sun.enterprise.security.provider.PolicyWrapper" name="default" policy-configuration-factory-provider="com.sun.enterprise.security.provider.PolicyConfigurationFactoryImpl">
          <property name="repository" value="${com.sun.aas.instanceRoot}/generated/policy"></property>
        </jacc-provider>
        <jacc-provider policy-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyProvider" name="simple" policy-configuration-factory-provider="com.sun.enterprise.security.jacc.provider.SimplePolicyConfigurationFactory"></jacc-provider>
        <audit-module classname="com.sun.enterprise.security.ee.Audit" name="default">
          <property name="auditOn" value="false"></property>
        </audit-module>
        <message-security-config auth-layer="SOAP">
          <provider-config provider-type="client" provider-id="XWS_ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="client" provider-id="ClientProvider" class-name="com.sun.xml.wss.provider.ClientSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="dynamic.username.password" value="false"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="XWS_ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
          </provider-config>
          <provider-config provider-type="server" provider-id="ServerProvider" class-name="com.sun.xml.wss.provider.ServerSecurityAuthModule">
            <request-policy auth-source="content"></request-policy>
            <response-policy auth-source="content"></response-policy>
            <property name="encryption.key.alias" value="s1as"></property>
            <property name="signature.key.alias" value="s1as"></property>
            <property name="debug" value="false"></property>
            <property name="security.config" value="${com.sun.aas.instanceRoot}/config/wss-server-config-1.0.xml"></property>
          </provider-config>
        </message-security-config>
      </security-service>
      <transaction-service tx-log-dir="${com.sun.aas.instanceRoot}/logs" automatic-recovery="true"></transaction-service>
      <diagnostic-service></diagnostic-service>
      <java-config debug-options="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=${JAVA_DEBUGGER_PORT}" system-classpath="" classpath-suffix="">
        <jvm-options>-XX:MaxPermSize=256m</jvm-options>
        <jvm-options>-server</jvm-options>
        <jvm-options>-Djava.awt.headless=true</jvm-options>
        <jvm-options>-Djdk.corba.allowOutputStreamSubclass=true</jvm-options>
        <jvm-options>-XX:+UnlockDiagnosticVMOptions</jvm-options>
        <jvm-options>-Djava.endorsed.dirs=${com.sun.aas.installRoot}/modules/endorsed${path.separator}${com.sun.aas.installRoot}/lib/endorsed</jvm-options>
        <jvm-options>-Djava.security.policy=${com.sun.aas.instanceRoot}/config/server.policy</jvm-options>
        <jvm-options>-Djava.security.auth.login.config=${com.sun.aas.instanceRoot}/config/login.conf</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as</jvm-options>
        <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options>
        <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options>
        <jvm-options>-Djava.ext.dirs=${com.sun.aas.javaRoot}/lib/ext${path.separator}${com.sun.aas.javaRoot}/jre/lib/ext${path.separator}${com.sun.aas.instanceRoot}/lib/ext</jvm-options>
        <jvm-options>-Djdbc.drivers=org.apache.derby.jdbc.ClientDriver</jvm-options>
        <jvm-options>-DANTLR_USE_DIRECT_CLASS_LOADING=true</jvm-options>
        <jvm-options>-Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory</jvm-options>
        <jvm-options>-XX:NewRatio=2</jvm-options>
        <jvm-options>-Xmx1024m</jvm-options>
        <jvm-options>-Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command,org.apache.felix.fileinstall</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.port=${OSGI_SHELL_TELNET_PORT}</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.maxconn=1</jvm-options>
        <jvm-options>-Dosgi.shell.telnet.ip=127.0.0.1</jvm-options>
        <jvm-options>-Dgosh.args=--noshutdown -c noop=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.dir=${com.sun.aas.installRoot}/modules/autostart/</jvm-options>
        <jvm-options>-Dfelix.fileinstall.poll=5000</jvm-options>
        <jvm-options>-Dfelix.fileinstall.log.level=3</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.new.start=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.bundles.startTransient=true</jvm-options>
        <jvm-options>-Dfelix.fileinstall.disableConfigSave=false</jvm-options>
      </java-config>
      <availability-service>
        <web-container-availability></web-container-availability>
        <ejb-container-availability sfsb-store-pool-name="jdbc/hastore"></ejb-container-availability>
        <jms-availability></jms-availability>
      </availability-service>
      <network-config>
        <protocols>
          <protocol name="http-listener-1">
            <http default-virtual-server="server">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="http-listener-2">
            <http default-virtual-server="server">
              <file-cache></file-cache>
            </http>
            <ssl classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" ssl3-enabled="false" cert-nickname="s1as"></ssl>
          </protocol>
          <protocol name="admin-listener">
            <http default-virtual-server="__asadmin" max-connections="250">
              <file-cache></file-cache>
            </http>
          </protocol>
          <protocol security-enabled="true" name="sec-admin-listener">
            <http default-virtual-server="__asadmin" encoded-slash-enabled="true">
              <file-cache></file-cache>
            </http>
            <ssl client-auth="want" classname="com.sun.enterprise.security.ssl.GlassfishSSLImpl" cert-nickname="glassfish-instance"></ssl>
          </protocol>
          <protocol name="admin-http-redirect">
            <http-redirect secure="true"></http-redirect>
          </protocol>
          <protocol name="pu-protocol">
            <port-unification>
              <protocol-finder protocol="sec-admin-listener" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder" name="http-finder"></protocol-finder>
              <protocol-finder protocol="admin-http-redirect" classname="org.glassfish.grizzly.config.portunif.HttpProtocolFinder" name="admin-http-redirect"></protocol-finder>
            </port-unification>
          </protocol>
        </protocols>
        <network-listeners>
          <network-listener port="${HTTP_LISTENER_PORT}" protocol="http-listener-1" transport="tcp" name="http-listener-1" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="${HTTP_SSL_LISTENER_PORT}" protocol="http-listener-2" transport="tcp" name="http-listener-2" thread-pool="http-thread-pool"></network-listener>
          <network-listener port="${ASADMIN_LISTENER_PORT}" protocol="pu-protocol" transport="tcp" name="admin-listener" thread-pool="http-thread-pool"></network-listener>
        </network-listeners>
        <transports>
          <transport name="tcp"></transport>
        </transports>
      </network-config>
      <thread-pools>
        <thread-pool name="http-thread-pool"></thread-pool>
        <thread-pool max-thread-pool-size="200" name="thread-pool-1"></thread-pool>
        <thread-pool max-thread-pool-size="50" name="admin-thread-pool" max-queue-size="256"></thread-pool>
      </thread-pools>
      <group-management-service>
        <failure-detection></failure-detection>
      </group-management-service>
      <system-property description="Port Number that JMS Service will listen for remote clients connection." name="JMS_PROVIDER_PORT" value="27676"></system-property>
      <system-property name="ASADMIN_LISTENER_PORT" value="24848"></system-property>
      <system-property name="HTTP_LISTENER_PORT" value="28080"></system-property>
      <system-property name="HTTP_SSL_LISTENER_PORT" value="28181"></system-property>
      <system-property name="IIOP_LISTENER_PORT" value="23700"></system-property>
      <system-property name="IIOP_SSL_LISTENER_PORT" value="23820"></system-property>
      <system-property name="IIOP_SSL_MUTUALAUTH_PORT" value="23920"></system-property>
      <system-property name="JMX_SYSTEM_CONNECTOR_PORT" value="28686"></system-property>
      <system-property name="OSGI_SHELL_TELNET_PORT" value="26666"></system-property>
      <system-property name="JAVA_DEBUGGER_PORT" value="29009"></system-property>
      <monitoring-service>
        <module-monitoring-levels></module-monitoring-levels>
      </monitoring-service>
    </config>
  </configs>
  <property name="administrative.domain.name" value="domain1"></property>
  <secure-admin enabled="true" special-admin-indicator="8666d952-1f41-4ba9-a0c0-ec7396d974b7">
    <secure-admin-principal dn="CN=localhost,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
    <secure-admin-principal dn="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"></secure-admin-principal>
  </secure-admin>
  <clusters></clusters>
  <applications>
    <application context-root="/vegos" location="file:/D:/NetbeansProjects/VegosPortal/target/VegosPortal-1.0/" name="VegosPortal" directory-deployed="true" object-type="user">
      <property name="archiveType" value="war"></property>
      <property name="appLocation" value="file:/D:/NetbeansProjects/VegosPortal/target/VegosPortal-1.0/"></property>
      <property name="org.glassfish.ejb.container.application_unique_id" value="91995825334714368"></property>
      <property name="defaultAppName" value="VegosPortal-1.0"></property>
      <module name="VegosPortal">
        <engine sniffer="ejb"></engine>
        <engine sniffer="security"></engine>
        <engine sniffer="weld"></engine>
        <engine sniffer="web"></engine>
      </module>
    </application>
  </applications>
</domain>

I tried the exact same code in WildFly 8.1.0 and works perfectly, so I know it's grizzly's NIO configuration.



 Comments   
Comment by raulgd [ 10/Dec/14 ]

This problem doesn't happen in Glassfish 4.1b13 with JDK 8u25 anymore.

Only there are 2 things to specify, first, the http-thread-pool timeout also has to be set to the same timeout as the tcp read timeout and connection upload timeout.

The other one, on the async servlet, right after initiating the async context with:

AsyncContext context = request.startAsync(); 

You have to set the timeout for the context with:

context.setTimeout(3600000);

Just replace the 3600000 with whatever timeout you set.

It would be ideal to just have the servlet get the timeout specified from the domain configuration, but at least is working now setting it by hand.

Comment by Masoud Kalali [ 07/Dec/15 ]

Considering that I am no longer active in GlassFish space I am assigning all the tickets to Chris Kasso in Java EE/ Application Servers team and he can reassign them as appropriate.





[GLASSFISH-7109] [JDK ISSUE] GlassFish performance degradation caused by invoking setSoLinger/setReuseAddress Created: 30/Jan/09  Updated: 06/Mar/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: v2.1
Fix Version/s: not determined

Type: Bug Priority: Minor
Reporter: jfarcand Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All
URL: http://bugs.sun.com/view_bug.do?bug_id=6799574


Issuezilla Id: 7,109
Status Whiteboard:

V2.1.1_exclude


 Description   

[Umbrella bug for JDK 5/6/7 issue 6799574)

Grizzly is suffering performance degradation when setSoLinger and setReuseAddess
starts throwing the following exception:

[#|2009-01-26T00:33:56.325-0800|WARNING|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=SelectorReaderThread-8084;_RequestID=11ae0030-c392-4217-8408-cfa7efe0a879;|setSoLinger
exception
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:261)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:166)
at sun.nio.ch.SocketAdaptor.setIntOption(SocketAdaptor.java:296)
at sun.nio.ch.SocketAdaptor.setSoLinger(SocketAdaptor.java:331)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketOptions(SelectorThread.java:1893)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.registerNewChannels(SelectorReadThread.java:93)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.startEndpoint(SelectorReadThread.java:121)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)

#]

[#|2009-01-26T00:33:56.327-0800|WARNING|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=SelectorReaderThread-8084;_RequestID=11ae0030-c392-4217-8408-cfa7efe0a879;|setReuseAddress
exception
java.net.SocketException: Invalid argument
at sun.nio.ch.Net.setIntOption0(Native Method)
at sun.nio.ch.Net.setSocketOption(Net.java:261)
at sun.nio.ch.SocketChannelImpl.setOption(SocketChannelImpl.java:166)
at sun.nio.ch.SocketAdaptor.setBooleanOption(SocketAdaptor.java:286)
at sun.nio.ch.SocketAdaptor.setReuseAddress(SocketAdaptor.java:399)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.setSocketOptions(SelectorThread.java:1910)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.registerNewChannels(SelectorReadThread.java:93)
at
com.sun.enterprise.web.connector.grizzly.SelectorReadThread.startEndpoint(SelectorReadThread.java:121)
at
com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)

#]

This has been discussed here:

https://glassfish.dev.java.net/servlets/ReadMsg?listName=users&msgNo=26597

One user reported: ..that these errors are harmless, however, as you can see
below, the throughput of my application was reduced by 50% in the minutes
surrounding the spurt of errors:

minute requests setSoLinger/setReuseAddress exceptions
----- ----- -
14:23 7620 0
14:24 10063 0
14:25 9714 0
14:26 8847 0
14:28 7370 0
14:29 9787 0
14:30 9104 0
14:31 8171 0
14:32 4066 15 errors in two groups: 8 @ 14:32:33 and 7 @ 14:32:58
14:33 6908 0
14:34 10463 0
14:35 9870 0
14:36 8236 0
14:37 8685 0
14:38 8098 0

My application constantly serves 200-350 requests per second - and has been
doing so for 1 week now. This is the only incident that any errors have been
thrown since the application began operation.



 Comments   
Comment by jfarcand [ 30/Jan/09 ]

Trying to find a workaround. Disabling setSoLinger (setting it to false) cause
some performance issue, but at least the exception is not reproducible.

Comment by jfarcand [ 15/Apr/09 ]

JDK issue

http://monaco.sfbay.sun.com/detail.jsf?cr=6799574

Re-assign to docs.

Comment by Paul Davies [ 13/May/09 ]

Description of this issue added to the Release Notes. Review pending.

Comment by kumara [ 01/Sep/09 ]

Changing version from 9.1.1 to v2.1 to reflect new name/version.

Comment by Paul Davies [ 30/Sep/09 ]

To be added to the v3 Release Notes

Comment by jagadesh [ 15/Oct/09 ]

Will not be fixed for V2.1.1

Comment by Paul Davies [ 22/Oct/09 ]

Reassigned to grisdal.

Comment by Gail Risdal [ 08/Dec/09 ]

Added to v3 Release Notes.

Comment by Gail Risdal [ 09/Dec/09 ]

Closed prematurely. Reopened to enable code fix.

Comment by Gail Risdal [ 09/Dec/09 ]

Reassigned to enable code fix.

Comment by Paul Davies [ 26/May/11 ]

This issue was noted as required in the Release Notes. I have changed the subcomponent to enable engineering to resolve or close this bug as necessary.

Comment by Tom Mueller [ 06/Mar/12 ]

Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.





[GLASSFISH-18257] On URI decode exception the access log is not used Created: 26/Jan/12  Updated: 26/Jan/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1_b43
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: benjamin_m Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux x86_64


Tags: grizzly, logging

 Description   

When Grizzly throws an "Invalid URI character encoding" exception, the URI is part of the stack trace but the HTTP request info isn't saved on the access log.
This is a problem if the request URI makes it obvious that the requester is trying an exploit/vulnerability.
Without the access log used, there is no way of seeing the IP/hostname of the requester to identify the source of this attack attempt.



 Comments   
Comment by oleksiys [ 26/Jan/12 ]

Can you pls. check if it's still the case in the latest promoted Glassfish 3.1.2
http://dlc.sun.com.edgesuite.net/glassfish/3.1.2/promoted/glassfish-3.1.2-b19.zip

Thanks.

Comment by benjamin_m [ 26/Jan/12 ]

After trying to replicate in a VM with the suggested build, a similar error is not thrown.
To specify, here is the stack trace of the URI decoding issue which is not being logged in the access log.

[#|2012-01-26T07:50:40.472+0100|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=23;_ThreadName=Thread-1;|Internal Server error: /../../../../../../../../boot.ini
java.io.IOException: Invalid URI character encoding
at com.sun.grizzly.util.http.HttpRequestURIDecoder.decode(HttpRequestURIDecoder.java:101)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:185)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:822)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1013)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:619)

Comment by oleksiys [ 26/Jan/12 ]

Just to make sure, you didn't see the record in the access.log related to the corrupted request?

Comment by benjamin_m [ 26/Jan/12 ]

Exactly.
Because the exception is thrown on URI decode, Grizzly gives up there and nothing is written to the access log.
Which becomes very problematic when you have a rogue client trying some exploit like it's the case here.

Comment by oleksiys [ 26/Jan/12 ]

I agree, just wanted to make sure this is true for the latest 3.1.2 build.





[GLASSFISH-19372] ClientAbort Exception Default Servlet Created: 27/Nov/12  Updated: 27/Nov/12

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_b05
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: peppeme Assignee: oleksiys
Resolution: Unresolved Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

linux centos


Tags: brokenpipe, clientabortexception, grizzly

 Description   

I can't reproduce the issue but I get many and many Client Abort Exception:
It seems a client side communication interruption but
The server.log is filled by these becoming unreadable..

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:430)
at com.sun.grizzly.util.buf.ByteChunk.flushBuffer(ByteChunk.java:458)
at com.sun.grizzly.util.buf.ByteChunk.append(ByteChunk.java:380)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:455)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:442)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:160)
at org.apache.catalina.servlets.DefaultServlet.copy(DefaultServlet.java:2010)
at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:1040)
at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:466)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:343)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:217)
at filter.AdwFilter.doFilter(AdwFilter.java:206)
...
...
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
at






[GLASSFISH-15803] GUI console and instance are not available after http-listener-1.port is changed into 4848 Created: 02/Feb/11  Updated: 02/Dec/11

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1_b38
Fix Version/s: 4.0

Type: Bug Priority: Minor
Reporter: tetyanac Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows



 Description   

I changed setting configs.config.server-config.network-config.network-listeners.network-listener.http-listener-1.port=4848 via GUI admin console and after that GUI console is not available and GF instance as well. Why is it possible to change the port to already occupied with process one?



 Comments   
Comment by Anissa Lam [ 02/Feb/11 ]

The enforcement of not assigning conflicting port # will need to happen at the backend.
Assigning to 'admin'.

Comment by Tom Mueller [ 18/Feb/11 ]

Validation of network listener config changes happens within Grizzly.
Assigning to the grizzly category.

Comment by oleksiys [ 18/Feb/11 ]

reassigning to Justin.





[GLASSFISH-20955] Building grizzly-config with JDK 1.8 fails Created: 15/Jan/14  Updated: 24/Sep/15

Status: Open
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 4.1
Fix Version/s: None

Type: Bug Priority: Minor
Reporter: Ryan Lubke Assignee: oleksiys
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to GLASSFISH-21427 Build against Java 8 Open
Tags: jdk8

 Description   

Compiling nucleus/grizzly/config using JDK 1.8.0b121 fails:

Failed to execute goal org.glassfish.build:command-security-maven-plugin:1.0.6:check (default-check) on project grizzly-config: Error loading config beans: java.lang.RuntimeException: org.apache.maven.plugin.MojoExecutionException: Error analyzing java/lang/Object: IllegalArgumentException -> [Help 1]






Generated at Sun Feb 07 06:21:31 UTC 2016 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.