[GLASSFISH-18446] JK listener with Apache + mod_ajp_proxy causes truncated downloads Created: 03/Mar/12  Updated: 20/Dec/16  Resolved: 06/Mar/12

Status: Resolved
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_dev
Fix Version/s:

Type: Bug Priority: Blocker
Reporter: buddypine Assignee: oleksiys
Resolution: Fixed Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File ajp_trunc.cap     Text File chrome_dump.txt     Java Archive File grizzly-http-ajp.jar     Java Archive File grizzly-http.jar     Text File mod_jk.log     Text File wget_dump.txt    
Tags: 3_1_2-next, apache, jk, jl-listener, mod_ajp_proxy


Since upgrading to 3.1.2 I am experiencing truncated file downloads from my app with GF behind Apache using a lk-listener and Apache mod_ajp_proxy. It seems to be ajp related as downloads complete fully when accessing app directly on GF http port.

With jk downloads truncate at 256Kb, the original file is 700Kb.

This appears in the Apache logs:

ajp_check_msg_header() got bad signature 28e9
ajp_ilink_receive() received bad header
ajp_read_header: ajp_ilink_receive failed
(120007)APR does not understand this error code: proxy: dialog to (ww.mydomain.com) failed


asadmin create-network-listener --jkenabled true --protocol http-listener-1 --listenerport 8009 jk-listener
ProxyPass / ajp://www.mydomain.com:8009/
ProxyPassReverse / ajp://www.mydomain.com:8009/

Packet capture of failed download attached.

Comment by buddypine [ 03/Mar/12 ]

I have also tried with mod_jk Tomcat Connector and downloads are truncated at 512Kb.
Attaching error log from mod_jk.

Comment by buddypine [ 03/Mar/12 ]

mod_jk log file showing error

Comment by jsl123 [ 03/Mar/12 ]

Similar problem, webapp returning jsp pages works fine in 3.1.1 and when connecting to glassfish directly, but when you use mod_proxy_ajp, browsers time out waiting for the remaining data

Comment by oleksiys [ 05/Mar/12 ]

to buddypine:
sorry, looks like a bug on our side.
pls. try the attached patch (copy it to glassfish3/glassfish/modules folder)

Comment by oleksiys [ 05/Mar/12 ]

to jsl123:
can you pls. also try the patch, if it doesn't help - pls. let us know how we can reproduce the issue.

Comment by christoph.bernhofer [ 05/Mar/12 ]

Patch doesn't work for me. Have the same issue.
Files still truncated

--> Filesizes compared:
theme.css: should be 31.3 kb
primefaces.css: should be 41 kb
jquery.js: should be 299.8 kb
primefaces.js: should be 179.5 kb

GET adminLogin.jsf?loc=en-GB 200 OK 3.1 KB 16ms
GET theme.css.jsf?ln=primefaces-aristo 200 OK 31.3 KB 11ms
GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 41 KB 12ms
GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 1.4 KB 21ms
GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 299.8 KB 22ms
GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 179.5 KB 22ms
GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 4.7 KB 205ms
GET iPCP.css 304 Not Modified 2 KB 15ms

8 Anfragen

562.8 KB

(559.6 KB vom Cache)

GET adminLogin.jsf?loc=en-GB 200 OK 1.1 KB 16ms
GET theme.css.jsf?ln=primefaces-aristo 200 OK 0 16ms
GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 0 16ms
GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 433 B 5ms
GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 0 47ms
GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 0 32ms
GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 1.9 KB 8ms
GET iPCP.css 304 Not Modified 733 B 16ms
GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 0 16ms

9 Anfragen

4.1 KB

(3 KB vom Cache)

Comment by buddypine [ 05/Mar/12 ]

@oleksiys - thanks the patch does work for me.

Comment by oleksiys [ 06/Mar/12 ]

fixed in Grizzly

Comment by oleksiys [ 06/Mar/12 ]


can you pls. create a separate issue and provide more data, logs etc. Cause w/ the patch I'm not able to reproduce truncate issue (tried both secure and plain connections).

Comment by oleksiys [ 06/Mar/12 ]

fix will be available in the next 3.1.x release

Comment by rainerfrey [ 06/Mar/12 ]

Will that be a regular several-month release cycle, or is there a chance for a quick release with this fix? AFAIS this is a general problem with Glassfish and JK, and in that case it will keep us from upgrading to 3.1.2.

Or can the circumstances be narrowed down to more special configuration than using a jk-listener in general?

Comment by oleksiys [ 06/Mar/12 ]

Normally we upgrade Grizzly as part of GF release cycle or as patch (which we provided).
I can try to investigate if we can make Grizzly upgrade using GF update tool...

Comment by oleksiys [ 06/Mar/12 ]

changing priority and module

Comment by oleksiys [ 06/Mar/12 ]


guys, wanted to ask you to give me more info on your usecases, so we can fix and deliver updates asap.


Comment by christoph.bernhofer [ 06/Mar/12 ]

Hi oleksiys

My Application is a JSF2 WAR Application behind Apache 2.2. mod_proxy_ajp.

Created with
asadmin create-threadpool --minthreadpoolsize 5 --maxthreadpoolsize 150 jk-thread-pool
asadmin create-network-listener --protocol http-listener-1 --listenerport 8091 --jkenabled true --threadpool jk-thread-pool jk-connector


ProxyPass /iPCP ajp://localhost:8091/iPCP
ProxyPassReverse /iPCP ajp://localhost:8091/iPCP

works flawless with glassfish 3.1.1 in same configuration

updated now Apache ond Debian squeeze x64 to apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_perl/2.0.4 Perl/v5.10.1 configured

but this hasn't changed anything.

error.log apache2:

ajp_check_msg_header() got bad signature 6575
[Mon Mar 05 14:14:22 2012] [error] ajp_ilink_receive() received bad header
[Mon Mar 05 14:14:22 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Mon Mar 05 14:14:22 2012] [error] (120007)APR does not understand this error code: proxy: dialog to (localhost) failed
[Mon Mar 05 14:14:23 2012] [error] [client] File does not exist: /var/www/favicon.ico
[Mon Mar 05 14:14:23 2012] [error] [client] File does not exist: /var/www/favicon.ico
[Mon Mar 05 14:14:23 2012] [error] [client] File does not exist: /var/www/favicon.ico
[Mon Mar 05 14:15:06 2012] [error] [client] proxy: error processing end, referer:
[Mon Mar 05 14:15:06 2012] [error] [client] proxy: error processing body. Client aborted connection., referer:
[Mon Mar 05 14:15:16 2012] [error] ajp_check_msg_header() got bad signature 6575
[Mon Mar 05 14:15:16 2012] [error] ajp_ilink_receive() received bad header
[Mon Mar 05 14:15:16 2012] [error] ajp_read_header: ajp_ilink_receive failed

which more info do you need?


Comment by oleksiys [ 06/Mar/12 ]

can you pls. attach the app, or better very reduced version of the app, so I'll be able to debug the problem.


Comment by rainerfrey [ 07/Mar/12 ]

An uĆ¼date via updatetool would be great! Thank you.

Comment by christoph.bernhofer [ 07/Mar/12 ]

Hi oleksiys

I just tried withthe Primefces showcase App.
Same issue. Glassfish 3.1.1. same configuration, same Apache --> works flawless
Glassfish 3.1.2 don't work with proxy, Without proxy it works
I tried to access the App via mod_ssl at Apache at 443 and ajp behind as written above.

Here is the Link to the sample war file:




Comment by jsl123 [ 07/Mar/12 ]

Hi guys, sorry didn't get updates about this and was distracted. Trying patch now. Will report on progress.

Comment by oleksiys [ 07/Mar/12 ]

Hi Christoph,

what exactly doesn't work in that app? Cause I'm able to load it in a browser and go throw couple of menu items w/o problems.


Comment by christoph.bernhofer [ 07/Mar/12 ]

Hi oleksiys

Now I get a complete timeout of the whole App. App is not loaded via proxy.

Can you plese tell me your exact versions for Apache, Java...
have you tried also via SSL?



Comment by oleksiys [ 07/Mar/12 ]

I tried mod_ssl and plain, both work fine.

Apache 2.2.17:

ProxyPass /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT retry=0 ttl=30
ProxyPassReverse /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT
ProxyPreserveHost on


java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

Comment by jsl123 [ 07/Mar/12 ]

OK, further investigation.

Ubuntu 11.10 x64
Apache 2.2.20-1ubuntu1.2
Glassfish 3.1.2 with grizzly patch

If I disable mod_deflate in apache it works fine, pages appear as normal, however as soon as I enable mod_deflate, things break.

With mod_deflate:
Using standard wget works fine - no compression used
Using wget with "accept-encoding:gzip" header set works.
Using firefox/chrome doesn't work. Firefox silently shows a blank page however chrome gives us a clue and reports
"Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error

This all works fine in 3.1.1

Looking at the net traffic (attached) they are different both in content (should be the same) and also size, the chunked version is bigger. The wget compressed data expands into the same as if compression isn't used.

So it looks like a different bug

Comment by jsl123 [ 07/Mar/12 ]

tcpdump of chrome page fetch

Comment by jsl123 [ 07/Mar/12 ]

tcp dump of wget

Comment by christoph.bernhofer [ 07/Mar/12 ]

Get the same Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error
Deactiveated now mod_deflate --> works

Comment by oleksiys [ 07/Mar/12 ]

thank you guys, i'll investigate that and let you know asap.

Comment by oleksiys [ 07/Mar/12 ]

additional patch

Comment by oleksiys [ 07/Mar/12 ]

guys, can you pls. try to apply another jar (attached) additionally to one which had been attached before?

Comment by christoph.bernhofer [ 07/Mar/12 ]

Wworks for me with an without mod_deflate!

Great Work, thank you

Comment by jsl123 [ 07/Mar/12 ]

Looks to work for me as well, with mod_deflate.

Will do more testing and post if I find anything else..

good work, Cheers

Comment by jsl123 [ 08/Mar/12 ]

One thing I've noticed and don't think it is expected, but wanted to check before I file a bug? With 3.1.2 when we use HttpServletRequest.getRemoteAddr/Host it returns when using mod_proxy_ajp and apache. With 3.1.1 it returns the address of the client.
I assume this is a bug

Comment by oleksiys [ 08/Mar/12 ]

ok, will check that.


Comment by oleksiys [ 09/Mar/12 ]

fix for getRemoteXXX()

Comment by oleksiys [ 09/Mar/12 ]

attached the grizzly-http-ajp.jar with the getRemoteAddr/Host fix.

Comment by jsl123 [ 09/Mar/12 ]

Cheers, looks to have fixed it. Would it be possible to push this via the update-tool?


Comment by oleksiys [ 09/Mar/12 ]

thank you for confirming!

w/ update tool, we're still investigating this...

Comment by AndrewJames [ 18/Mar/12 ]

I faced the same issue - very happy to see this patch, which resolved the problem - thank you. A life-saver (figuratively speaking).

Comment by oleksiys [ 29/Mar/12 ]

internal bugdb# 13902008.

Comment by oleksiys [ 25/Apr/12 ]

update the patch to fix issue

Comment by jsl123 [ 02/May/12 ]

Although the download truncation seems to have been fixed, it looks like we are getting an upload truncation issue. One of my clients is posting files to our server which all worked fine with 3.1.1, but after upgrading and applying the above patches, it bombs out part way through the transfer.

Small files work, but larger ones don't. In apache I get
[Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(273): proxy: data to read (max 8186 at 4)
[Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(288): proxy: got 1448 bytes of data
[Tue May 01 15:06:32 2012] [error] ajp_check_msg_header() got bad signature 4854
[Tue May 01 15:06:32 2012] [error] ajp_ilink_receive() received bad header
[Tue May 01 15:06:32 2012] [error] ajp_read_header: ajp_ilink_receive failed
[Tue May 01 15:06:32 2012] [error] (120007)APR does not understand this error code: proxy: read response failed from (localhost)
[Tue May 01 15:06:32 2012] [debug] proxy_util.c(2029): proxy: AJP: has released connection for (localhost)
[Tue May 01 15:06:32 2012] [debug] mod_headers.c(756): headers: ap_headers_output_filter()
[Tue May 01 15:06:32 2012] [debug] mod_dumpio.c(142): mod_dumpio: dumpio_out

which then results in a 500 error from the server ultimately being returned.


Comment by oleksiys [ 04/May/12 ]

Did you try the latest grizzly-http-ajp.jar I attached recently?
Does this problem occur all the time you try to upload files? Do you see anything interesting in the Glassfish server.log?


Comment by jsl123 [ 04/May/12 ]

I've used the file version from yesterday which was the latest, I'll try and do some more tests to narrow down the problem.

Comment by oleksiys [ 14/May/12 ]

any updates?


Comment by jsl123 [ 15/May/12 ]

Hi, sorry for the delay. I've done some more testing and doesn't appear to be a truncation issue as such, although it might be related. I'm not sure if this is a bug in the grizzly ajp code or apaches mod_proxy_ajp. I've created a new issue for this:



Comment by oleksiys [ 02/Jun/12 ]

update grizzly-http-ajp.jar , which has to have a fix for

Comment by bland999 [ 02/Jun/12 ]

The original issue of truncated transmissions is biting my organization (healthcare, 14000+ employees) pretty badly now. We recently upgraded to Glassfish 3.1.2 and we are trying to move away from a very expensive commercial server to this excellent open source choice.

However, this particular issue is a major stopper for us... is there any chance to please roll this fix into an update through the "updatetool" interface?

(We are running the Oracle supported release by the way.)

Thank you!

Comment by benjamin_m [ 08/Jun/12 ]

Ran into similar issue for content generated by Servlets behind mod_jk .
Fix was to remove response.setContentLength() and add "chunked" in HTTP header. Otherwise HTTP chunked data would get truncated for any response bigger than 512KB. Make sure chunked encoding is enable in your Glassfish listeners.

Comment by bland999 [ 13/Jun/12 ]

Chunked encoding is already enabled for the JK listener that we created. And also for the existing HTTP listeners... so I don't think this was the issue.

Comment by benjamin_m [ 13/Jun/12 ]

Having JK listener enabled for chunked encoding is not enough. If your servlet/producer sets the content length then the transfer will be truncated over 512KB.
That part takes its roots in the principle of chunking in HTTP 1.1 against HTTP 1.0 but I don't know if it's an actual error from the GRIZZLY implementation to by pass the chunked setting if setContentLength() is used.

Comment by oleksiys [ 13/Jun/12 ]

@bland999: just to make sure, do you see any issues after applying the patch (2 jar files attached here)? Thanks.

Comment by bland999 [ 14/Jun/12 ]

Thank you both (benjamin_m and oleksiys) for trying to help.

To oleksiys in particular, may I respectfully quote you in an earlier comment: "I can try to investigate if we can make Grizzly upgrade using GF update tool..."

This is actually important to my organization, as they are not very comfortable with open source (this is the first major installation in 30 years to give you an idea), so I was politely hoping that there would be an official release through the update toolset as opposed to applying a patch.

I apologize to request this, but even though I personally am an open source user for many years (I built my first Slackware distribution back in the mid-90's), there is a culture that I am trying to change here. I am sure my organization would be thrilled to see these patches "automagically" appear through updatetool.

Just my two cents, and thanks for taking it into consideration please.

Comment by tmpsa [ 17/Jun/12 ]

I can also confirm that the patched jar files solves the problem. I also got a broken connection to Apache after upgrading GF from 3.1.1 to 3.1.2 . Very frustrating! Thanks for the fix, you saved my day!!

I'll second the suggestion that these kind of critical patches should be distributed, e.g. at the GF 3.1.2 download page.

Comment by oleksiys [ 18/Jun/12 ]

latest patch for fix
"AJP connector can not recover after unexpected EOF"

Comment by tmpsa [ 14/Aug/12 ]

I suddenly get get an intense stream of SEVERE loggings, all identical. Same ThreadID, too. See below.
I run GF behind Apache, and have installed the patched jar files (which helped fix the file download issue).

I don't know if this is related, or perhaps another issue?

[#|2012-08-14T10:06:48.961+0200|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception.
java.lang.IllegalStateException: Invalid packet magic number: 4745 pos=0lastValid=54 end=0
at com.sun.grizzly.http.ajp.AjpInputBuffer.readAjpMessageHeader(AjpInputBuffer.java:85)
at com.sun.grizzly.http.ajp.AjpProcessorTask.parseRequest(AjpProcessorTask.java:107)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:706)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

Comment by oleksiys [ 14/Aug/12 ]

That's interesting, may be you try to send plain HTTP request to the AJP port?
Cause instead of AJP protocol magic 0x1234 we read 0x4745, which corresponds to characters "GE", which is probably part of "GET ...."

Comment by robbob [ 27/Aug/12 ]

We are having the same exact issue (same exception & magic number 4745) running GF behind Apache as well. The problem is random, and we have not been able to replicate it consistently. We will go a week or more without any issues, and then when it does occur, it continues occurring for every single request from that point on until the GF domain is restarted. We upgraded to GF hoping that would clear up the issue, but it didn't

Comment by oleksiys [ 27/Aug/12 ]

is it possible to capture tcp traffic (for example using wireshark) when you see this issue?
it really looks like plain HTTP messages come to AJP port, but i might be wrong.


Comment by oleksiys [ 05/Oct/12 ]

ajp related update for GF

Generated at Thu Mar 23 05:47:28 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.