glassfish
  1. glassfish
  2. GLASSFISH-18446

JK listener with Apache + mod_ajp_proxy causes truncated downloads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1.2_b23
    • Fix Version/s: 3.1.2.2
    • Component/s: grizzly-kernel
    • Labels:
      None

      Description

      Since upgrading to 3.1.2 I am experiencing truncated file downloads from my app with GF behind Apache using a lk-listener and Apache mod_ajp_proxy. It seems to be ajp related as downloads complete fully when accessing app directly on GF http port.

      With jk downloads truncate at 256Kb, the original file is 700Kb.

      This appears in the Apache logs:

      ajp_check_msg_header() got bad signature 28e9
      ajp_ilink_receive() received bad header
      ajp_read_header: ajp_ilink_receive failed
      (120007)APR does not understand this error code: proxy: dialog to 192.168.200.44:8009 (ww.mydomain.com) failed

      Config:

      asadmin create-network-listener --jkenabled true --protocol http-listener-1 --listenerport 8009 jk-listener
      ProxyPass / ajp://www.mydomain.com:8009/
      ProxyPassReverse / ajp://www.mydomain.com:8009/

      Packet capture of failed download attached.

      1. ajp_trunc.cap
        331 kB
        buddypine
      2. chrome_dump.txt
        6 kB
        jsl123
      3. mod_jk.log
        3.02 MB
        buddypine
      4. wget_dump.txt
        12 kB
        jsl123

        Activity

        Hide
        buddypine added a comment -

        I have also tried with mod_jk Tomcat Connector and downloads are truncated at 512Kb.
        Attaching error log from mod_jk.

        Show
        buddypine added a comment - I have also tried with mod_jk Tomcat Connector and downloads are truncated at 512Kb. Attaching error log from mod_jk.
        Hide
        buddypine added a comment -

        mod_jk log file showing error

        Show
        buddypine added a comment - mod_jk log file showing error
        Hide
        jsl123 added a comment -

        Similar problem, webapp returning jsp pages works fine in 3.1.1 and when connecting to glassfish directly, but when you use mod_proxy_ajp, browsers time out waiting for the remaining data

        Show
        jsl123 added a comment - Similar problem, webapp returning jsp pages works fine in 3.1.1 and when connecting to glassfish directly, but when you use mod_proxy_ajp, browsers time out waiting for the remaining data
        Hide
        oleksiys added a comment -

        to buddypine:
        sorry, looks like a bug on our side.
        pls. try the attached patch (copy it to glassfish3/glassfish/modules folder)

        Show
        oleksiys added a comment - to buddypine: sorry, looks like a bug on our side. pls. try the attached patch (copy it to glassfish3/glassfish/modules folder)
        Hide
        oleksiys added a comment -

        to jsl123:
        can you pls. also try the patch, if it doesn't help - pls. let us know how we can reproduce the issue.

        Show
        oleksiys added a comment - to jsl123: can you pls. also try the patch, if it doesn't help - pls. let us know how we can reproduce the issue.
        Hide
        christoph.bernhofer added a comment - - edited

        Patch doesn't work for me. Have the same issue.
        Files still truncated

        --> Filesizes compared:
        theme.css: should be 31.3 kb
        primefaces.css: should be 41 kb
        jquery.js: should be 299.8 kb
        primefaces.js: should be 179.5 kb

        WITHOUT AJP PROXY:
        GET adminLogin.jsf?loc=en-GB 200 OK 192.168.78.77:8181 3.1 KB 192.168.78.77:8181 16ms
        GET theme.css.jsf?ln=primefaces-aristo 200 OK 192.168.78.77:8181 31.3 KB 11ms
        GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 41 KB 12ms
        GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 1.4 KB 21ms
        GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 299.8 KB 22ms
        GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 179.5 KB 22ms
        GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 4.7 KB 205ms
        GET iPCP.css 304 Not Modified 192.168.78.77:8181 2 KB 192.168.78.77:8181 15ms

        8 Anfragen

        562.8 KB

        (559.6 KB vom Cache)

        WITH AJP PROXY:
        GET adminLogin.jsf?loc=en-GB 200 OK 192.168.78.77 1.1 KB 192.168.78.77:443 16ms
        GET theme.css.jsf?ln=primefaces-aristo 200 OK 192.168.78.77 0 192.168.78.77:443 16ms
        GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 16ms
        GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 433 B 5ms
        GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 47ms
        GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 32ms
        GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 1.9 KB 8ms
        GET iPCP.css 304 Not Modified 192.168.78.77 733 B 192.168.78.77:443 16ms
        GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 16ms

        9 Anfragen

        4.1 KB

        (3 KB vom Cache)

        Show
        christoph.bernhofer added a comment - - edited Patch doesn't work for me. Have the same issue. Files still truncated --> Filesizes compared: theme.css: should be 31.3 kb primefaces.css: should be 41 kb jquery.js: should be 299.8 kb primefaces.js: should be 179.5 kb WITHOUT AJP PROXY: GET adminLogin.jsf?loc=en-GB 200 OK 192.168.78.77:8181 3.1 KB 192.168.78.77:8181 16ms GET theme.css.jsf?ln=primefaces-aristo 200 OK 192.168.78.77:8181 31.3 KB 11ms GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 41 KB 12ms GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 1.4 KB 21ms GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 299.8 KB 22ms GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 179.5 KB 22ms GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77:8181 4.7 KB 205ms GET iPCP.css 304 Not Modified 192.168.78.77:8181 2 KB 192.168.78.77:8181 15ms 8 Anfragen 562.8 KB (559.6 KB vom Cache) WITH AJP PROXY: GET adminLogin.jsf?loc=en-GB 200 OK 192.168.78.77 1.1 KB 192.168.78.77:443 16ms GET theme.css.jsf?ln=primefaces-aristo 200 OK 192.168.78.77 0 192.168.78.77:443 16ms GET primefaces.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 16ms GET password.css.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 433 B 5ms GET jquery.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 47ms GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 32ms GET password.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 1.9 KB 8ms GET iPCP.css 304 Not Modified 192.168.78.77 733 B 192.168.78.77:443 16ms GET primefaces.js.jsf?ln=primefaces&v=3.1.1 200 OK 192.168.78.77 0 192.168.78.77:443 16ms 9 Anfragen 4.1 KB (3 KB vom Cache)
        Hide
        buddypine added a comment -

        @oleksiys - thanks the patch does work for me.

        Show
        buddypine added a comment - @oleksiys - thanks the patch does work for me.
        Hide
        oleksiys added a comment -
        Show
        oleksiys added a comment - fixed in Grizzly http://java.net/jira/browse/GRIZZLY-1213
        Hide
        oleksiys added a comment -

        @jsl123
        @christoph.bernhofer

        can you pls. create a separate issue and provide more data, logs etc. Cause w/ the patch I'm not able to reproduce truncate issue (tried both secure and plain connections).

        Show
        oleksiys added a comment - @jsl123 @christoph.bernhofer can you pls. create a separate issue and provide more data, logs etc. Cause w/ the patch I'm not able to reproduce truncate issue (tried both secure and plain connections).
        Hide
        oleksiys added a comment -

        fix will be available in the next 3.1.x release

        Show
        oleksiys added a comment - fix will be available in the next 3.1.x release
        Hide
        rainerfrey added a comment -

        Will that be a regular several-month release cycle, or is there a chance for a quick release with this fix? AFAIS this is a general problem with Glassfish and JK, and in that case it will keep us from upgrading to 3.1.2.

        Or can the circumstances be narrowed down to more special configuration than using a jk-listener in general?

        Show
        rainerfrey added a comment - Will that be a regular several-month release cycle, or is there a chance for a quick release with this fix? AFAIS this is a general problem with Glassfish and JK, and in that case it will keep us from upgrading to 3.1.2. Or can the circumstances be narrowed down to more special configuration than using a jk-listener in general?
        Hide
        oleksiys added a comment -

        Normally we upgrade Grizzly as part of GF release cycle or as patch (which we provided).
        I can try to investigate if we can make Grizzly upgrade using GF update tool...

        Show
        oleksiys added a comment - Normally we upgrade Grizzly as part of GF release cycle or as patch (which we provided). I can try to investigate if we can make Grizzly upgrade using GF update tool...
        Hide
        oleksiys added a comment -

        changing priority and module

        Show
        oleksiys added a comment - changing priority and module
        Hide
        oleksiys added a comment -

        @jsl123
        @christoph.bernhofer

        guys, wanted to ask you to give me more info on your usecases, so we can fix and deliver updates asap.

        thanks.

        Show
        oleksiys added a comment - @jsl123 @christoph.bernhofer guys, wanted to ask you to give me more info on your usecases, so we can fix and deliver updates asap. thanks.
        Hide
        christoph.bernhofer added a comment -

        Hi oleksiys

        My Application is a JSF2 WAR Application behind Apache 2.2. mod_proxy_ajp.

        Created with
        asadmin create-threadpool --minthreadpoolsize 5 --maxthreadpoolsize 150 jk-thread-pool
        asadmin create-network-listener --protocol http-listener-1 --listenerport 8091 --jkenabled true --threadpool jk-thread-pool jk-connector

        and

        ProxyPass /iPCP ajp://localhost:8091/iPCP
        ProxyPassReverse /iPCP ajp://localhost:8091/iPCP

        works flawless with glassfish 3.1.1 in same configuration

        updated now Apache ond Debian squeeze x64 to apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_perl/2.0.4 Perl/v5.10.1 configured

        but this hasn't changed anything.

        error.log apache2:

        ajp_check_msg_header() got bad signature 6575
        [Mon Mar 05 14:14:22 2012] [error] ajp_ilink_receive() received bad header
        [Mon Mar 05 14:14:22 2012] [error] ajp_read_header: ajp_ilink_receive failed
        [Mon Mar 05 14:14:22 2012] [error] (120007)APR does not understand this error code: proxy: dialog to 127.0.0.1:8091 (localhost) failed
        [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico
        [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico
        [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico
        [Mon Mar 05 14:15:06 2012] [error] [client 192.168.178.14] proxy: error processing end, referer: http://192.168.78.77/iPCP/admin/adminLogin.jsf?loc=en-GB
        [Mon Mar 05 14:15:06 2012] [error] [client 192.168.178.14] proxy: error processing body. Client aborted connection., referer: http://192.168.78.77/iPCP/admin/adminLogin.jsf?loc=en-GB
        [Mon Mar 05 14:15:16 2012] [error] ajp_check_msg_header() got bad signature 6575
        [Mon Mar 05 14:15:16 2012] [error] ajp_ilink_receive() received bad header
        [Mon Mar 05 14:15:16 2012] [error] ajp_read_header: ajp_ilink_receive failed

        which more info do you need?

        Christoph

        Show
        christoph.bernhofer added a comment - Hi oleksiys My Application is a JSF2 WAR Application behind Apache 2.2. mod_proxy_ajp. Created with asadmin create-threadpool --minthreadpoolsize 5 --maxthreadpoolsize 150 jk-thread-pool asadmin create-network-listener --protocol http-listener-1 --listenerport 8091 --jkenabled true --threadpool jk-thread-pool jk-connector and ProxyPass /iPCP ajp://localhost:8091/iPCP ProxyPassReverse /iPCP ajp://localhost:8091/iPCP works flawless with glassfish 3.1.1 in same configuration updated now Apache ond Debian squeeze x64 to apache/2.2.16 (Debian) PHP/5.3.3-7+squeeze1 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_perl/2.0.4 Perl/v5.10.1 configured but this hasn't changed anything. error.log apache2: ajp_check_msg_header() got bad signature 6575 [Mon Mar 05 14:14:22 2012] [error] ajp_ilink_receive() received bad header [Mon Mar 05 14:14:22 2012] [error] ajp_read_header: ajp_ilink_receive failed [Mon Mar 05 14:14:22 2012] [error] (120007)APR does not understand this error code: proxy: dialog to 127.0.0.1:8091 (localhost) failed [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico [Mon Mar 05 14:14:23 2012] [error] [client 192.168.178.14] File does not exist: /var/www/favicon.ico [Mon Mar 05 14:15:06 2012] [error] [client 192.168.178.14] proxy: error processing end, referer: http://192.168.78.77/iPCP/admin/adminLogin.jsf?loc=en-GB [Mon Mar 05 14:15:06 2012] [error] [client 192.168.178.14] proxy: error processing body. Client aborted connection., referer: http://192.168.78.77/iPCP/admin/adminLogin.jsf?loc=en-GB [Mon Mar 05 14:15:16 2012] [error] ajp_check_msg_header() got bad signature 6575 [Mon Mar 05 14:15:16 2012] [error] ajp_ilink_receive() received bad header [Mon Mar 05 14:15:16 2012] [error] ajp_read_header: ajp_ilink_receive failed which more info do you need? Christoph
        Hide
        oleksiys added a comment -

        can you pls. attach the app, or better very reduced version of the app, so I'll be able to debug the problem.

        Thanks.

        Show
        oleksiys added a comment - can you pls. attach the app, or better very reduced version of the app, so I'll be able to debug the problem. Thanks.
        Hide
        rainerfrey added a comment -

        An uüdate via updatetool would be great! Thank you.

        Show
        rainerfrey added a comment - An uüdate via updatetool would be great! Thank you.
        Hide
        christoph.bernhofer added a comment -

        Hi oleksiys

        I just tried withthe Primefces showcase App.
        Same issue. Glassfish 3.1.1. same configuration, same Apache --> works flawless
        Glassfish 3.1.2 don't work with proxy, Without proxy it works
        I tried to access the App via mod_ssl at Apache at 443 and ajp behind as written above.

        Here is the Link to the sample war file:

        http://repository.primefaces.org/org/primefaces/prime-showcase/1.0.0-SNAPSHOT/prime-showcase-1.0.0-SNAPSHOT.war

        Thx

        Christoph

        Show
        christoph.bernhofer added a comment - Hi oleksiys I just tried withthe Primefces showcase App. Same issue. Glassfish 3.1.1. same configuration, same Apache --> works flawless Glassfish 3.1.2 don't work with proxy, Without proxy it works I tried to access the App via mod_ssl at Apache at 443 and ajp behind as written above. Here is the Link to the sample war file: http://repository.primefaces.org/org/primefaces/prime-showcase/1.0.0-SNAPSHOT/prime-showcase-1.0.0-SNAPSHOT.war Thx Christoph
        Hide
        jsl123 added a comment -

        Hi guys, sorry didn't get updates about this and was distracted. Trying patch now. Will report on progress.

        Show
        jsl123 added a comment - Hi guys, sorry didn't get updates about this and was distracted. Trying patch now. Will report on progress.
        Hide
        oleksiys added a comment -

        Hi Christoph,

        what exactly doesn't work in that app? Cause I'm able to load it in a browser and go throw couple of menu items w/o problems.

        Thanks.

        Show
        oleksiys added a comment - Hi Christoph, what exactly doesn't work in that app? Cause I'm able to load it in a browser and go throw couple of menu items w/o problems. Thanks.
        Hide
        christoph.bernhofer added a comment -

        Hi oleksiys

        Now I get a complete timeout of the whole App. App is not loaded via proxy.

        Can you plese tell me your exact versions for Apache, Java...
        have you tried also via SSL?

        Tx

        Christoph

        Show
        christoph.bernhofer added a comment - Hi oleksiys Now I get a complete timeout of the whole App. App is not loaded via proxy. Can you plese tell me your exact versions for Apache, Java... have you tried also via SSL? Tx Christoph
        Hide
        oleksiys added a comment - - edited

        I tried mod_ssl and plain, both work fine.

        Apache 2.2.17:

        ProxyPass /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT retry=0 ttl=30
        ProxyPassReverse /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT
        ProxyPreserveHost on

        JDK:

        java version "1.6.0_30"
        Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
        Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)

        Show
        oleksiys added a comment - - edited I tried mod_ssl and plain, both work fine. Apache 2.2.17: ProxyPass /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT retry=0 ttl=30 ProxyPassReverse /prime-showcase-1.0.0-SNAPSHOT ajp://localhost:8009/prime-showcase-1.0.0-SNAPSHOT ProxyPreserveHost on JDK: java version "1.6.0_30" Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
        Hide
        jsl123 added a comment - - edited

        OK, further investigation.

        Ubuntu 11.10 x64
        Apache 2.2.20-1ubuntu1.2
        Glassfish 3.1.2 with grizzly patch

        If I disable mod_deflate in apache it works fine, pages appear as normal, however as soon as I enable mod_deflate, things break.

        With mod_deflate:
        Using standard wget works fine - no compression used
        Using wget with "accept-encoding:gzip" header set works.
        Using firefox/chrome doesn't work. Firefox silently shows a blank page however chrome gives us a clue and reports
        "Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error

        This all works fine in 3.1.1

        Looking at the net traffic (attached) they are different both in content (should be the same) and also size, the chunked version is bigger. The wget compressed data expands into the same as if compression isn't used.

        So it looks like a different bug

        Show
        jsl123 added a comment - - edited OK, further investigation. Ubuntu 11.10 x64 Apache 2.2.20-1ubuntu1.2 Glassfish 3.1.2 with grizzly patch If I disable mod_deflate in apache it works fine, pages appear as normal, however as soon as I enable mod_deflate, things break. With mod_deflate: Using standard wget works fine - no compression used Using wget with "accept-encoding:gzip" header set works. Using firefox/chrome doesn't work. Firefox silently shows a blank page however chrome gives us a clue and reports "Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error This all works fine in 3.1.1 Looking at the net traffic (attached) they are different both in content (should be the same) and also size, the chunked version is bigger. The wget compressed data expands into the same as if compression isn't used. So it looks like a different bug
        Hide
        jsl123 added a comment -

        tcpdump of chrome page fetch

        Show
        jsl123 added a comment - tcpdump of chrome page fetch
        Hide
        jsl123 added a comment -

        tcp dump of wget

        Show
        jsl123 added a comment - tcp dump of wget
        Hide
        christoph.bernhofer added a comment - - edited

        Get the same Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error
        Deactiveated now mod_deflate --> works

        Show
        christoph.bernhofer added a comment - - edited Get the same Error 321 (net::ERR_INVALID_CHUNKED_ENCODING): Unknown error Deactiveated now mod_deflate --> works
        Hide
        oleksiys added a comment -

        thank you guys, i'll investigate that and let you know asap.

        Show
        oleksiys added a comment - thank you guys, i'll investigate that and let you know asap.
        Hide
        oleksiys added a comment -

        additional patch

        Show
        oleksiys added a comment - additional patch
        Hide
        oleksiys added a comment -

        guys, can you pls. try to apply another jar (attached) additionally to one which had been attached before?
        thx.

        Show
        oleksiys added a comment - guys, can you pls. try to apply another jar (attached) additionally to one which had been attached before? thx.
        Hide
        christoph.bernhofer added a comment -

        Wworks for me with an without mod_deflate!

        Great Work, thank you

        Show
        christoph.bernhofer added a comment - Wworks for me with an without mod_deflate! Great Work, thank you
        Hide
        jsl123 added a comment -

        Looks to work for me as well, with mod_deflate.

        Will do more testing and post if I find anything else..

        good work, Cheers

        Show
        jsl123 added a comment - Looks to work for me as well, with mod_deflate. Will do more testing and post if I find anything else.. good work, Cheers
        Hide
        jsl123 added a comment -

        One thing I've noticed and don't think it is expected, but wanted to check before I file a bug? With 3.1.2 when we use HttpServletRequest.getRemoteAddr/Host it returns 127.0.0.1 when using mod_proxy_ajp and apache. With 3.1.1 it returns the address of the client.
        I assume this is a bug

        Show
        jsl123 added a comment - One thing I've noticed and don't think it is expected, but wanted to check before I file a bug? With 3.1.2 when we use HttpServletRequest.getRemoteAddr/Host it returns 127.0.0.1 when using mod_proxy_ajp and apache. With 3.1.1 it returns the address of the client. I assume this is a bug
        Hide
        oleksiys added a comment -

        ok, will check that.

        thanks.

        Show
        oleksiys added a comment - ok, will check that. thanks.
        Hide
        oleksiys added a comment -

        fix for getRemoteXXX()

        Show
        oleksiys added a comment - fix for getRemoteXXX()
        Hide
        oleksiys added a comment -

        attached the grizzly-http-ajp.jar with the getRemoteAddr/Host fix.
        thanks!

        Show
        oleksiys added a comment - attached the grizzly-http-ajp.jar with the getRemoteAddr/Host fix. thanks!
        Hide
        jsl123 added a comment -

        Cheers, looks to have fixed it. Would it be possible to push this via the update-tool?

        Thanks

        Show
        jsl123 added a comment - Cheers, looks to have fixed it. Would it be possible to push this via the update-tool? Thanks
        Hide
        oleksiys added a comment -

        thank you for confirming!

        w/ update tool, we're still investigating this...

        Show
        oleksiys added a comment - thank you for confirming! w/ update tool, we're still investigating this...
        Hide
        AndrewJames added a comment -

        I faced the same issue - very happy to see this patch, which resolved the problem - thank you. A life-saver (figuratively speaking).

        Show
        AndrewJames added a comment - I faced the same issue - very happy to see this patch, which resolved the problem - thank you. A life-saver (figuratively speaking).
        Hide
        oleksiys added a comment -

        internal bugdb# 13902008.

        Show
        oleksiys added a comment - internal bugdb# 13902008.
        Hide
        oleksiys added a comment -

        update the patch to fix issue
        http://java.net/jira/browse/GRIZZLY-1254

        Show
        oleksiys added a comment - update the patch to fix issue http://java.net/jira/browse/GRIZZLY-1254
        Hide
        jsl123 added a comment -

        Although the download truncation seems to have been fixed, it looks like we are getting an upload truncation issue. One of my clients is posting files to our server which all worked fine with 3.1.1, but after upgrading and applying the above patches, it bombs out part way through the transfer.

        Small files work, but larger ones don't. In apache I get
        [Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(273): proxy: data to read (max 8186 at 4)
        [Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(288): proxy: got 1448 bytes of data
        [Tue May 01 15:06:32 2012] [error] ajp_check_msg_header() got bad signature 4854
        [Tue May 01 15:06:32 2012] [error] ajp_ilink_receive() received bad header
        [Tue May 01 15:06:32 2012] [error] ajp_read_header: ajp_ilink_receive failed
        [Tue May 01 15:06:32 2012] [error] (120007)APR does not understand this error code: proxy: read response failed from 127.0.0.1:8009 (localhost)
        [Tue May 01 15:06:32 2012] [debug] proxy_util.c(2029): proxy: AJP: has released connection for (localhost)
        [Tue May 01 15:06:32 2012] [debug] mod_headers.c(756): headers: ap_headers_output_filter()
        [Tue May 01 15:06:32 2012] [debug] mod_dumpio.c(142): mod_dumpio: dumpio_out

        which then results in a 500 error from the server ultimately being returned.

        John

        Show
        jsl123 added a comment - Although the download truncation seems to have been fixed, it looks like we are getting an upload truncation issue. One of my clients is posting files to our server which all worked fine with 3.1.1, but after upgrading and applying the above patches, it bombs out part way through the transfer. Small files work, but larger ones don't. In apache I get [Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(273): proxy: data to read (max 8186 at 4) [Tue May 01 15:06:32 2012] [debug] mod_proxy_ajp.c(288): proxy: got 1448 bytes of data [Tue May 01 15:06:32 2012] [error] ajp_check_msg_header() got bad signature 4854 [Tue May 01 15:06:32 2012] [error] ajp_ilink_receive() received bad header [Tue May 01 15:06:32 2012] [error] ajp_read_header: ajp_ilink_receive failed [Tue May 01 15:06:32 2012] [error] (120007)APR does not understand this error code: proxy: read response failed from 127.0.0.1:8009 (localhost) [Tue May 01 15:06:32 2012] [debug] proxy_util.c(2029): proxy: AJP: has released connection for (localhost) [Tue May 01 15:06:32 2012] [debug] mod_headers.c(756): headers: ap_headers_output_filter() [Tue May 01 15:06:32 2012] [debug] mod_dumpio.c(142): mod_dumpio: dumpio_out which then results in a 500 error from the server ultimately being returned. John
        Hide
        oleksiys added a comment -

        Did you try the latest grizzly-http-ajp.jar I attached recently?
        Does this problem occur all the time you try to upload files? Do you see anything interesting in the Glassfish server.log?

        thx.

        Show
        oleksiys added a comment - Did you try the latest grizzly-http-ajp.jar I attached recently? Does this problem occur all the time you try to upload files? Do you see anything interesting in the Glassfish server.log? thx.
        Hide
        jsl123 added a comment -

        I've used the file version from yesterday which was the latest, I'll try and do some more tests to narrow down the problem.
        Thanks

        Show
        jsl123 added a comment - I've used the file version from yesterday which was the latest, I'll try and do some more tests to narrow down the problem. Thanks
        Hide
        oleksiys added a comment -

        any updates?

        thx.

        Show
        oleksiys added a comment - any updates? thx.
        Hide
        jsl123 added a comment -

        Hi, sorry for the delay. I've done some more testing and doesn't appear to be a truncation issue as such, although it might be related. I'm not sure if this is a bug in the grizzly ajp code or apaches mod_proxy_ajp. I've created a new issue for this:
        http://java.net/jira/browse/GLASSFISH-18731

        Thanks

        John

        Show
        jsl123 added a comment - Hi, sorry for the delay. I've done some more testing and doesn't appear to be a truncation issue as such, although it might be related. I'm not sure if this is a bug in the grizzly ajp code or apaches mod_proxy_ajp. I've created a new issue for this: http://java.net/jira/browse/GLASSFISH-18731 Thanks John
        Hide
        oleksiys added a comment -

        update grizzly-http-ajp.jar , which has to have a fix for
        http://java.net/jira/browse/GRIZZLY-1276

        Show
        oleksiys added a comment - update grizzly-http-ajp.jar , which has to have a fix for http://java.net/jira/browse/GRIZZLY-1276
        Hide
        bland999 added a comment -

        The original issue of truncated transmissions is biting my organization (healthcare, 14000+ employees) pretty badly now. We recently upgraded to Glassfish 3.1.2 and we are trying to move away from a very expensive commercial server to this excellent open source choice.

        However, this particular issue is a major stopper for us... is there any chance to please roll this fix into an update through the "updatetool" interface?

        (We are running the Oracle supported release by the way.)

        Thank you!

        Show
        bland999 added a comment - The original issue of truncated transmissions is biting my organization (healthcare, 14000+ employees) pretty badly now. We recently upgraded to Glassfish 3.1.2 and we are trying to move away from a very expensive commercial server to this excellent open source choice. However, this particular issue is a major stopper for us... is there any chance to please roll this fix into an update through the "updatetool" interface? (We are running the Oracle supported release by the way.) Thank you!
        Hide
        benjamin_m added a comment -

        Ran into similar issue for content generated by Servlets behind mod_jk .
        Fix was to remove response.setContentLength() and add "chunked" in HTTP header. Otherwise HTTP chunked data would get truncated for any response bigger than 512KB. Make sure chunked encoding is enable in your Glassfish listeners.

        Show
        benjamin_m added a comment - Ran into similar issue for content generated by Servlets behind mod_jk . Fix was to remove response.setContentLength() and add "chunked" in HTTP header. Otherwise HTTP chunked data would get truncated for any response bigger than 512KB. Make sure chunked encoding is enable in your Glassfish listeners.
        Hide
        bland999 added a comment -

        Chunked encoding is already enabled for the JK listener that we created. And also for the existing HTTP listeners... so I don't think this was the issue.

        Show
        bland999 added a comment - Chunked encoding is already enabled for the JK listener that we created. And also for the existing HTTP listeners... so I don't think this was the issue.
        Hide
        benjamin_m added a comment - - edited

        Having JK listener enabled for chunked encoding is not enough. If your servlet/producer sets the content length then the transfer will be truncated over 512KB.
        That part takes its roots in the principle of chunking in HTTP 1.1 against HTTP 1.0 but I don't know if it's an actual error from the GRIZZLY implementation to by pass the chunked setting if setContentLength() is used.

        Show
        benjamin_m added a comment - - edited Having JK listener enabled for chunked encoding is not enough. If your servlet/producer sets the content length then the transfer will be truncated over 512KB. That part takes its roots in the principle of chunking in HTTP 1.1 against HTTP 1.0 but I don't know if it's an actual error from the GRIZZLY implementation to by pass the chunked setting if setContentLength() is used.
        Hide
        oleksiys added a comment -

        @bland999: just to make sure, do you see any issues after applying the patch (2 jar files attached here)? Thanks.

        Show
        oleksiys added a comment - @bland999: just to make sure, do you see any issues after applying the patch (2 jar files attached here)? Thanks.
        Hide
        bland999 added a comment -

        Thank you both (benjamin_m and oleksiys) for trying to help.

        To oleksiys in particular, may I respectfully quote you in an earlier comment: "I can try to investigate if we can make Grizzly upgrade using GF update tool..."

        This is actually important to my organization, as they are not very comfortable with open source (this is the first major installation in 30 years to give you an idea), so I was politely hoping that there would be an official release through the update toolset as opposed to applying a patch.

        I apologize to request this, but even though I personally am an open source user for many years (I built my first Slackware distribution back in the mid-90's), there is a culture that I am trying to change here. I am sure my organization would be thrilled to see these patches "automagically" appear through updatetool.

        Just my two cents, and thanks for taking it into consideration please.

        Show
        bland999 added a comment - Thank you both (benjamin_m and oleksiys) for trying to help. To oleksiys in particular, may I respectfully quote you in an earlier comment: "I can try to investigate if we can make Grizzly upgrade using GF update tool..." This is actually important to my organization, as they are not very comfortable with open source (this is the first major installation in 30 years to give you an idea), so I was politely hoping that there would be an official release through the update toolset as opposed to applying a patch. I apologize to request this, but even though I personally am an open source user for many years (I built my first Slackware distribution back in the mid-90's), there is a culture that I am trying to change here. I am sure my organization would be thrilled to see these patches "automagically" appear through updatetool. Just my two cents, and thanks for taking it into consideration please.
        Hide
        tmpsa added a comment -

        I can also confirm that the patched jar files solves the problem. I also got a broken connection to Apache after upgrading GF from 3.1.1 to 3.1.2 . Very frustrating! Thanks for the fix, you saved my day!!

        I'll second the suggestion that these kind of critical patches should be distributed, e.g. at the GF 3.1.2 download page.

        Show
        tmpsa added a comment - I can also confirm that the patched jar files solves the problem. I also got a broken connection to Apache after upgrading GF from 3.1.1 to 3.1.2 . Very frustrating! Thanks for the fix, you saved my day!! I'll second the suggestion that these kind of critical patches should be distributed, e.g. at the GF 3.1.2 download page.
        Hide
        oleksiys added a comment -

        latest patch for fix
        http://java.net/jira/browse/GRIZZLY-1284
        "AJP connector can not recover after unexpected EOF"

        Show
        oleksiys added a comment - latest patch for fix http://java.net/jira/browse/GRIZZLY-1284 "AJP connector can not recover after unexpected EOF"
        Hide
        tmpsa added a comment -

        I suddenly get get an intense stream of SEVERE loggings, all identical. Same ThreadID, too. See below.
        I run GF behind Apache, and have installed the patched jar files (which helped fix the file download issue).

        I don't know if this is related, or perhaps another issue?

        [#|2012-08-14T10:06:48.961+0200|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception.
        java.lang.IllegalStateException: Invalid packet magic number: 4745 pos=0lastValid=54 end=0
        at com.sun.grizzly.http.ajp.AjpInputBuffer.readAjpMessageHeader(AjpInputBuffer.java:85)
        at com.sun.grizzly.http.ajp.AjpProcessorTask.parseRequest(AjpProcessorTask.java:107)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:706)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)

        #]
        Show
        tmpsa added a comment - I suddenly get get an intense stream of SEVERE loggings, all identical. Same ThreadID, too. See below. I run GF behind Apache, and have installed the patched jar files (which helped fix the file download issue). I don't know if this is related, or perhaps another issue? [#|2012-08-14T10:06:48.961+0200|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception. java.lang.IllegalStateException: Invalid packet magic number: 4745 pos=0lastValid=54 end=0 at com.sun.grizzly.http.ajp.AjpInputBuffer.readAjpMessageHeader(AjpInputBuffer.java:85) at com.sun.grizzly.http.ajp.AjpProcessorTask.parseRequest(AjpProcessorTask.java:107) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:706) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:662) #]
        Hide
        oleksiys added a comment -

        That's interesting, may be you try to send plain HTTP request to the AJP port?
        Cause instead of AJP protocol magic 0x1234 we read 0x4745, which corresponds to characters "GE", which is probably part of "GET ...."

        Show
        oleksiys added a comment - That's interesting, may be you try to send plain HTTP request to the AJP port? Cause instead of AJP protocol magic 0x1234 we read 0x4745, which corresponds to characters "GE", which is probably part of "GET ...."
        Hide
        robbob added a comment -

        We are having the same exact issue (same exception & magic number 4745) running GF behind Apache as well. The problem is random, and we have not been able to replicate it consistently. We will go a week or more without any issues, and then when it does occur, it continues occurring for every single request from that point on until the GF domain is restarted. We upgraded to GF 3.1.2.2 hoping that would clear up the issue, but it didn't

        Show
        robbob added a comment - We are having the same exact issue (same exception & magic number 4745) running GF behind Apache as well. The problem is random, and we have not been able to replicate it consistently. We will go a week or more without any issues, and then when it does occur, it continues occurring for every single request from that point on until the GF domain is restarted. We upgraded to GF 3.1.2.2 hoping that would clear up the issue, but it didn't
        Hide
        oleksiys added a comment -

        is it possible to capture tcp traffic (for example using wireshark) when you see this issue?
        it really looks like plain HTTP messages come to AJP port, but i might be wrong.

        thx.

        Show
        oleksiys added a comment - is it possible to capture tcp traffic (for example using wireshark) when you see this issue? it really looks like plain HTTP messages come to AJP port, but i might be wrong. thx.
        Hide
        oleksiys added a comment -

        ajp related update for GF 3.1.2.2
        http://java.net/jira/browse/GRIZZLY-1340

        Show
        oleksiys added a comment - ajp related update for GF 3.1.2.2 http://java.net/jira/browse/GRIZZLY-1340

          People

          • Assignee:
            oleksiys
            Reporter:
            buddypine
          • Votes:
            4 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: