glassfish
  1. glassfish
  2. GLASSFISH-18446

JK listener with Apache + mod_ajp_proxy causes truncated downloads

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.1.2_b23
    • Fix Version/s: 3.1.2.2
    • Component/s: grizzly-kernel
    • Labels:
      None

      Description

      Since upgrading to 3.1.2 I am experiencing truncated file downloads from my app with GF behind Apache using a lk-listener and Apache mod_ajp_proxy. It seems to be ajp related as downloads complete fully when accessing app directly on GF http port.

      With jk downloads truncate at 256Kb, the original file is 700Kb.

      This appears in the Apache logs:

      ajp_check_msg_header() got bad signature 28e9
      ajp_ilink_receive() received bad header
      ajp_read_header: ajp_ilink_receive failed
      (120007)APR does not understand this error code: proxy: dialog to 192.168.200.44:8009 (ww.mydomain.com) failed

      Config:

      asadmin create-network-listener --jkenabled true --protocol http-listener-1 --listenerport 8009 jk-listener
      ProxyPass / ajp://www.mydomain.com:8009/
      ProxyPassReverse / ajp://www.mydomain.com:8009/

      Packet capture of failed download attached.

      1. ajp_trunc.cap
        331 kB
        buddypine
      2. chrome_dump.txt
        6 kB
        jsl123
      3. mod_jk.log
        3.02 MB
        buddypine
      4. wget_dump.txt
        12 kB
        jsl123

        Activity

        Hide
        tmpsa added a comment -

        I suddenly get get an intense stream of SEVERE loggings, all identical. Same ThreadID, too. See below.
        I run GF behind Apache, and have installed the patched jar files (which helped fix the file download issue).

        I don't know if this is related, or perhaps another issue?

        [#|2012-08-14T10:06:48.961+0200|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception.
        java.lang.IllegalStateException: Invalid packet magic number: 4745 pos=0lastValid=54 end=0
        at com.sun.grizzly.http.ajp.AjpInputBuffer.readAjpMessageHeader(AjpInputBuffer.java:85)
        at com.sun.grizzly.http.ajp.AjpProcessorTask.parseRequest(AjpProcessorTask.java:107)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:706)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)

        #]
        Show
        tmpsa added a comment - I suddenly get get an intense stream of SEVERE loggings, all identical. Same ThreadID, too. See below. I run GF behind Apache, and have installed the patched jar files (which helped fix the file download issue). I don't know if this is related, or perhaps another issue? [#|2012-08-14T10:06:48.961+0200|SEVERE|glassfish3.1.2|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=25;_ThreadName=Thread-2;|GRIZZLY0051: ProcessorTask exception. java.lang.IllegalStateException: Invalid packet magic number: 4745 pos=0lastValid=54 end=0 at com.sun.grizzly.http.ajp.AjpInputBuffer.readAjpMessageHeader(AjpInputBuffer.java:85) at com.sun.grizzly.http.ajp.AjpProcessorTask.parseRequest(AjpProcessorTask.java:107) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:706) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:662) #]
        Hide
        oleksiys added a comment -

        That's interesting, may be you try to send plain HTTP request to the AJP port?
        Cause instead of AJP protocol magic 0x1234 we read 0x4745, which corresponds to characters "GE", which is probably part of "GET ...."

        Show
        oleksiys added a comment - That's interesting, may be you try to send plain HTTP request to the AJP port? Cause instead of AJP protocol magic 0x1234 we read 0x4745, which corresponds to characters "GE", which is probably part of "GET ...."
        Hide
        robbob added a comment -

        We are having the same exact issue (same exception & magic number 4745) running GF behind Apache as well. The problem is random, and we have not been able to replicate it consistently. We will go a week or more without any issues, and then when it does occur, it continues occurring for every single request from that point on until the GF domain is restarted. We upgraded to GF 3.1.2.2 hoping that would clear up the issue, but it didn't

        Show
        robbob added a comment - We are having the same exact issue (same exception & magic number 4745) running GF behind Apache as well. The problem is random, and we have not been able to replicate it consistently. We will go a week or more without any issues, and then when it does occur, it continues occurring for every single request from that point on until the GF domain is restarted. We upgraded to GF 3.1.2.2 hoping that would clear up the issue, but it didn't
        Hide
        oleksiys added a comment -

        is it possible to capture tcp traffic (for example using wireshark) when you see this issue?
        it really looks like plain HTTP messages come to AJP port, but i might be wrong.

        thx.

        Show
        oleksiys added a comment - is it possible to capture tcp traffic (for example using wireshark) when you see this issue? it really looks like plain HTTP messages come to AJP port, but i might be wrong. thx.
        Hide
        oleksiys added a comment -

        ajp related update for GF 3.1.2.2
        http://java.net/jira/browse/GRIZZLY-1340

        Show
        oleksiys added a comment - ajp related update for GF 3.1.2.2 http://java.net/jira/browse/GRIZZLY-1340

          People

          • Assignee:
            oleksiys
            Reporter:
            buddypine
          • Votes:
            4 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: