[GRIZZLY-1787] Streaming data via AJP causes OOME Created: 28/Jun/15  Updated: 30/Jun/15  Resolved: 30/Jun/15

Status: Resolved
Project: grizzly
Component/s: ajp
Affects Version/s: 2.3.19
Fix Version/s: 2.3.22, 3.0

Type: Bug Priority: Major
Reporter: payara_steve Assignee: Ryan Lubke
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Archive File grizzly-http-ajp-2.3.19.jar    
Tags: AJP

 Description   

Streaming data via AJP without setting a Content-Length header results in the data accumulating in-memory resulting in an eventual OutOfMemoryError.

See https://github.com/payara/Payara/issues/350 for detailed diagnostics and reproducer



 Comments   
Comment by Ryan Lubke [ 29/Jun/15 ]

Thanks for raising this. I'll take a look this week.

Comment by Ryan Lubke [ 29/Jun/15 ]

Able to reproduce. Investigating.

Comment by Ryan Lubke [ 29/Jun/15 ]

Root cause is that the AjpHttpResponse implementation doesn't take chunking configuration into account at all, so that code path is disabled. This means that the response has to be buffered in order to provide a content-length.

I've made a quick change to the aforementioned mentioned class and it appears to have resolved the problem.

We can provide a drop-in replacement JAR for the Grizzly runtime within GlassFish for you to test/confirm with in your environment, but we'll need to know what version of GlassFish you're using.

Comment by payara_steve [ 29/Jun/15 ]

We are using Payara 4.1.152.1 which is based off GlassFish 4.1 but ships with Grizzly 2.3.19. I can test a patched Grizzly 2.3.19.

Comment by Ryan Lubke [ 29/Jun/15 ]

Modified 2.3.19 AJP code to resolve chunking issue.

Comment by Ryan Lubke [ 29/Jun/15 ]

I've attached a modified 2.3.19 AJP jar.

You should be able to replace the existing 2.3.19 jar in your local maven repo and re-build your custom GF server to pick the JAR up.

Comment by payara_steve [ 30/Jun/15 ]

Thanks I'll test the patch. Do you know which Grizzly release it will appear in if everything is OK?

BTW your repo synch to github seems to have stopped https://github.com/GrizzlyNIO/grizzly-mirror

Comment by Ryan Lubke [ 30/Jun/15 ]

The next release version is 2.3.22. I'll have to check with the team regarding a release timeline.

As to the mirror, you're right. Thanks for pointing that out. We'll look into it.

Comment by Ryan Lubke [ 30/Jun/15 ]

Unrelated to this issue, but the mirror issue is resolved.

Comment by Ryan Lubke [ 30/Jun/15 ]

2.3.x: 6c08805499f7c6b64c3c8806a71f0b58b52c960b
master: 102c432aa2004c581db1c297e94b0eb90dd74122





[GLASSFISH-20694] Glassfish 4.0 and jk Unable to populate SSL attributes Created: 10/Jul/13  Updated: 19/Sep/14  Resolved: 17/Jul/13

Status: Resolved
Project: glassfish
Component/s: web_container
Affects Version/s: 4.0
Fix Version/s: 4.1

Type: Bug Priority: Major
Reporter: buddypine Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 4.0 b89, Apache 2.2.22 mod_proxy_ajp


Tags: ajp, apache, proxy

 Description   

I am using a jk enabled listener behind Apache.

GF listener created as follows:
% asadmin create-network-listener --jkenabled true --protocol http-listener-1 --listenerport 8009 jk-listener

Apache httpd-ssl.conf mod_proxy_ajp
------------------------------------------
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

All seems to work fine and I can connect using SSL to Apache which proxies the requests through to Glassfish but I get the exception below in the logs (this does not happen with GF 3):

[2013-07-10T15:06:37.929+0200] [glassfish 4.0] [WARNING] [] [org.glassfish.grizzly.http.server.util.RequestUtils] [tid: _ThreadID=53 _ThreadName=jk-listener(4)] [timeMillis: 1373461597929] [levelValue: 900] [[
Unable to populate SSL attributes
java.lang.IllegalStateException: SSLEngine is null
at org.glassfish.grizzly.ssl.SSLSupportImpl.<init>(SSLSupportImpl.java:87)
at org.glassfish.grizzly.http.server.util.RequestUtils.populateSSLAttributes(RequestUtils.java:85)
at org.glassfish.grizzly.http.server.Request.getAttribute(Request.java:865)
at org.apache.catalina.connector.Request.populateSSLAttributes(Request.java:4581)
at org.apache.catalina.connector.Request.getAttributeNames(Request.java:1412)
at org.apache.catalina.connector.RequestFacade.getAttributeNames(RequestFacade.java:367)
at org.jboss.weld.context.beanstore.http.RequestBeanStore.getAttributeNames(RequestBeanStore.java:48)
at org.jboss.weld.context.beanstore.AttributeBeanStore.getPrefixedAttributeNames(AttributeBeanStore.java:207)
at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:106)
at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:52)
at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:37)
at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:190)
at org.apache.catalina.core.StandardContext.fireRequestInitializedEvent(StandardContext.java:5225)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:647)
at org.apache.catalina.core.StandardHostValve.__invoke(StandardHostValve.java:166)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
at java.lang.Thread.run(Thread.java:724)
]]



 Comments   
Comment by Shing Wai Chan [ 10/Jul/13 ]

The exception is from org.glassfish.grizzly.ssl.SSLSupportImpl in Grizzly. Assign to Alexey for further investigation.

Comment by oleksiys [ 12/Jul/13 ]

looks like apache doesn't pass all the SSL information to Glassfish.
Can you pls. share the entire ssl.conf?

Comment by buddypine [ 12/Jul/13 ]

Sure:

 
#
# This is the Apache server configuration file providing SSL support.
# It contains the configuration directives to instruct the server how to
# serve pages over an https connection. For detailing information about these 
# directives see <URL:http://httpd.apache.org/docs/2.2/mod/mod_ssl.html>
# 
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#

#
# Pseudo Random Number Generator (PRNG):
# Configure one or more sources to seed the PRNG of the SSL library.
# The seed data should be of good random quality.
# WARNING! On some platforms /dev/random blocks if not enough entropy
# is available. This means you then cannot use the /dev/random device
# because it would lead to very long connection times (as long as
# it requires to make more entropy available). But usually those
# platforms additionally provide a /dev/urandom device which doesn't
# block. So, if available, use this one instead. Read the mod_ssl User
# Manual for more details.
#
#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed startup file:/dev/urandom 512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512


#
# When we also provide SSL we have to listen to the 
# standard HTTP port (see above) and to the HTTPS port
#
# Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
#       Listen directives: "Listen [::]:443" and "Listen 0.0.0.0:443"
#
Listen 443

##
##  SSL Global Context
##
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.
##

#
#   Some MIME-types for downloading Certificates and CRLs
#
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#   Pass Phrase Dialog:
#   Configure the pass phrase gathering process.
#   The filtering dialog program (`builtin' is a internal
#   terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog  builtin

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism 
#   to use and second the expiring timeout (in seconds).
#SSLSessionCache         "dbm:/private/var/run/ssl_scache"
SSLSessionCache        "shmcb:/private/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

#   Semaphore:
#   Configure the path to the mutual exclusion semaphore the
#   SSL engine uses internally for inter-process synchronization. 
SSLMutex  "file:/private/var/run/ssl_mutex"

##
## SSL Virtual Host Context
##

<VirtualHost _default_:443>

#   General setup for the virtual host
DocumentRoot "/Library/WebServer/Documents"

ServerAdmin you@example.com
ErrorLog "/private/var/log/apache2/error_log"
TransferLog "/private/var/log/apache2/access_log"

ServerName myapp.local.inversebit.com:443
ProxyPass / ajp://myapp.local.inversebit.com:8009/
ProxyPassReverse / ajp://myapp.local.inversebit.com:8009/

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   SSL Cipher Suite:
#   List the ciphers that the client is permitted to negotiate.
#   See the mod_ssl documentation for a complete list.
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

#   Server Certificate:
#   Point SSLCertificateFile at a PEM encoded certificate.  If
#   the certificate is encrypted, then you will be prompted for a
#   pass phrase.  Note that a kill -HUP will prompt again.  Keep
#   in mind that if you have both an RSA and a DSA certificate you
#   can configure both in parallel (to also allow the use of DSA
#   ciphers, etc.)
#SSLCertificateFile "/private/etc/apache2/server.crt"
#SSLCertificateFile "/private/etc/apache2/server-dsa.crt"

SSLCertificateFile /Users/phill/etc/certs/newcert.pem
SSLCertificateKeyFile /Users/phill/etc/certs/webserver.nopass.key
SSLCACertificateFile /Users/phill/etc/certs/demoCA/cacert.pem
SSLCARevocationPath /Users/phill/etc/certs/demoCA/crl

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
#SSLCertificateKeyFile "/private/etc/apache2/server.key"
#SSLCertificateKeyFile "/private/etc/apache2/server-dsa.key"

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convinience.
#SSLCertificateChainFile "/private/etc/apache2/server-ca.crt"

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#   Note: Inside SSLCACertificatePath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCACertificatePath "/private/etc/apache2/ssl.crt"
#SSLCACertificateFile "/private/etc/apache2/ssl.crt/ca-bundle.crt"

#   Certificate Revocation Lists (CRL):
#   Set the CA revocation path where to find CA CRLs for client
#   authentication or alternatively one huge file containing all
#   of them (file must be PEM encoded)
#   Note: Inside SSLCARevocationPath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCARevocationPath "/private/etc/apache2/ssl.crl"
#SSLCARevocationFile "/private/etc/apache2/ssl.crl/ca-bundle.crl"

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/
#</Location>

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context. 
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>
<Directory "/Library/WebServer/CGI-Executables">
    SSLOptions +StdEnvVars
</Directory>

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is send and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly. 
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog "/private/var/log/apache2/ssl_request_log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

</VirtualHost>                                  
Comment by oleksiys [ 16/Jul/13 ]

Ok, looks like jboss.weld tries to retrieve client certificate, which is not available because Apache didn't forward it to us.
The exception itself doesn't have any impact on execution flow, except flooding server.log w/ the stacktrace.

We have reduced the logging level of the exception to FINE, so it will not occur in the server.log in normal case.
As a workaround you can change the logging level (via admin gui) of "org.glassfish.grizzly.http.server.util.RequestUtils" to "SEVERE"

Comment by oleksiys [ 17/Jul/13 ]

marking as fixed and closing the issue.
feel free to reopen it if needed.

thanks.

Comment by juliohm [ 04/Aug/13 ]

Greetings,

I am attempting to pass on the SSL authentication from Apache into Glassfish using mod_jk. I found the same exception reported here, and a call to

request.getAttribute("javax.servlet.request.X509Certificate");

returns null.

Other have also reported:

The exception itself doesn't have any impact on execution flow, except flooding server.log w/ the stacktrace.

Could this exception be the cause of it? Or am I doing something wrong with mod_jk at this point?

Comment by oleksiys [ 06/Aug/13 ]

@juliohm
most probably your client (may be browser) doesn't sent its certificate.
You may want to force clients to send their certificates by adding/uncommenting this in the Apache ssl.conf:

SSLVerifyClient require
Comment by juliohm [ 06/Aug/13 ]

Oddly, I already have the SSL config in the VirtualHost.

I can connect fine using a web browser and the appropriate certificates.

I've been trying to understand how the mod_jk module works, but I can't get a grip on the concept. Since it's forwarding the SSL authentication to a backend glassfish, does the java server need to receive this through its SSL listener as well (standard port 8181)? Or are the SSL credentials simply filled within the clear HTTP listener (8080)?.

Do I even need the same certificates in Java keystore as well as in Apache?

Comment by oleksiys [ 06/Aug/13 ]

Client certificate should be forwarded to Glassfish as part of Jk request, so you don't need any additional listeners neither https nor http.
But in your case you said

request.getAttribute("javax.servlet.request.X509Certificate");

return null, which IMO means Apache/mod_jk doesn't forward the certificate. Did you try to set SSLVerfyClient flag to "require"?

Comment by juliohm [ 06/Aug/13 ]

This is the basic setup I have in my VirtualHost (comments and related file paths redacted)

<VirtualHost ...>
   ......
   JkMount /gf_forward/* ajp13_worker
   SSLEngine on
   SSLProtocol all -SSLv2
   SSLHonorCipherOrder On
   SSLCipherSuite .......
   SSLCertificateFile    /path/to/certfile
   SSLCertificateKeyFile /path/to/keyfile
   SSLCACertificateFile /path/to/ca/certfile
   SSLCARevocationFile /path/to/crlfile
   SSLVerifyClient require
   SSLVerifyDepth  10
   ......
</VirtualHost>

I just find it funny that connections are correctly directed to glassfish. I can navigate whatever service is deployed in there. For example https://myserver/gf_forward/myapp works just fine. But I can see from my server.log that the request attribute "javax.servlet.request.X509Certificate" resolves to null.

I'm actually not sure what's going on

Comment by oleksiys [ 07/Aug/13 ]

Do you have this set?

JkOptions +ForwardSSLCertChain 

If it still doesn't work, can you pls. share your apache configuration (pls. send to oleksiys [at] java.net).

Comment by juliohm [ 07/Aug/13 ]

I was finally able to get it working! After several days, one of my friends had this insight.

SSLOptions +StdEnvVars +ExportCertData

You need ExportCertData enabled in SSLOptions. Now the certificate shows up in the request inside glassfish.

This is not much related to the ticket here, so I'm glad to this out of the way from this therad. Thank you very much for the help.





[GLASSFISH-20627] Downloading of larger files stalls when using AJP Created: 12/Jun/13  Updated: 20/Dec/16

Status: Open
Project: glassfish
Component/s: None
Affects Version/s: 4.0_dev
Fix Version/s: None

Type: Bug Priority: Critical
Reporter: Peter Salomonsen Assignee: michael.y.chen
Resolution: Unresolved Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu server 13.04, OpenJDK 1.7.0_21, Glassfish 4.0 b89 (official version), Apache 2


Tags: AJP, Glassfish

 Description   

I have a 6 MB file served from Glassfish through AJP/mod_jk and Apache 2. Downloading this using wget stops after approx 735 KB. If I truncate the file to e.g. 512KB downloading works fine.

Download works fine when accessing port 8080 directly, the problem is only through AJP. I've put the files in a directory under the docroot folder of domain 1.

I've set up a test case server and can also provide Amazon AMI image of this if you need this. The problem can be seen by trying downloading the following files:

original file (stalls):
http://ec2-54-228-91-14.eu-west-1.compute.amazonaws.com/gf4ajptest/motions.mp3

truncated file (works fine):
http://ec2-54-228-91-14.eu-west-1.compute.amazonaws.com/gf4ajptest/motions_truncated.mp3



 Comments   
Comment by Peter Salomonsen [ 20/Jun/13 ]

Since no response on this I've stopped the server, but as said I can provide the AMI image so that you can see the bug in action if you need it.

Comment by Simon Martinelli [ 19/Jul/13 ]

Same issue here:
Works: https://jtaf.ch:8181/jtaf/images/background.jpg
Don't: https://jtaf.ch/jtaf/images/background.jpg

Comment by traxus [ 24/Jul/13 ]

I have the same problem, downloads stall and ultimately time out. Had to rollback to GF 3.1.1, when it last time worked.





[GLASSFISH-18731] Errow when posting data with "Expect: 100-continue" header via ajp with mod_proxy_ajp in apache Created: 15/May/12  Updated: 20/Dec/16  Resolved: 18/May/12

Status: Resolved
Project: glassfish
Component/s: grizzly-kernel
Affects Version/s: 3.1.2_dev
Fix Version/s: None

Type: Bug Priority: Major
Reporter: jsl123 Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 11.04


Attachments: Text File failed.txt     Text File working.txt    
Tags: ajp, apache, expect, mod_proxy_ajp

 Description   

Hi, not sure if this is a grizzly issue or mod_proxy_ajp in apache. Anyway the problem is as follows:
If a client does a POST (presumably other methods do the same) and specifies a Expect header (tested with 100-continue as the value) then apache accepts the headers, marshalls them for ajp and returns a 100-continue response. It then starts to read the body of the message but gets a bad signature from glassfish. This results in glassfish getting a "connection reset from peer" error and return a 500 error back to apache.

I've attached dumps of the process for both a working and not working (with Expects) content.



 Comments   
Comment by jsl123 [ 15/May/12 ]

possible although not great work around are to switch apache to http 1.0 only or ignore the expect header and hope the client doesn't stick to the spec and sends the data straight after the headers without waiting for a response.

Comment by Bas van Gils [ 16/May/12 ]

I experienced the same issue with the Expect header. After thorough investigation we found out that this caused it. As a workaround we strip this header in Apache.

We also had another problem, like described here: http://marc.info/?l=tomcat-user&m=121467240210359.
The suggested solution to add JkOptions +DisableReuse in mod_jk solved it.

Comment by jsl123 [ 16/May/12 ]

I've worked around it by making apache remove the header effectively ignoring it by adding "RequestHeader unset Expect early" to the config. Although this isn't ideal, as it should break any client that sticks to the specifications (luckily most don't) as they should wait for a response before sending the rest of their request body.

Comment by oleksiys [ 18/May/12 ]

fixed

mirror Grizzly issue
http://java.net/jira/browse/GRIZZLY-1267

the patch for GF 3.1.2 can be downloaded here:
http://java.net/jira/browse/GLASSFISH-18446

Comment by jsl123 [ 18/May/12 ]

Looks to work, thanks for your quick turnaround..





[GLASSFISH-18341] Extra-content in HTML using AJP with GF 3.1.2 and https on IE Created: 08/Feb/12  Updated: 20/Dec/16  Resolved: 09/Feb/12

Status: Closed
Project: glassfish
Component/s: web_container
Affects Version/s: 3.1.2_dev
Fix Version/s: None

Type: Bug Priority: Major
Reporter: loonesp Assignee: Ryan Lubke
Resolution: Invalid Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7 64bits
IE 9
Apache 2.2


Attachments: HTML File index.jsp    
Tags: 3_1_2-exclude, ajp

 Description   

I have a GF 3.1.2 domain behind an Apache 2.2 with AJP enabled.
When I try to access a JSP file larger than 8192 bytes from IE using HTTPS, the page starts with "2000\r\n".
This page is correct:

  • if I try to connect with IE using HTTP or with Firefox (HTTP or HTTPS).
  • if the page is 8192 bytes or less.
  • if I use GF 3.1.1 with AJP.

Steps to reproduce:

  1. asadmin create-domain domain_ajp
  2. asadmin start-domain domain_ajp
  3. asadmin create-http-listener --listenerport 8009 --listeneraddress 0.0.0.0 --defaultvs server jk-connector
  4. asadmin set configs.config.server-config.network-config.network-listeners.network-listener.jk-connector.jk-enabled=true
  5. copy the attached file to domain_ajp/docroot
  6. access the page with IE on https://localhost/index.jsp
    Result: the HTML source code starts with 2000\r\n

AJP is set in Apache with:

<IfModule proxy_module>
<IfModule proxy_ajp_module>
  <Location />
    ProxyPass ajp://127.0.0.1:8009/
    ProxyPassReverse ajp://127.0.0.1:8009/
  </Location>
</IfModule>
</IfModule>


 Comments   
Comment by Amy Roh [ 08/Feb/12 ]

Did this work with 3.1.1? 3.1.1 didn't officially support mod_proxy_ajp.
You might want to try mod_jk since the max packet size for mod_proxy_ajp is 8k [1] and it might be related.

[1] http://httpd.apache.org/docs/2.2/mod/mod_proxy_ajp.html#basppacketstruct

Comment by Amy Roh [ 08/Feb/12 ]

GF 3.1.2 switched to use Grizzly ajp implementation. Assigning to Ryan for evaluation.

Comment by Ryan Lubke [ 08/Feb/12 ]

In addition to Amy's suggestion of trying mod_jk, can you capture the headers sent between IE and apache and attach them to the issue?

Comment by Ryan Lubke [ 09/Feb/12 ]

Alexey and I were discussing this issue while I was getting my environment setup. The 2000 in the response sounded like an issue with chunking.

We both were able to reproduce this and found that the apache was setting the response protocol to HTTP/1.0 along with the transfer-encoding header set to chunked. Chunking isn't supported in HTTP 1.0.

As you stated, the issue only occurred when using SSL. We checked the ssl config file and found this:

BrowserMatch ".MSIE." \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0

This configuration forces a downgrade of the response when the user-agent is MSIE.

Based on this article [1], this configuration should be changed to:

BrowserMatch ".MSIE [2-5]\.." \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0

This only forces the downgrade for older versions of IE. The comments above the BrowserMatch
explain why this is needed.

Once we updated the match expression, IE worked as expected.

[1] http://blogs.msdn.com/b/ieinternals/archive/2011/03/26/https-and-connection-close-is-your-apache-modssl-server-configuration-set-to-slow.aspx

Comment by loonesp [ 09/Feb/12 ]

It works fine when I modify the Apache configuration as suggested.
Thanks for the quick resolution.
It's strange that it worked correctly with GF 3.1.1

Does 3.1.2 officially support mod_proxy_ajp or should I use mod_jk instead ?

Comment by oleksiys [ 09/Feb/12 ]

> It works fine when I modify the Apache configuration as suggested.
> Thanks for the quick resolution.
> It's strange that it worked correctly with GF 3.1.1
In GF 3.1.1 we used different AJP implementation, which didn't use HTTP/1.1 features.

> Does 3.1.2 officially support mod_proxy_ajp or should I use mod_jk instead ?
3.1.2 should support both.

Comment by Ryan Lubke [ 09/Feb/12 ]

At this point, I think we can close this issue out.

Comment by corti [ 09/Aug/12 ]

I am using 3.1.2.2 and even by changing the Apache configuration as suggested the problem persists.

Comment by oleksiys [ 09/Aug/12 ]

can you pls. share the steps how we can reproduce it?

Comment by corti [ 09/Aug/12 ]

Sorry I just noticed that in addition of the BrowserMatch directive there was an additional

  1. SetEnvIf User-Agent ".MSIE." \
  2. nokeepalive ssl-unclean-shutdown \
  3. downgrade-1.0 force-response-1.0

which I had to comment out. It now works as expected.





Generated at Fri Jan 20 12:40:48 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.