[GLASSFISH-20694] Glassfish 4.0 and jk Unable to populate SSL attributes Created: 10/Jul/13  Updated: 19/Sep/14  Resolved: 17/Jul/13

Status: Resolved
Project: glassfish
Component/s: web_container
Affects Version/s: 4.0
Fix Version/s: 4.1

Type: Bug Priority: Major
Reporter: buddypine Assignee: oleksiys
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Glassfish 4.0 b89, Apache 2.2.22 mod_proxy_ajp

Tags: ajp, apache, proxy


I am using a jk enabled listener behind Apache.

GF listener created as follows:
% asadmin create-network-listener --jkenabled true --protocol http-listener-1 --listenerport 8009 jk-listener

Apache httpd-ssl.conf mod_proxy_ajp
ProxyPass / ajp://localhost:8009/
ProxyPassReverse / ajp://localhost:8009/

All seems to work fine and I can connect using SSL to Apache which proxies the requests through to Glassfish but I get the exception below in the logs (this does not happen with GF 3):

[2013-07-10T15:06:37.929+0200] [glassfish 4.0] [WARNING] [] [org.glassfish.grizzly.http.server.util.RequestUtils] [tid: _ThreadID=53 _ThreadName=jk-listener(4)] [timeMillis: 1373461597929] [levelValue: 900] [[
Unable to populate SSL attributes
java.lang.IllegalStateException: SSLEngine is null
at org.glassfish.grizzly.ssl.SSLSupportImpl.<init>(SSLSupportImpl.java:87)
at org.glassfish.grizzly.http.server.util.RequestUtils.populateSSLAttributes(RequestUtils.java:85)
at org.glassfish.grizzly.http.server.Request.getAttribute(Request.java:865)
at org.apache.catalina.connector.Request.populateSSLAttributes(Request.java:4581)
at org.apache.catalina.connector.Request.getAttributeNames(Request.java:1412)
at org.apache.catalina.connector.RequestFacade.getAttributeNames(RequestFacade.java:367)
at org.jboss.weld.context.beanstore.http.RequestBeanStore.getAttributeNames(RequestBeanStore.java:48)
at org.jboss.weld.context.beanstore.AttributeBeanStore.getPrefixedAttributeNames(AttributeBeanStore.java:207)
at org.jboss.weld.context.beanstore.AttributeBeanStore.attach(AttributeBeanStore.java:106)
at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:52)
at org.jboss.weld.context.http.HttpRequestContextImpl.associate(HttpRequestContextImpl.java:37)
at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:190)
at org.apache.catalina.core.StandardContext.fireRequestInitializedEvent(StandardContext.java:5225)
at org.apache.catalina.core.StandardHostValve.preInvoke(StandardHostValve.java:647)
at org.apache.catalina.core.StandardHostValve.__invoke(StandardHostValve.java:166)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:357)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:260)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:188)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:191)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:168)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:189)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:838)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:113)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:564)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:544)
at java.lang.Thread.run(Thread.java:724)

Comment by Shing Wai Chan [ 10/Jul/13 ]

The exception is from org.glassfish.grizzly.ssl.SSLSupportImpl in Grizzly. Assign to Alexey for further investigation.

Comment by oleksiys [ 12/Jul/13 ]

looks like apache doesn't pass all the SSL information to Glassfish.
Can you pls. share the entire ssl.conf?

Comment by buddypine [ 12/Jul/13 ]


# This is the Apache server configuration file providing SSL support.
# It contains the configuration directives to instruct the server how to
# serve pages over an https connection. For detailing information about these 
# directives see <URL:http://httpd.apache.org/docs/2.2/mod/mod_ssl.html>
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  

# Pseudo Random Number Generator (PRNG):
# Configure one or more sources to seed the PRNG of the SSL library.
# The seed data should be of good random quality.
# WARNING! On some platforms /dev/random blocks if not enough entropy
# is available. This means you then cannot use the /dev/random device
# because it would lead to very long connection times (as long as
# it requires to make more entropy available). But usually those
# platforms additionally provide a /dev/urandom device which doesn't
# block. So, if available, use this one instead. Read the mod_ssl User
# Manual for more details.
#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed startup file:/dev/urandom 512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512

# When we also provide SSL we have to listen to the 
# standard HTTP port (see above) and to the HTTPS port
# Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
#       Listen directives: "Listen [::]:443" and "Listen"
Listen 443

##  SSL Global Context
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.

#   Some MIME-types for downloading Certificates and CRLs
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#   Pass Phrase Dialog:
#   Configure the pass phrase gathering process.
#   The filtering dialog program (`builtin' is a internal
#   terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog  builtin

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism 
#   to use and second the expiring timeout (in seconds).
#SSLSessionCache         "dbm:/private/var/run/ssl_scache"
SSLSessionCache        "shmcb:/private/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300

#   Semaphore:
#   Configure the path to the mutual exclusion semaphore the
#   SSL engine uses internally for inter-process synchronization. 
SSLMutex  "file:/private/var/run/ssl_mutex"

## SSL Virtual Host Context

<VirtualHost _default_:443>

#   General setup for the virtual host
DocumentRoot "/Library/WebServer/Documents"

ServerAdmin you@example.com
ErrorLog "/private/var/log/apache2/error_log"
TransferLog "/private/var/log/apache2/access_log"

ServerName myapp.local.inversebit.com:443
ProxyPass / ajp://myapp.local.inversebit.com:8009/
ProxyPassReverse / ajp://myapp.local.inversebit.com:8009/

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   SSL Cipher Suite:
#   List the ciphers that the client is permitted to negotiate.
#   See the mod_ssl documentation for a complete list.

#   Server Certificate:
#   Point SSLCertificateFile at a PEM encoded certificate.  If
#   the certificate is encrypted, then you will be prompted for a
#   pass phrase.  Note that a kill -HUP will prompt again.  Keep
#   in mind that if you have both an RSA and a DSA certificate you
#   can configure both in parallel (to also allow the use of DSA
#   ciphers, etc.)
#SSLCertificateFile "/private/etc/apache2/server.crt"
#SSLCertificateFile "/private/etc/apache2/server-dsa.crt"

SSLCertificateFile /Users/phill/etc/certs/newcert.pem
SSLCertificateKeyFile /Users/phill/etc/certs/webserver.nopass.key
SSLCACertificateFile /Users/phill/etc/certs/demoCA/cacert.pem
SSLCARevocationPath /Users/phill/etc/certs/demoCA/crl

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
#SSLCertificateKeyFile "/private/etc/apache2/server.key"
#SSLCertificateKeyFile "/private/etc/apache2/server-dsa.key"

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convinience.
#SSLCertificateChainFile "/private/etc/apache2/server-ca.crt"

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#   Note: Inside SSLCACertificatePath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCACertificatePath "/private/etc/apache2/ssl.crt"
#SSLCACertificateFile "/private/etc/apache2/ssl.crt/ca-bundle.crt"

#   Certificate Revocation Lists (CRL):
#   Set the CA revocation path where to find CA CRLs for client
#   authentication or alternatively one huge file containing all
#   of them (file must be PEM encoded)
#   Note: Inside SSLCARevocationPath you need hash symlinks
#         to point to the certificate files. Use the provided
#         Makefile to update the hash symlinks after changes.
#SSLCARevocationPath "/private/etc/apache2/ssl.crl"
#SSLCARevocationFile "/private/etc/apache2/ssl.crl/ca-bundle.crl"

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context. 
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
<Directory "/Library/WebServer/CGI-Executables">
    SSLOptions +StdEnvVars

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is send and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly. 
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
BrowserMatch ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog "/private/var/log/apache2/ssl_request_log" \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"

Comment by oleksiys [ 16/Jul/13 ]

Ok, looks like jboss.weld tries to retrieve client certificate, which is not available because Apache didn't forward it to us.
The exception itself doesn't have any impact on execution flow, except flooding server.log w/ the stacktrace.

We have reduced the logging level of the exception to FINE, so it will not occur in the server.log in normal case.
As a workaround you can change the logging level (via admin gui) of "org.glassfish.grizzly.http.server.util.RequestUtils" to "SEVERE"

Comment by oleksiys [ 17/Jul/13 ]

marking as fixed and closing the issue.
feel free to reopen it if needed.


Comment by juliohm [ 04/Aug/13 ]


I am attempting to pass on the SSL authentication from Apache into Glassfish using mod_jk. I found the same exception reported here, and a call to


returns null.

Other have also reported:

The exception itself doesn't have any impact on execution flow, except flooding server.log w/ the stacktrace.

Could this exception be the cause of it? Or am I doing something wrong with mod_jk at this point?

Comment by oleksiys [ 06/Aug/13 ]

most probably your client (may be browser) doesn't sent its certificate.
You may want to force clients to send their certificates by adding/uncommenting this in the Apache ssl.conf:

SSLVerifyClient require
Comment by juliohm [ 06/Aug/13 ]

Oddly, I already have the SSL config in the VirtualHost.

I can connect fine using a web browser and the appropriate certificates.

I've been trying to understand how the mod_jk module works, but I can't get a grip on the concept. Since it's forwarding the SSL authentication to a backend glassfish, does the java server need to receive this through its SSL listener as well (standard port 8181)? Or are the SSL credentials simply filled within the clear HTTP listener (8080)?.

Do I even need the same certificates in Java keystore as well as in Apache?

Comment by oleksiys [ 06/Aug/13 ]

Client certificate should be forwarded to Glassfish as part of Jk request, so you don't need any additional listeners neither https nor http.
But in your case you said


return null, which IMO means Apache/mod_jk doesn't forward the certificate. Did you try to set SSLVerfyClient flag to "require"?

Comment by juliohm [ 06/Aug/13 ]

This is the basic setup I have in my VirtualHost (comments and related file paths redacted)

<VirtualHost ...>
   JkMount /gf_forward/* ajp13_worker
   SSLEngine on
   SSLProtocol all -SSLv2
   SSLHonorCipherOrder On
   SSLCipherSuite .......
   SSLCertificateFile    /path/to/certfile
   SSLCertificateKeyFile /path/to/keyfile
   SSLCACertificateFile /path/to/ca/certfile
   SSLCARevocationFile /path/to/crlfile
   SSLVerifyClient require
   SSLVerifyDepth  10

I just find it funny that connections are correctly directed to glassfish. I can navigate whatever service is deployed in there. For example https://myserver/gf_forward/myapp works just fine. But I can see from my server.log that the request attribute "javax.servlet.request.X509Certificate" resolves to null.

I'm actually not sure what's going on

Comment by oleksiys [ 07/Aug/13 ]

Do you have this set?

JkOptions +ForwardSSLCertChain 

If it still doesn't work, can you pls. share your apache configuration (pls. send to oleksiys [at] java.net).

Comment by juliohm [ 07/Aug/13 ]

I was finally able to get it working! After several days, one of my friends had this insight.

SSLOptions +StdEnvVars +ExportCertData

You need ExportCertData enabled in SSLOptions. Now the certificate shows up in the request inside glassfish.

This is not much related to the ticket here, so I'm glad to this out of the way from this therad. Thank you very much for the help.

Generated at Sun Aug 02 14:53:18 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.