grizzly
  1. grizzly
  2. GRIZZLY-915

Remote client connecting to secure GlassFish admin port causes Grizzly errors

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 1.9.22
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      any

      Description

      (copied from a message I sent to Ryan)

      This is blocking SQE testing.

      If do this:

      (with a newly-installed or created domain)

      asadmin start-domain
      asadmin enable-secure-admin
      asadmin stop-domain
      asadmin start-domain

      Then I can nicely use

      asadmin uptime

      from the same host and it works. (I get prompted about the DAS's self-signed cert, I accept it, and the rest is fine.)

      But if I go to some other host and try this:

      asadmin --host whereDASis uptime

      Then the redirection works, the user is prompted about the DAS's self-signed cert, I accept it, and then client times out eventually with a java.net.SocketException: Unexpected end of file from server

      The server.log for the DAS contains the error I've pasted below. The AdminAdapter has invoked GrizzlyRequest.getUserPrincipal which seems to trigger problems.

      The last error block occurs when the AdminAdapter is doing this

      res.getOutputStream().flush();

      after it has written the correct results into the output stream and just before is invokes res.finishResponse().

      So I thought I'd try wget.

      wget -t 1 -S -O - --no-check-certificate https://localhost:4848/__asadmin/uptime

      from either the local system or the remote one, I get the same error in the server.log and I see this from wget:

      -2010-11-18 11:35:37- https://localhost:4848/__asadmin/uptime
      Resolving localhost... ::1, fe80::1, 127.0.0.1
      Connecting to localhost|::1|:4848... connected.
      WARNING: cannot verify localhost's certificate, issued by `/C=US/ST=California/L=Santa Clara/O=Oracle Corporation/OU=GlassFish/CN=tims-macbook-pro-2.local':
      Self-signed certificate encountered.
      WARNING: certificate common name `tims-macbook-pro-2.local' doesn't match requested host name `localhost'.
      HTTP request sent, awaiting response... No data received.
      Giving up.

      I do not know why local asadmin works while local or remote wget fails and remote asadmin remote.

      • Tim

      [#|2010-11-18T11:27:26.771-0600|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-1;|processorTask.errorSSL
      javax.net.ssl.SSLHandshakeException: Insecure renegotiation is not allowed
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:635)
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:689)
      at com.sun.grizzly.util.SSLUtils.doPeerCertificateChain(SSLUtils.java:530)
      at com.sun.grizzly.filter.SSLReadFilter.doPeerCertificateChain(SSLReadFilter.java:337)
      at com.sun.grizzly.ssl.SSLProcessorTask.action(SSLProcessorTask.java:124)
      at com.sun.grizzly.tcp.Request.action(Request.java:432)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getAttribute(GrizzlyRequest.java:839)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getUserPrincipal(GrizzlyRequest.java:1830)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:262)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:304)
      at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:214)
      at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
      at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
      at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
      at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      at java.lang.Thread.run(Thread.java:680)

      #]

      [#|2010-11-18T11:27:26.778-0600|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-1;|processorTask.exceptionSSLcert
      javax.net.ssl.SSLException: SSLEngine is closing/closed
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:656)
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:689)
      at com.sun.grizzly.util.SSLUtils.doPeerCertificateChain(SSLUtils.java:530)
      at com.sun.grizzly.filter.SSLReadFilter.doPeerCertificateChain(SSLReadFilter.java:337)
      at com.sun.grizzly.ssl.SSLProcessorTask.action(SSLProcessorTask.java:153)
      at com.sun.grizzly.tcp.Request.action(Request.java:430)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getAttribute(GrizzlyRequest.java:834)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getUserPrincipal(GrizzlyRequest.java:1833)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:262)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:304)
      at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:214)
      at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
      at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
      at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
      at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      at java.lang.Thread.run(Thread.java:680)

      #]

      [#|2010-11-18T11:27:26.779-0600|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-1;|processorTask.errorSSL
      javax.net.ssl.SSLException: SSLEngine is closing/closed
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:656)
      at com.sun.net.ssl.internal.ssl.SSLEngineImpl.beginHandshake(SSLEngineImpl.java:689)
      at com.sun.grizzly.util.SSLUtils.doPeerCertificateChain(SSLUtils.java:530)
      at com.sun.grizzly.filter.SSLReadFilter.doPeerCertificateChain(SSLReadFilter.java:337)
      at com.sun.grizzly.ssl.SSLProcessorTask.action(SSLProcessorTask.java:124)
      at com.sun.grizzly.tcp.Request.action(Request.java:432)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getAttribute(GrizzlyRequest.java:839)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getAttribute(GrizzlyRequest.java:835)
      at com.sun.grizzly.tcp.http11.GrizzlyRequest.getUserPrincipal(GrizzlyRequest.java:1833)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:262)
      at com.sun.enterprise.v3.admin.AdminAdapter.authenticate(AdminAdapter.java:304)
      at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:214)
      at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
      at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
      at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
      at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      at java.lang.Thread.run(Thread.java:680)

      #]

      [#|2010-11-18T11:27:26.926-0600|SEVERE|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-1;|service exception
      java.lang.RuntimeException: ClientAbortException: java.io.IOException: SSLOutputWriter: CLOSED
      at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:250)
      at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
      at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
      at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:817)
      at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:718)
      at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1007)
      at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
      at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      at java.lang.Thread.run(Thread.java:680)
      Caused by: ClientAbortException: java.io.IOException: SSLOutputWriter: CLOSED
      at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:439)
      at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.flush(GrizzlyOutputBuffer.java:405)
      at com.sun.grizzly.tcp.http11.GrizzlyOutputStream.flush(GrizzlyOutputStream.java:140)
      at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:247)
      ... 17 more
      Caused by: java.io.IOException: SSLOutputWriter: CLOSED
      at com.sun.grizzly.util.SSLOutputWriter.flushChannel(SSLOutputWriter.java:98)
      at com.sun.grizzly.ssl.SSLOutputBuffer.flushChannel(SSLOutputBuffer.java:138)
      at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:398)
      at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:376)
      at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1235)
      at com.sun.grizzly.ssl.SSLProcessorTask.action(SSLProcessorTask.java:164)
      at com.sun.grizzly.tcp.Response.action(Response.java:268)
      at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:434)
      ... 20 more

      #]

        Activity

        Hide
        Ryan Lubke added a comment -

        Hi Tim,

        I'm unable to reproduce the remote asadmin issue you've reported here. It works like a charm.

        Regarding this message: javax.net.ssl.SSLException: SSLEngine is closing/closed,
        this is resolved in the current trunk of Grizzly. Even with this message being written
        to the log, it doesn't prevent asadmin from returning the uptime.

        Regarding: javax.net.ssl.SSLHandshakeException: Insecure renegotiation is not allowed
        and : ClientAbortException: java.io.IOException: SSLOutputWriter: CLOSED

        when using wget or curl, well, this appears to be an issue with the openssl libraries
        on macos.

        The insecure renegotiation issue is described in detail here [1]. However, even with the
        system property to allow such renegotiations, the client never responds as it closes the
        connection on the negotiation attempt by the server, hence the SSLOutputWriter: CLOSED
        messages.

        After digging around, it turns out that Apple neutered their SSL libraries shipped with macos
        to eliminate the renegotiation vulnerability described in [1]. To get around this I had to a)
        install a later version of openssl that has a proper fix for the renegotiation issue, and then
        b) recompile wget (mac doesn't appear to have the concept of dynamic load paths).'

        The following instructions are pretty high level and assume much. So if you have issues with it,
        let me know.

        1. install latest openssl - may need to install mac ports
          sudo port install libevent openssl
        1. openssl is installed in /opt/local
        2. download wget tar ball and compile sing :
          ./configure --with-libssl-prefix=/opt/local
        1. verify the new wget binary references the proper
        2. openssl libraries by running relative to the directory
        3. in which the configure command above was run
          #
        4. It should produce output similar to:
        5. /opt/local/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        6. /opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        7. /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.1)

        otool -L src/wget

        1. verify wget works against GF
          ./wget -d -t 1 -S -O - --no-check-certificate https://localhost:4848/__asadmin/uptime

        ---------------------------------------------
        HTTP/1.1 200 OK
        Content-Type: text/html
        Date: Fri, 19 Nov 2010 00:41:28 GMT
        Connection: close
        Length: unspecified [text/html]
        Saving to: `STDOUT'

        [<=> ] 0 --.-K/s <html><head/><body><h1>GlassFish uptime AdminCommand command report</h1><br><br>
        Exit Code : SUCCESS
        <hr>
        <h2>Up 14 secs</h2>
        <hr>
        [ <=> ] 145 --.-K/s in 0s

        Closed 3/SSL 0x00000001003195a0
        -----------------------------------------------

        Show
        Ryan Lubke added a comment - Hi Tim, I'm unable to reproduce the remote asadmin issue you've reported here. It works like a charm. Regarding this message: javax.net.ssl.SSLException: SSLEngine is closing/closed, this is resolved in the current trunk of Grizzly. Even with this message being written to the log, it doesn't prevent asadmin from returning the uptime. Regarding: javax.net.ssl.SSLHandshakeException: Insecure renegotiation is not allowed and : ClientAbortException: java.io.IOException: SSLOutputWriter: CLOSED when using wget or curl, well, this appears to be an issue with the openssl libraries on macos. The insecure renegotiation issue is described in detail here [1] . However, even with the system property to allow such renegotiations, the client never responds as it closes the connection on the negotiation attempt by the server, hence the SSLOutputWriter: CLOSED messages. After digging around, it turns out that Apple neutered their SSL libraries shipped with macos to eliminate the renegotiation vulnerability described in [1] . To get around this I had to a) install a later version of openssl that has a proper fix for the renegotiation issue, and then b) recompile wget (mac doesn't appear to have the concept of dynamic load paths).' The following instructions are pretty high level and assume much. So if you have issues with it, let me know. install latest openssl - may need to install mac ports sudo port install libevent openssl openssl is installed in /opt/local download wget tar ball and compile sing : ./configure --with-libssl-prefix=/opt/local verify the new wget binary references the proper openssl libraries by running relative to the directory in which the configure command above was run # It should produce output similar to: /opt/local/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /opt/local/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.1) otool -L src/wget verify wget works against GF ./wget -d -t 1 -S -O - --no-check-certificate https://localhost:4848/__asadmin/uptime --------------------------------------------- HTTP/1.1 200 OK Content-Type: text/html Date: Fri, 19 Nov 2010 00:41:28 GMT Connection: close Length: unspecified [text/html] Saving to: `STDOUT' [<=> ] 0 --.-K/s <html><head/><body><h1>GlassFish uptime AdminCommand command report</h1><br><br> Exit Code : SUCCESS <hr> <h2>Up 14 secs</h2> <hr> [ <=> ] 145 --.-K/s in 0s Closed 3/SSL 0x00000001003195a0 -----------------------------------------------
        Hide
        Ryan Lubke added a comment -

        I spoke with Tim briefly this morning. He hasn't had time to re-test. He'll be following up later today.

        Show
        Ryan Lubke added a comment - I spoke with Tim briefly this morning. He hasn't had time to re-test. He'll be following up later today.
        Hide
        Ryan Lubke added a comment -

        Latest status. When using JDK 1.6.0_22 across all instances, asadmin connection problems go away.

        It's probably best to use _22 instead of trying to mix and match secure and insecure jdk implementations.

        Show
        Ryan Lubke added a comment - Latest status. When using JDK 1.6.0_22 across all instances, asadmin connection problems go away. It's probably best to use _22 instead of trying to mix and match secure and insecure jdk implementations.

          People

          • Assignee:
            Ryan Lubke
            Reporter:
            Tim Quinn
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: