glassfish
  1. glassfish
  2. GLASSFISH-14734

Many asadmin commands failed for instances that were created before a security was enabled.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Cannot Reproduce
    • Affects Version/s: 3.1
    • Fix Version/s: not determined
    • Component/s: admin
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • Issuezilla Id:
      14,734

      Description

      Build 30. If instances were created before enable-secure-admin, then many
      commands with these instances - failed. There has to be a method to enable
      security for the instances that were created before.

      =========================================================

      Command create-admin-object failed on server instance in4:
      javax.net.ssl.SSLHandshakeException: Remote host closed connection during
      handshake
      Command create-jdbc-resource failed on server instance in4:
      javax.net.ssl.SSLHandshakeException: Remote host closed connection during
      handshake
      org.glassfish.api.admin.CommandException: remote failure: Error trying to stop
      the instance named in
      1 : javax.net.ssl.SSLHandshakeException: Remote host closed connection during
      handshake
      Command stop-instance failed.
      Application deployed with name webapps-caching.
      WARNING: Command _deploy did not complete successfully on server instance in1:
      javax.net.ssl.SSLHand
      shakeException: Remote host closed connection during handshake
      WARNING: Command _deploy did not complete successfully on server instance in2:
      javax.net.ssl.SSLHand
      shakeException: Remote host closed connection during handshake
      Command deploy completed with warnings.

      Command set failed on server instance in1: javax.net.ssl.SSLHandshakeException:
      Remote host closed connection during handshake
      Command set failed on server instance in2: javax.net.ssl.SSLHandshakeException:
      Remote host closed connection during handshake
      org.glassfish.api.admin.CommandException: remote failure: Error trying to
      restart the instance named
      in4 : javax.net.ssl.SSLHandshakeException: Remote host closed connection
      during handshake
      Command restart-instance failed.

        Activity

        Hide
        Tom Mueller added a comment -

        We don't support upgrade from one build to the next. If there is a problem with upgrade of.
        Instances that were created with 2.1, then reopen the bug.

        Show
        Tom Mueller added a comment - We don't support upgrade from one build to the next. If there is a problem with upgrade of. Instances that were created with 2.1, then reopen the bug.
        Hide
        easarina added a comment -

        I did not do upgrade!
        The fresh build was installed. Then was created a cluster with instances. After
        that security was enabled. And then many asadmin commands failed for that
        instances. There has to be a method to enable security for the instances that were
        created before the security was enabled.

        Show
        easarina added a comment - I did not do upgrade! The fresh build was installed. Then was created a cluster with instances. After that security was enabled. And then many asadmin commands failed for that instances. There has to be a method to enable security for the instances that were created before the security was enabled.
        Hide
        easarina added a comment -

        Installed b30, started domain, removed truststore, enabled security, executed
        list-instances. Then created cluster, instances (create-local-instance), started
        instances (start-local-instance). After that stop-instance, stop-cluster, restart-
        instance, create-jdbc-pool - failed with a message:

        SSLHandshakeException: Remote host closed connection during
        handshake

        Show
        easarina added a comment - Installed b30, started domain, removed truststore, enabled security, executed list-instances. Then created cluster, instances (create-local-instance), started instances (start-local-instance). After that stop-instance, stop-cluster, restart- instance, create-jdbc-pool - failed with a message: SSLHandshakeException: Remote host closed connection during handshake
        Hide
        Tom Mueller added a comment -

        Sorry for misreading the initial bug report.

        Why did you remove the truststore as part of the test?
        Does the problem occur if the truststore is not removed?

        I find that when I repeat the test without removing the truststore, then
        everything works fine. I suspect that when you remove the truststore before
        enabling security, this causes it to be regenerated so that the DAS now has
        different certificates than have been downloaded to the instances. After this, a
        manual sync (export-sync-bundle, import-sync-bundle) would be needed to
        reestablish synchronization between the instances.

        Assigning to Tim for confirmation.

        Show
        Tom Mueller added a comment - Sorry for misreading the initial bug report. Why did you remove the truststore as part of the test? Does the problem occur if the truststore is not removed? I find that when I repeat the test without removing the truststore, then everything works fine. I suspect that when you remove the truststore before enabling security, this causes it to be regenerated so that the DAS now has different certificates than have been downloaded to the instances. After this, a manual sync (export-sync-bundle, import-sync-bundle) would be needed to reestablish synchronization between the instances. Assigning to Tim for confirmation.
        Hide
        Tim Quinn added a comment -

        There are several things wrong with this scenario.

        1. The Grizzly team says that any server that is running - whether the DAS or an
        instance - must be restarted after enabling or disabling secure admin because of
        the changes to the admin listener configuration. They might be able to remove
        that restriction but currently that's the requirement. The two commands -
        enable-secure-admin and disable-secure-admin - need to say this. That's on my
        to-do list.

        2. I suspect that Tom is correct, that the security subsystem recreates the
        truststore. The security team would know; this is not something that the secure
        admin feature itself controls. I've added Kumar to the cc list.

        The keystore and truststore on the DAS and the instances contain the keys and
        certificates that allow secure admin to work. We added the two commands and the
        infrastructure behind them to save administrators from manually dealing with
        keys and certs and keystores and truststores and Grizzly configurations
        themselves. Users should not expect to be able to remove internal GlassFish
        files and have the system continue to work properly.

        Is removing the truststore part of the test definition? If so, what's the user
        scenario in which we expect users to do this?

        I am closing this as "invalid" because this is not something we support or users
        should expect to work. If I have missed something please reopen it with more
        information.

        Show
        Tim Quinn added a comment - There are several things wrong with this scenario. 1. The Grizzly team says that any server that is running - whether the DAS or an instance - must be restarted after enabling or disabling secure admin because of the changes to the admin listener configuration. They might be able to remove that restriction but currently that's the requirement. The two commands - enable-secure-admin and disable-secure-admin - need to say this. That's on my to-do list. 2. I suspect that Tom is correct, that the security subsystem recreates the truststore. The security team would know; this is not something that the secure admin feature itself controls. I've added Kumar to the cc list. The keystore and truststore on the DAS and the instances contain the keys and certificates that allow secure admin to work. We added the two commands and the infrastructure behind them to save administrators from manually dealing with keys and certs and keystores and truststores and Grizzly configurations themselves. Users should not expect to be able to remove internal GlassFish files and have the system continue to work properly. Is removing the truststore part of the test definition? If so, what's the user scenario in which we expect users to do this? I am closing this as "invalid" because this is not something we support or users should expect to work. If I have missed something please reopen it with more information.
        Hide
        Tim Quinn added a comment -

        I now think I understand that the truststore you referred to is the
        .asadmintruststore file that is used only by asadmin. I had thought you meant
        you had removed the DAS truststore.

        Removing .asadmintruststore is OK. It is not necessary but it is OK.

        You might want a single test that includes removing ~/.asadmintruststore to
        verify that the DAS offers its certificate as expected. But in subsequent tests
        I'd suggest you leave the .asadmintruststore file alone. In fact that would
        more closely mirror what most users will see and do.

        Sorry for the confusion.

        Show
        Tim Quinn added a comment - I now think I understand that the truststore you referred to is the .asadmintruststore file that is used only by asadmin. I had thought you meant you had removed the DAS truststore. Removing .asadmintruststore is OK. It is not necessary but it is OK. You might want a single test that includes removing ~/.asadmintruststore to verify that the DAS offers its certificate as expected. But in subsequent tests I'd suggest you leave the .asadmintruststore file alone. In fact that would more closely mirror what most users will see and do. Sorry for the confusion.
        Hide
        easarina added a comment -

        Remove .asadmintruststore before security was enabled was include in the Tim's
        script, that he presented. I've strongly followed Tim's steps. And then many
        commands failed for the instances that were created, using local commands. By the
        way, I've trued did not remove .asadmintruststore before security was enabled, in
        this case readdressing to https was not seen.

        Show
        easarina added a comment - Remove .asadmintruststore before security was enabled was include in the Tim's script, that he presented. I've strongly followed Tim's steps. And then many commands failed for the instances that were created, using local commands. By the way, I've trued did not remove .asadmintruststore before security was enabled, in this case readdressing to https was not seen.
        Hide
        easarina added a comment -

        I did the steps from Tim's script:
        =============================================================
        bash-3.2$ # Remove any trace of the DAS's self-signed cert

        bash-3.2$ rm ~/.asadmntruststore
        rm: /Users/Tim/.asadmintruststore: No such file or directory

        bash-3.2$

        bash-3.2$ # Enable secure admin

        bash-3.2$ asadmin enable-secure-admin

        Command enable-secure-admin executed successfully.

        bash-3.2$ #

        bash-3.2$ #

        bash-3.2$ # Restart the DAS

        bash-3.2$ asadmin stop-domain

        Waiting for the domain to stop a.sadmin. start-.doma

        Command stop-domain executed successfully.

        bash-3.2$ asadmin start-domain

        Waiting for domain1 to start ................................................

        Successfully started the domain : domain1

        domain Location:
        /Users/Tim/asgroup/v3/H/publish/glassfish3/glassfish/domains/domain1

        Log File:
        /Users/Tim/asgroup/v3/H/publish/glassfish3/glassfish/domains/domain1/logs/server
        .log

        Admin Port: 4848

        Command start-domain executed successfully.
        ========================================================

        Then in this testes eexecuted list-instances to issued a certificate.

        Then created local instances and run a test, most tests failed, the main issue
        is decribed in this bug.

        Show
        easarina added a comment - I did the steps from Tim's script: ============================================================= bash-3.2$ # Remove any trace of the DAS's self-signed cert bash-3.2$ rm ~/.asadmntruststore rm: /Users/Tim/.asadmintruststore: No such file or directory bash-3.2$ bash-3.2$ # Enable secure admin bash-3.2$ asadmin enable-secure-admin Command enable-secure-admin executed successfully. bash-3.2$ # bash-3.2$ # bash-3.2$ # Restart the DAS bash-3.2$ asadmin stop-domain Waiting for the domain to stop a.sadmin. start-.doma Command stop-domain executed successfully. bash-3.2$ asadmin start-domain Waiting for domain1 to start ................................................ Successfully started the domain : domain1 domain Location: /Users/Tim/asgroup/v3/H/publish/glassfish3/glassfish/domains/domain1 Log File: /Users/Tim/asgroup/v3/H/publish/glassfish3/glassfish/domains/domain1/logs/server .log Admin Port: 4848 Command start-domain executed successfully. ======================================================== Then in this testes eexecuted list-instances to issued a certificate. Then created local instances and run a test, most tests failed, the main issue is decribed in this bug.
        Hide
        easarina added a comment -

        Re-run the tests using latests b30. Did not see this issue any more.

        Show
        easarina added a comment - Re-run the tests using latests b30. Did not see this issue any more.
        Hide
        Tim Quinn added a comment -

        Closing. Cannot reproduce.

        Show
        Tim Quinn added a comment - Closing. Cannot reproduce.
        Hide
        easarina added a comment -

        I was able to reproduce this issue using 3.1.2 fcs build 23. I've unzipped the build, started domain, executed change-admin-password, the executed the follow steps. As you can see I was not able to stop an instance, also the instance. According to list-instance, the instance was not running, according to start-instance, it was running.

        asadmin enable-secure-admin
        Enter admin user name> admin
        Enter admin password for user "admin">
        You must restart all running servers for the change in secure admin to take effect.
        Command enable-secure-admin executed successfully.
        asadmin disable-secure-admin
        Enter admin user name> admin
        Enter admin password for user "admin">
        You must restart all running servers for the change in secure admin to take effect.
        Command disable-secure-admin executed successfully.
        asadmin create-node-ssh --nodehost jed-asqe-7 node2
        Enter admin user name> admin
        Enter admin password for user "admin">
        Command create-node-ssh executed successfully.
        asadmin create-instance --node node2 i2
        Enter admin user name> admin
        Enter admin password for user "admin">
        Command _create-instance-filesystem executed successfully.
        Port Assignments for server instance i2:
        JMX_SYSTEM_CONNECTOR_PORT=28687
        JMS_PROVIDER_PORT=27677
        HTTP_LISTENER_PORT=28081
        ASADMIN_LISTENER_PORT=24849
        JAVA_DEBUGGER_PORT=29010
        IIOP_SSL_LISTENER_PORT=23821
        IIOP_LISTENER_PORT=23701
        OSGI_SHELL_TELNET_PORT=26667
        HTTP_SSL_LISTENER_PORT=28182
        IIOP_SSL_MUTUALAUTH_PORT=23921
        The instance, i2, was created on host jed-asqe-7
        Command create-instance executed successfully.
        asadmin start-instance i2
        Enter admin user name> admin
        Enter admin password for user "admin">
        Waiting for i2 to start ...................................
        Successfully started the instance: i2
        instance Location: /opt/glassfish3/glassfish/nodes/node2/i2
        Log File: /opt/glassfish3/glassfish/nodes/node2/i2/logs/server.log
        Admin Port: 24849
        Command start-local-instance executed successfully.
        The instance, i2, was started on host jed-asqe-7
        Command start-instance executed successfully.
        asadmin stop-instance i2
        Enter admin user name> admin
        Enter admin password for user "admin">
        remote failure: Error trying to stop the instance named i2 : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        Command stop-instance failed.
        asadmin enable-secure-admin
        Enter admin user name> admin
        Enter admin password for user "admin">
        You must restart all running servers for the change in secure admin to take effect.
        Command enable-secure-admin executed successfully.
        asadmin stop-instance i2
        Enter admin user name> admin
        Enter admin password for user "admin">
        remote failure: Error trying to stop the instance named i2 : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
        Command stop-instance failed.

        asadmin list-instances
        Enter admin user name> admin
        Enter admin password for user "admin">
        i2 not running
        Command list-instances executed successfully.
        asadmin start-instance i2
        Enter admin user name> admin
        Enter admin password for user "admin">
        Instance i2 is already running.
        Command start-instance executed successfully.
        =========================================================

        Reproduced the issue on two platforms.

        Show
        easarina added a comment - I was able to reproduce this issue using 3.1.2 fcs build 23. I've unzipped the build, started domain, executed change-admin-password, the executed the follow steps. As you can see I was not able to stop an instance, also the instance. According to list-instance, the instance was not running, according to start-instance, it was running. asadmin enable-secure-admin Enter admin user name> admin Enter admin password for user "admin"> You must restart all running servers for the change in secure admin to take effect. Command enable-secure-admin executed successfully. asadmin disable-secure-admin Enter admin user name> admin Enter admin password for user "admin"> You must restart all running servers for the change in secure admin to take effect. Command disable-secure-admin executed successfully. asadmin create-node-ssh --nodehost jed-asqe-7 node2 Enter admin user name> admin Enter admin password for user "admin"> Command create-node-ssh executed successfully. asadmin create-instance --node node2 i2 Enter admin user name> admin Enter admin password for user "admin"> Command _create-instance-filesystem executed successfully. Port Assignments for server instance i2: JMX_SYSTEM_CONNECTOR_PORT=28687 JMS_PROVIDER_PORT=27677 HTTP_LISTENER_PORT=28081 ASADMIN_LISTENER_PORT=24849 JAVA_DEBUGGER_PORT=29010 IIOP_SSL_LISTENER_PORT=23821 IIOP_LISTENER_PORT=23701 OSGI_SHELL_TELNET_PORT=26667 HTTP_SSL_LISTENER_PORT=28182 IIOP_SSL_MUTUALAUTH_PORT=23921 The instance, i2, was created on host jed-asqe-7 Command create-instance executed successfully. asadmin start-instance i2 Enter admin user name> admin Enter admin password for user "admin"> Waiting for i2 to start ................................... Successfully started the instance: i2 instance Location: /opt/glassfish3/glassfish/nodes/node2/i2 Log File: /opt/glassfish3/glassfish/nodes/node2/i2/logs/server.log Admin Port: 24849 Command start-local-instance executed successfully. The instance, i2, was started on host jed-asqe-7 Command start-instance executed successfully. asadmin stop-instance i2 Enter admin user name> admin Enter admin password for user "admin"> remote failure: Error trying to stop the instance named i2 : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake Command stop-instance failed. asadmin enable-secure-admin Enter admin user name> admin Enter admin password for user "admin"> You must restart all running servers for the change in secure admin to take effect. Command enable-secure-admin executed successfully. asadmin stop-instance i2 Enter admin user name> admin Enter admin password for user "admin"> remote failure: Error trying to stop the instance named i2 : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake Command stop-instance failed. asadmin list-instances Enter admin user name> admin Enter admin password for user "admin"> i2 not running Command list-instances executed successfully. asadmin start-instance i2 Enter admin user name> admin Enter admin password for user "admin"> Instance i2 is already running. Command start-instance executed successfully. ========================================================= Reproduced the issue on two platforms.
        Hide
        Tim Quinn added a comment -

        I was not able to reproduce this problem using the current trunk for GlassFish 4.0.

        We did have issues with earlier releases but to my knowledge those were all fixed.

        If this problem recurs with 4.0 please re-open the issue.

        Show
        Tim Quinn added a comment - I was not able to reproduce this problem using the current trunk for GlassFish 4.0. We did have issues with earlier releases but to my knowledge those were all fixed. If this problem recurs with 4.0 please re-open the issue.

          People

          • Assignee:
            Tim Quinn
            Reporter:
            easarina
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: