glassfish
  1. glassfish
  2. GLASSFISH-16565

AIX 6.1, created a cluster, but: "GMS failed to start"

    Details

      Description

      Aix 6.1, Glassfish 3.1.1 build 04. Installed the build on two machines, configured ssh without password between the machines. Started domain. Then created a cluster, the cluster was created successfully, according a message at the terminal window. But during the cluster creation in the server.log I saw such error messages:

      ================================================================================

      [#|2011-05-05T14:54:31.365-0700|INFO|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=11;_Thre
      adName=Thread-8;|GRIZZLY0001: Starting Grizzly Framework 1.9.34 - 5/5/11 2:54 PM|#]

      [#|2011-05-05T14:54:31.370-0700|CONFIG|glassfish3.1|ShoalLogger|_ThreadID=10;_ThreadName=Thread-8;|Grizzly control
      ler listening on /0:0:0:0:0:0:0:0:9164. Controller started in 9 ms|#]

      [#|2011-05-05T14:54:31.371-0700|SEVERE|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=10;_Thread
      Name=Thread-8;|GMSAD1017: GMS failed to start. See stack trace for additional information.
      com.sun.enterprise.ee.cms.core.GMSException: failed to join group c2
      at com.sun.enterprise.ee.cms.impl.base.GMSContextImpl.join(GMSContextImpl.java:182)
      at com.sun.enterprise.ee.cms.impl.common.GroupManagementServiceImpl.join(GroupManagementServiceImpl.java:3
      82)
      at org.glassfish.gms.GMSAdapterImpl.initializeGMS(GMSAdapterImpl.java:576)
      at org.glassfish.gms.GMSAdapterImpl.initialize(GMSAdapterImpl.java:199)
      at org.glassfish.gms.bootstrap.GMSAdapterService.loadModule(GMSAdapterService.java:218)
      at org.glassfish.gms.bootstrap.GMSAdapterService.checkCluster(GMSAdapterService.java:192)
      at org.glassfish.gms.bootstrap.GMSAdapterService.access$100(GMSAdapterService.java:79)
      at org.glassfish.gms.bootstrap.GMSAdapterService$1.changed(GMSAdapterService.java:248)
      at org.jvnet.hk2.config.ConfigSupport.sortAndDispatch(ConfigSupport.java:289)
      at org.glassfish.gms.bootstrap.GMSAdapterService.changed(GMSAdapterService.java:240)
      at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:379)
      at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:369)
      at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:259)
      at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:257)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:315)
      at java.util.concurrent.FutureTask.run(FutureTask.java:150)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:736)
      Caused by: com.sun.enterprise.ee.cms.core.GMSException: initialization failure
      at com.sun.enterprise.mgmt.ClusterManager.<init>(ClusterManager.java:142)
      at com.sun.enterprise.ee.cms.impl.base.GroupCommunicationProviderImpl.initializeGroupCommunicationProvider
      (GroupCommunicationProviderImpl.java:164)
      at com.sun.enterprise.ee.cms.impl.base.GMSContextImpl.join(GMSContextImpl.java:176)
      ... 18 more
      Caused by: java.io.IOException: can not find a first InetAddress
      at com.sun.enterprise.mgmt.transport.grizzly.GrizzlyNetworkManager.start(GrizzlyNetworkManager.java:376)
      at com.sun.enterprise.mgmt.ClusterManager.<init>(ClusterManager.java:140)
      ... 20 more

      #]

      ===========================================================

      And then, when instances were created for the cluster, I did not see any GMS messages or events in the server.log.

        Activity

        Hide
        Tom Mueller added a comment -

        Reassigning to GMS subcategory.

        Show
        Tom Mueller added a comment - Reassigning to GMS subcategory.
        Hide
        Joe Fialli added a comment -

        More information is needed to investigate why basic lookup of a network address is not working.
        We need to investigate why this failure occurred in reported stack trace.

        Caused by: java.io.IOException: can not find a first InetAddress
        at com.sun.enterprise.mgmt.transport.grizzly.GrizzlyNetworkManager.start(GrizzlyNetworkManager.java:376)

        A simple NetworkUtility method is failing to find an InternetAddress for the machine.
        There is a simple test that can be run to diagnose why the network configuration is
        not working correctly.

        $ cd <GlassFishInstallation>/glassfish/modules
        $ java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility
        AllLocalAddresses() = [/10.152.23.224, /fe80:0:0:0:223:32ff:fe97:5cf7%4, /fe80:0:0:0:0:0:0:1%1]
        getFirstNetworkInterface() = name:en0 (en0) index: 4 addresses:
        /10.152.23.224;
        /fe80:0:0:0:223:32ff:fe97:5cf7%4;

        getFirstInetAddress( true ) = /fe80:0:0:0:223:32ff:fe97:5cf7%4
        getFirstInetAddress( false ) = /10.152.23.224
        getFirstNetworkInteface() = name:en0 (en0) index: 4 addresses:
        /10.152.23.224;
        /fe80:0:0:0:223:32ff:fe97:5cf7%4;

        getFirstInetAddress(firstNetworkInteface, true) = /fe80:0:0:0:223:32ff:fe97:5cf7%4
        getFirstInetAddress(firstNetworkInteface, false) = /10.152.23.224

        Additionally, please submit ifconfig -a so an assessment can be made of the network configuration of the machine.

        Show
        Joe Fialli added a comment - More information is needed to investigate why basic lookup of a network address is not working. We need to investigate why this failure occurred in reported stack trace. Caused by: java.io.IOException: can not find a first InetAddress at com.sun.enterprise.mgmt.transport.grizzly.GrizzlyNetworkManager.start(GrizzlyNetworkManager.java:376) A simple NetworkUtility method is failing to find an InternetAddress for the machine. There is a simple test that can be run to diagnose why the network configuration is not working correctly. $ cd <GlassFishInstallation>/glassfish/modules $ java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility AllLocalAddresses() = [/10.152.23.224, /fe80:0:0:0:223:32ff:fe97:5cf7%4, /fe80:0:0:0:0:0:0:1%1] getFirstNetworkInterface() = name:en0 (en0) index: 4 addresses: /10.152.23.224; /fe80:0:0:0:223:32ff:fe97:5cf7%4; getFirstInetAddress( true ) = /fe80:0:0:0:223:32ff:fe97:5cf7%4 getFirstInetAddress( false ) = /10.152.23.224 getFirstNetworkInteface() = name:en0 (en0) index: 4 addresses: /10.152.23.224; /fe80:0:0:0:223:32ff:fe97:5cf7%4; getFirstInetAddress(firstNetworkInteface, true) = /fe80:0:0:0:223:32ff:fe97:5cf7%4 getFirstInetAddress(firstNetworkInteface, false) = /10.152.23.224 Additionally, please submit ifconfig -a so an assessment can be made of the network configuration of the machine.
        Hide
        easarina added a comment -

        Please see bellow the DAS machine information:
        ======================================================
        -bash-3.00$ uname -n
        aixas13
        -bash-3.00$ java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility
        AllLocalAddresses() = [/10.133.169.1]
        getFirstNetworkInterface() = name:lo0 (lo0) index: 1 addresses:
        /0:0:0:0:0:0:0:1;
        /127.0.0.1;

        getFirstInetAddress( true ) = null
        getFirstInetAddress( false ) = null
        getFirstNetworkInteface() = name:lo0 (lo0) index: 1 addresses:
        /0:0:0:0:0:0:0:1;
        /127.0.0.1;

        getFirstInetAddress(firstNetworkInteface, true) = null
        getFirstInetAddress(firstNetworkInteface, false) = null

        -bash-3.00$ ifconfig -a
        en0: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
        inet 10.133.169.1 netmask 0xfffff800 broadcast 10.133.175.255
        tcp_sendspace 131072 tcp_recvspace 65536
        lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT>
        inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255
        inet6 ::1/0
        tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

        Show
        easarina added a comment - Please see bellow the DAS machine information: ====================================================== -bash-3.00$ uname -n aixas13 -bash-3.00$ java -classpath shoal-gms-impl.jar com.sun.enterprise.mgmt.transport.NetworkUtility AllLocalAddresses() = [/10.133.169.1] getFirstNetworkInterface() = name:lo0 (lo0) index: 1 addresses: /0:0:0:0:0:0:0:1; /127.0.0.1; getFirstInetAddress( true ) = null getFirstInetAddress( false ) = null getFirstNetworkInteface() = name:lo0 (lo0) index: 1 addresses: /0:0:0:0:0:0:0:1; /127.0.0.1; getFirstInetAddress(firstNetworkInteface, true) = null getFirstInetAddress(firstNetworkInteface, false) = null -bash-3.00$ ifconfig -a en0: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN> inet 10.133.169.1 netmask 0xfffff800 broadcast 10.133.175.255 tcp_sendspace 131072 tcp_recvspace 65536 lo0: flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT> inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255 inet6 ::1/0 tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1
        Hide
        Joe Fialli added a comment -

        java.net.NetworkInterface.supportsMulticast() is returning FALSE for network interface en0.

        So even though ifconfig is stating that MULTICAST is enabled, something in the network configuration
        is not allowing that method to return true. This is why GMS is not able to find a local address.

        Show
        Joe Fialli added a comment - java.net.NetworkInterface.supportsMulticast() is returning FALSE for network interface en0. So even though ifconfig is stating that MULTICAST is enabled, something in the network configuration is not allowing that method to return true. This is why GMS is not able to find a local address.
        Hide
        scatari added a comment -

        Pre-approved for integration as this is a test blocker.

        Show
        scatari added a comment - Pre-approved for integration as this is a test blocker.
        Hide
        Joe Fialli added a comment -

        Fix is checked into shoal gms trunk. It has been verified to work.
        Fix is still not integrated into GlassFish 3.1.1 yet.

        Use attached shoal-gms-impl.jar as patch till integration complete.

        Show
        Joe Fialli added a comment - Fix is checked into shoal gms trunk. It has been verified to work. Fix is still not integrated into GlassFish 3.1.1 yet. Use attached shoal-gms-impl.jar as patch till integration complete.
        Hide
        Joe Fialli added a comment -

        Install patch into <glassfish-install-dir>/glassfish/modules to workaround this issue on AIX 6.1.

        Show
        Joe Fialli added a comment - Install patch into <glassfish-install-dir>/glassfish/modules to workaround this issue on AIX 6.1.
        Hide
        Bobby Bissett added a comment -

        Integrated into GF:

        Sending packager/resources/pkg_conf.py
        Sending pom.xml
        Transmitting file data ..
        Committed revision 46895.

        I think this will be 3.1.1-b06 (looks like the tag for b05 has already been made).

        Show
        Bobby Bissett added a comment - Integrated into GF: Sending packager/resources/pkg_conf.py Sending pom.xml Transmitting file data .. Committed revision 46895. I think this will be 3.1.1-b06 (looks like the tag for b05 has already been made).

          People

          • Assignee:
            Joe Fialli
            Reporter:
            easarina
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: