glassfish
  1. glassfish
  2. GLASSFISH-16721

MS1042: failed to send heartbeatmessage with state=aliveandready to group

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 3.1
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      ubuntu, glassfish 3.1, apache 2.2.1, mod_jk 1.2.26

      Description

      I have an Apache and glassfish with mod_jk.
      Many times, I have an error http 500 when i request my apache.

      On my log, i have always this line
      [#|2011-05-23T19:54:19.749+0200|WARNING|glassfish3.1|ShoalLogger|_ThreadID=11;_ThreadName=Thread-1;|GMS1042: failed to send heartbeatmessage with state=aliveandready to group autre-voyage. Reason: IOException:Operation not permitted|#]

      For info, start log:

      19-May-2011 23:10:42 com.sun.enterprise.admin.launcher.GFLauncherLogger info
      INFO: JVM invocation command line:
      /usr/lib/jvm/java-6-openjdk/bin/java
      -cp
      /opt/glassfish3/glassfish/modules/glassfish.jar
      -XX:+UnlockDiagnosticVMOptions
      -XX:MaxPermSize=192m
      -XX:+AggressiveOpts
      -XX:NewRatio=2
      -XX:+UseParallelGC
      -Xmx512m
      -Xmn192m
      -javaagent:/opt/glassfish3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true
      -server
      -Dosgi.shell.telnet.maxconn=1
      -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver
      -Dfelix.fileinstall.disableConfigSave=false
      -Dfelix.fileinstall.dir=/opt/glassfish3/glassfish/modules/autostart/
      -Djavax.net.ssl.keyStore=/opt/glassfish3/glassfish/domains/domain1/config/keystore.jks
      -Dosgi.shell.telnet.port=6666
      -Djava.security.policy=/opt/glassfish3/glassfish/domains/domain1/config/server.policy
      -Dfelix.fileinstall.log.level=2
      -Dfelix.fileinstall.poll=5000
      -Dcom.sun.aas.instanceRoot=/opt/glassfish3/glassfish/domains/domain1
      -Dosgi.shell.telnet.ip=127.0.0.1
      -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory
      -Djava.endorsed.dirs=/opt/glassfish3/glassfish/modules/endorsed:/opt/glassfish3/glassfish/lib/endorsed
      -Dcom.sun.aas.installRoot=/opt/glassfish3/glassfish
      -Djava.ext.dirs=/usr/lib/jvm/java-6-openjdk/lib/ext:/usr/lib/jvm/java-6-openjdk/jre/lib/ext:/opt/glassfish3/glassfish/domains/domain1/lib/ext
      -Dfelix.fileinstall.bundles.startTransient=true
      -Dfelix.fileinstall.bundles.new.start=true
      -Djavax.net.ssl.trustStore=/opt/glassfish3/glassfish/domains/domain1/config/cacerts.jks
      -Dorg.glassfish.additionalOSGiBundlesToStart=org.apache.felix.shell,org.apache.felix.gogo.runtime,org.apache.felix.gogo.shell,org.apache.felix.gogo.command
      -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as
      -DANTLR_USE_DIRECT_CLASS_LOADING=true
      -Djava.security.auth.login.config=/opt/glassfish3/glassfish/domains/domain1/config/login.conf
      Dgosh.args=-nointeractive
      -Djava.library.path=/opt/glassfish3/glassfish/lib:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
      com.sun.enterprise.glassfish.bootstrap.ASMain
      -domainname
      domain1
      -asadmin-args
      -host,,,localhost,,,port,,,4848,,,secure=false,,,terse=false,,,echo=false,,,interactive=true,,,start-domain,,,verbose=false,,,debug=false,,,-domaindir,,,/opt/glassfish3/glassfish/domains,,,domain1
      -instancename
      server
      -verbose
      false
      -debug
      false
      -asadmin-classpath
      /opt/glassfish3/glassfish/modules/admin-cli.jar
      -asadmin-classname
      com.sun.enterprise.admin.cli.AsadminMain
      -upgrade
      false
      -type
      DAS
      -domaindir
      /opt/glassfish3/glassfish/domains/domain1
      -read-stdin
      true
      19-May-2011 23:10:42 com.sun.enterprise.admin.launcher.GFLauncherLogger info
      INFO: Successfully launched in 8 msec.
      [#|2011-05-19T23:11:11.590+0200|INFO|glassfish3.1|null|_ThreadID=1;_ThreadName=Thread-1;|Running GlassFish Version: GlassFish Server Open Source Edition 3.1 (build 43)|#]

      [#|2011-05-19T23:11:11.821+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1005: Member server joined group autre-voyage|#]

      [#|2011-05-19T23:11:11.828+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1004: Started GMS for instance server in group autre-voyage|#]

      [#|2011-05-19T23:11:11.882+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1005: Member server joined group evasion|#]

      [#|2011-05-19T23:11:11.883+0200|INFO|glassfish3.1|javax.org.glassfish.gms.org.glassfish.gms|_ThreadID=1;_ThreadName=Thread-1;|GMSAD1004: Started GMS for instance server in group evasion|#]

      [#|2011-05-19T23:11:12.117+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=59;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 76ms - bound to [0.0.0.0:4848]|#]

      [#|2011-05-19T23:11:12.117+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=56;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 87ms - bound to [0.0.0.0:8181]|#]

      [#|2011-05-19T23:11:12.112+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=57;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 32ms - bound to [0.0.0.0:7676]|#]

      [#|2011-05-19T23:11:12.112+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=58;_ThreadName=Thread-1;|Grizzly Framework 1.9.31 started in: 49ms - bound to [0.0.0.0:3700]|#]

      [#|2011-05-19T23:11:12.140+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.admin.adapter|_ThreadID=1;_ThreadName=Thread-1;|The Admin Console is already installed, but not yet loaded.|#]

      [#|2011-05-19T23:11:12.556+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0170: Apache mod_jk/jk2 attached to virtual-server [server] listening on port [8,080]|#]

      [#|2011-05-19T23:11:12.560+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:8181]|#]

      [#|2011-05-19T23:11:12.567+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:4848]|#]

      [#|2011-05-19T23:11:12.596+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0171: Created virtual server [server]|#]

      [#|2011-05-19T23:11:12.599+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0171: Created virtual server [__asadmin]|#]

      [#|2011-05-19T23:11:13.015+0200|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0172: Virtual server [server] loaded default web module []|#]

      [#|2011-05-19T23:11:13.117+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.120+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.120+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

      [#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

      [#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

      [#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.121+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

      [#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

      [#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.122+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

      [#|2011-05-19T23:11:13.123+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

      [#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker worker|#]

      [#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

      [#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

      [#|2011-05-19T23:11:13.124+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAdmin worker|#]

      [#|2011-05-19T23:11:13.125+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfEvasion worker|#]

      [#|2011-05-19T23:11:13.138+0200|WARNING|glassfish3.1|org.apache.tomcat.util.threads.ThreadPool|_ThreadID=10;_ThreadName=Thread-1;|threadpool.max_threads_too_low|#]

      [#|2011-05-19T23:11:14.206+0200|INFO|glassfish3.1|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=1;_ThreadName=Thread-1;|core.start_container_done|#]

      and my mod_jk config:
      glassfish-jk.properties file -->

      1. Define 1 real worker using ajp13
        worker.list=gfEvasion, gfAdmin, gfAutreVoyage
      1. Set properties for worker1 (ajp13)
        worker.gfAdmin.type=ajp13
        worker.gfAdmin.host=localhost
        worker.gfAdmin.port=8080
        worker.gfAdmin.lbfactor=25
        worker.gfAdmin.cachesize=10
        #worker.gfAdmin.cache_timeout=600
        #worker.gfAdmin.socket_keepalive=1
        #worker.gfAdmin.socket_timeout=300
      1. Set properties for worker1 (ajp13)
        worker.gfEvasion.type=ajp13
        worker.gfEvasion.host=localhost
        worker.gfEvasion.port=28081
        worker.gfEvasion.lbfactor=25
        worker.gfEvasion.cachesize=10
        #worker.gfEvasion.cache_timeout=600
        #worker.gfEvasion.socket_keepalive=1
        #worker.gfEvasion.socket_timeout=300

      worker.gfAutreVoyage.type=ajp13
      worker.gfAutreVoyage.host=localhost
      worker.gfAutreVoyage.port=28080
      worker.gfAutreVoyage.lbfactor=25
      worker.gfAutreVoyage.cachesize=10
      worker.gfAutreVoyage.cache_timeout=600
      worker.gfAutreVoyage.socket_keepalive=1
      worker.gfAutreVoyage.socket_timeout=300
      worker.gfAutreVoyage.connection_pool_timeout=600

      My virtual host config: JkMount /* gfAutreVoyage

      and my httpd.conf

      JkWorkersFile /opt/glassfish3/glassfish/domains/domain1/config/glassfish-jk.properties

      1. Where to put jk logs
        JkLogFile /var/log/apache2/mod_jk.log
      2. Set the jk log level [debug/error/info]
        JkLogLevel debug
      3. Select the log format
        JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
      4. JkOptions indicate to send SSL KEY SIZE,
        JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
      5. JkRequestLogFormat set the request format
        JkRequestLogFormat "%w %V %T"

        Activity

        Hide
        Nazrul added a comment -

        User is using openJDK. Perhaps it would be a good idea to try HotSpot (Sun) JDK from here: http://www.oracle.com/technetwork/java/javase/downloads/index.html

        Show
        Nazrul added a comment - User is using openJDK. Perhaps it would be a good idea to try HotSpot (Sun) JDK from here: http://www.oracle.com/technetwork/java/javase/downloads/index.html
        Hide
        Joe Fialli added a comment -

        These comments only pertain to following reported log message and the subject of this issue.

        [#|2011-05-23T19:54:19.749+0200|WARNING|glassfish3.1|ShoalLogger|_ThreadID=11;_ThreadName=Thread-1;|GMS1042: failed to send heartbeatmessage with state=aliveandready to group autre-voyage. Reason: IOException:Operation not permitted|#]

        Shoal GMS requires UDP multicast to be configured properly to be used between glassfish clustered instances.
        The above failure indicates that multicast may not be configured to be allowed at the OS networking layer.

        The following link provides guidance on verifying UDP multicast is configured properly.
        http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d

        The reported failure is a network or security configuration issue that is not allowing the UDP multicast operation
        to succeed so this issue will be marked invalid.

        *******

        The following error message is not a GMS issue.

        [#|2011-05-19T23:11:13.117+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#]

        It should be filed as different issue with a different subcomponent.

        Show
        Joe Fialli added a comment - These comments only pertain to following reported log message and the subject of this issue. [#|2011-05-23T19:54:19.749+0200|WARNING|glassfish3.1|ShoalLogger|_ThreadID=11;_ThreadName=Thread-1;|GMS1042: failed to send heartbeatmessage with state=aliveandready to group autre-voyage. Reason: IOException:Operation not permitted|#] Shoal GMS requires UDP multicast to be configured properly to be used between glassfish clustered instances. The above failure indicates that multicast may not be configured to be allowed at the OS networking layer. The following link provides guidance on verifying UDP multicast is configured properly. http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d The reported failure is a network or security configuration issue that is not allowing the UDP multicast operation to succeed so this issue will be marked invalid. ******* The following error message is not a GMS issue. [#|2011-05-19T23:11:13.117+0200|SEVERE|glassfish3.1|org.apache.jk.server.JkMain|_ThreadID=10;_ThreadName=Thread-1;|No class name for worker.gfAutreVoyage worker|#] It should be filed as different issue with a different subcomponent.
        Hide
        Joe Fialli added a comment -

        Marking this issue as incomplete since information was not provided that illustrated that
        UDP multicast was verified to be working properly. The reported failure "Operation invalid"
        on sending a UDP multicast message indicates that UDP multicast is not enabled.

        Following the directions on this
        link: http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d
        should resolve the network configuration issue.

        If it does not, please submit "ifconfig -a" output (or equivalent if OS does not support that command),
        the DAS server.log with ShoalLogger log level of CONFIG. (provides what network addresses and ports are being used)

        % asadmin set-log-levels ShoalLogger=CONFIG

        Run the above before creating cluster.

        The ifconfig output should have "MULTICAST" on a line similar to following:
        en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

        Show
        Joe Fialli added a comment - Marking this issue as incomplete since information was not provided that illustrated that UDP multicast was verified to be working properly. The reported failure "Operation invalid" on sending a UDP multicast message indicates that UDP multicast is not enabled. Following the directions on this link: http://blogs.oracle.com/bobby/entry/validating_multicast_transport_where_d should resolve the network configuration issue. If it does not, please submit "ifconfig -a" output (or equivalent if OS does not support that command), the DAS server.log with ShoalLogger log level of CONFIG. (provides what network addresses and ports are being used) % asadmin set-log-levels ShoalLogger=CONFIG Run the above before creating cluster. The ifconfig output should have "MULTICAST" on a line similar to following: en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

          People

          • Assignee:
            Joe Fialli
            Reporter:
            sebglon
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: