glassfish
  1. glassfish
  2. GLASSFISH-16803

Start cluster throw exceptions on IEC HA setup machines

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 3.1.1
    • Fix Version/s: None
    • Component/s: jms
    • Labels:
      None
    • Environment:

      Linux

      Description

      OS: We have 5 linux machines at IEC, each has 2 GF instances, total is 10 instances.
      The mysql server is installed at a windows machine at santa clara
      build: v311 build 06
      Steps to reproduce the bug:
      1. install GF on the system and create a domain st-comain, start it.
      /export/sonia/hatest/v3/glassfishv3/bin/asadmin --user admin create-domain
      --adminport 4848 --nopassword st-domain
      /export/sonia/hatest/v3/glassfishv3/bin/asadmin --user admin start-domain st-domain
      2. create nodes for each machine (5 machines)
      3. create 2 instances for each machine
      4. set the JMS service type to LOCAL for the cluster
      5. cp mysql-connector-java-5.1.13-bin.jar to
      /export/sonia/hatest/v3/glassfishv3/glassfish/lib dir
      6. Modify glassfishv3/mq/etc/imqenv.config file to make the following variable to point to the mysql driver location.
      IMQ_DEFAULT_EXT_JARS
      7. run asadmin configure-jms-cluster command to setup JMS HA cluster
      8. When I start st-cluster using asadmin start-cluster command, it said successful, however, I saw the following exception in the server.log:

      [#|2011-06-04T00:00:27.800+0530|SEVERE|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=10;_ThreadName=Thread-1;|com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Failed to start resource adapter : java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
      at com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundResourceAdapter.java:134)
      at com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(ActiveInboundResourceAdapterImpl.java:90)
      at com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(ActiveRAFactory.java:135)
      at com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:106)
      at com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:212)
      at com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:348)
      at com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:356)
      at com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(JmsProviderLifecycle.java:113)
      at com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsProviderLifecycle.java:93)
      at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
      at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
      at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:76)
      at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)
      at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
      at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
      Caused by: javax.resource.spi.ResourceAdapterInternalException: java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
      at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceAdapter(ActiveJmsResourceAdapter.java:369)
      at com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundResourceAdapter.java:129)
      ... 24 more
      Caused by: java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
      at java.security.AccessController.doPrivileged(Native Method)
      at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceAdapter(ActiveJmsResourceAdapter.java:353)
      ... 25 more
      Caused by: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
      at com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.java:433)
      at com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:383)
      at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJmsResourceAdapter.java:360)
      ... 27 more
      Caused by: java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
      at com.sun.messaging.jms.blc.LocalBrokerRunner.start(LocalBrokerRunner.java:366)
      at com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.java:431)
      ... 29 more

      #]
      ****************************************
      9. Though the exception showed that server can not ping broker, but I monitored that the broker processes were started after sometime, however, when I tried to stop cluster using command "asadmin stop-cluster st-cluster", it only stopped the GF instances but not the Brokers (It should stop the GF instances and brokers since the JMS service type is LOCAL). I did not see this behavior on the similar HA setup at Santa Clara campus. I only see the start cluster failure for this IEC HA setup. I am not sure if it because that the mysql server is at Santa Clara and the GF installations are at IEC and the network delayed the connection and caused GF threw the exception during starting up. Due to the start cluster problem, we could not run the MQ HA test cases successfully on the IEC HA setup. Could you please help us to investigate the issue. Thanks.
      – full server.log is attached.

        Activity

        Hide
        Satish Kumar added a comment -

        The exception is reported as a consequence of the delay in the broker start-up. Since MySQL server is in a remote location the response times for the broker are highly delayed leading to this exception. Pls configure a local MySQL setup.

        Show
        Satish Kumar added a comment - The exception is reported as a consequence of the delay in the broker start-up. Since MySQL server is in a remote location the response times for the broker are highly delayed leading to this exception. Pls configure a local MySQL setup.

          People

          • Assignee:
            Satish Kumar
            Reporter:
            sonialiu
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: