[GLASSFISH-16803] Start cluster throw exceptions on IEC HA setup machines Created: 05/Jun/11  Updated: 22/Jun/11  Resolved: 22/Jun/11

Status: Closed
Project: glassfish
Component/s: jms
Affects Version/s: 3.1.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: sonialiu Assignee: Satish Kumar
Resolution: Invalid Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux


Attachments: Text File server.log    
Tags: 3_1_1-review, 3_1_1-scrubbed

 Description   

OS: We have 5 linux machines at IEC, each has 2 GF instances, total is 10 instances.
The mysql server is installed at a windows machine at santa clara
build: v311 build 06
Steps to reproduce the bug:
1. install GF on the system and create a domain st-comain, start it.
/export/sonia/hatest/v3/glassfishv3/bin/asadmin --user admin create-domain
--adminport 4848 --nopassword st-domain
/export/sonia/hatest/v3/glassfishv3/bin/asadmin --user admin start-domain st-domain
2. create nodes for each machine (5 machines)
3. create 2 instances for each machine
4. set the JMS service type to LOCAL for the cluster
5. cp mysql-connector-java-5.1.13-bin.jar to
/export/sonia/hatest/v3/glassfishv3/glassfish/lib dir
6. Modify glassfishv3/mq/etc/imqenv.config file to make the following variable to point to the mysql driver location.
IMQ_DEFAULT_EXT_JARS
7. run asadmin configure-jms-cluster command to setup JMS HA cluster
8. When I start st-cluster using asadmin start-cluster command, it said successful, however, I saw the following exception in the server.log:

[#|2011-06-04T00:00:27.800+0530|SEVERE|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=10;_ThreadName=Thread-1;|com.sun.appserv.connectors.internal.api.ConnectorRuntimeException: Failed to start resource adapter : java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundResourceAdapter.java:134)
at com.sun.enterprise.connectors.inbound.ActiveInboundResourceAdapterImpl.init(ActiveInboundResourceAdapterImpl.java:90)
at com.sun.enterprise.connectors.ActiveRAFactory.instantiateActiveResourceAdapter(ActiveRAFactory.java:135)
at com.sun.enterprise.connectors.ActiveRAFactory.createActiveResourceAdapter(ActiveRAFactory.java:106)
at com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:212)
at com.sun.enterprise.connectors.service.ResourceAdapterAdminServiceImpl.createActiveResourceAdapter(ResourceAdapterAdminServiceImpl.java:348)
at com.sun.enterprise.connectors.ConnectorRuntime.createActiveResourceAdapter(ConnectorRuntime.java:356)
at com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.initializeBroker(JmsProviderLifecycle.java:113)
at com.sun.enterprise.connectors.jms.system.JmsProviderLifecycle.postConstruct(JmsProviderLifecycle.java:93)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:76)
at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:326)
at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:135)
at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: javax.resource.spi.ResourceAdapterInternalException: java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceAdapter(ActiveJmsResourceAdapter.java:369)
at com.sun.enterprise.connectors.ActiveOutboundResourceAdapter.init(ActiveOutboundResourceAdapter.java:129)
... 24 more
Caused by: java.security.PrivilegedActionException: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter.startResourceAdapter(ActiveJmsResourceAdapter.java:353)
... 25 more
Caused by: javax.resource.spi.ResourceAdapterInternalException: MQJMSRA_RA4001: start:Aborting:Exception starting LOCAL broker=MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.java:433)
at com.sun.messaging.jms.ra.ResourceAdapter.start(ResourceAdapter.java:383)
at com.sun.enterprise.connectors.jms.system.ActiveJmsResourceAdapter$1.run(ActiveJmsResourceAdapter.java:360)
... 27 more
Caused by: java.lang.RuntimeException: MQJMSRA_LB4001: start:Aborted:Unable to ping Broker within 60000 millis (startTimeOut)
at com.sun.messaging.jms.blc.LocalBrokerRunner.start(LocalBrokerRunner.java:366)
at com.sun.messaging.jms.blc.LifecycleManagedBroker.start(LifecycleManagedBroker.java:431)
... 29 more

#]
****************************************
9. Though the exception showed that server can not ping broker, but I monitored that the broker processes were started after sometime, however, when I tried to stop cluster using command "asadmin stop-cluster st-cluster", it only stopped the GF instances but not the Brokers (It should stop the GF instances and brokers since the JMS service type is LOCAL). I did not see this behavior on the similar HA setup at Santa Clara campus. I only see the start cluster failure for this IEC HA setup. I am not sure if it because that the mysql server is at Santa Clara and the GF installations are at IEC and the network delayed the connection and caused GF threw the exception during starting up. Due to the start cluster problem, we could not run the MQ HA test cases successfully on the IEC HA setup. Could you please help us to investigate the issue. Thanks.
– full server.log is attached.


 Comments   
Comment by Satish Kumar [ 22/Jun/11 ]

The exception is reported as a consequence of the delay in the broker start-up. Since MySQL server is in a remote location the response times for the broker are highly delayed leading to this exception. Pls configure a local MySQL setup.

Generated at Fri Mar 06 11:58:46 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.