[mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs

  • From: Amy Kang <amy.kang@...>
  • To: Jordan Redner <jredner@...>
  • Cc: users@...
  • Subject: [mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs
  • Date: Thu, 06 Jun 2013 18:11:44 -0700

Hi Jordan,


On 06/06/2013 03:39 PM, Jordan Redner wrote:
We're running openmq  4.4 build 5-a on OREL (Linux xxxxxxxxxxxxxx 
2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 
GNU/Linux)

We've seen a number of times the broker freeze up (no longer accepts sockets, 
sends or receives messages).

What error did you get when a client attempts to connect to the broker ?

   When this happens, there is nothing exceptional logged, and the broker 
stop logging completely.  We have to forcibly kill the process and start the 
broker to restore the node.


Is this a known issue, or do you have any recommended flags to throw for more 
diagnostics?


No. Is it local or remote file system for the broker's varhome where the broker log file resides ? Was there any network or file system access problem when it occurs ? Could you get a thread dump of the broker when it occurs ? Can imqcmd still be able to connect to the broker when the problem happens ? If yes, what does 'imqcmd list dst', 'imqcmd query bkr" show ?


Note the gap in time in the logs below right before the restart (15:06):

[03/Jun/2013:14:33:28 PDT]
Connections: 9720    JVM Heap: 2226192384 bytes (283809680 free) Threads: 
19446 (14-20010)
       In: 191267287 msgs (1762362337932 bytes)  480441096 pkts 
(1792003191173 bytes)
      Out: 191275937 msgs (1763196194467 bytes)  382718266 pkts 
(1781830057866 bytes)
  Rate In: 212 msgs/sec (1942958 bytes/sec)  343 pkts/sec (1953789 bytes/sec)
Rate Out: 21 msgs/sec (182798 bytes/sec)  43 pkts/sec (185016 bytes/sec)
[03/Jun/2013:14:35:30 PDT] [B1140]: Expired 1 messages from destination 
WD40nk11p00itucgovernor102 [Topic]
[03/Jun/2013:14:35:56 PDT] [B1140]: Expired 1 messages from destination 
WD40nk11p00itucgovernor110 [Topic]
[03/Jun/2013:14:35:56 PDT] [B1140]: Expired 1 messages from destination 
WD40nk11p00itucgovernor105 [Topic]
[03/Jun/2013:14:38:28 PDT]
Connections: 9720    JVM Heap: 2786066432 bytes (70134552 free) Threads: 
19444 (14-20010)
       In: 191331689 msgs (1762953277691 bytes)  480539258 pkts 
(1792596726080 bytes)
      Out: 191276201 msgs (1763196329746 bytes)  382719537 pkts 
(1781830322428 bytes)
  Rate In: 214 msgs/sec (1969786 bytes/sec)  327 pkts/sec (1978436 bytes/sec)
Rate Out: 0 msgs/sec (450 bytes/sec)  4 pkts/sec (881 bytes/sec)
[03/Jun/2013:15:06:07 PDT] ERROR [B3142]: Unable to use JMQ_ETC_HOME 
./mq/bin/./mq/bin/../../etc/mq
[03/Jun/2013:15:06:07 PDT] WARNING [S2004]: Log output channel syslog is 
disabled: no imqutil in java.library.path
[03/Jun/2013:15:06:07 PDT]
================================================================================
Open Message Queue 4.4
Oracle
Version:  4.4 Update 2  (Build 5-a)
Compile:  Fri May 14 23:24:45 PDT 2010

Copyright (c) 2010, Oracle and/or its affiliates.  All rights reserved.
================================================================================
Java Runtime: 1.6.0_38 Sun Microsystems Inc. /usr/java/jdk1.6.0_38/jre
[03/Jun/2013:15:06:07 PDT]    
IMQ_HOME=/opt/ais/local/Infrastructure/OpenMQ/openmq-4.4u2-1/mq/bin/mq
[03/Jun/2013:15:06:07 PDT] 
IMQ_VARHOME=/opt/ais/local/Infrastructure/OpenMQ/openmq-4.4u2-1/var/mq
[03/Jun/2013:15:06:07 PDT] Linux 2.6.18-194.el5 amd64 nk11p00it-carnival002 
(16 cpu) worun
[03/Jun/2013:15:06:07 PDT] Java Heap Size: max=3728320k, current=188416k


amy


Thanks,


Jordan




[mq-users] 4.4 Update 2 (Build 5-a) broker hangs

Jordan Redner 06/06/2013

[mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs

Ed Bratt 06/07/2013

[mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs

Jordan Redner 06/07/2013

[mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs

Amy Kang 06/07/2013

[mq-users] Re: 4.4 Update 2 (Build 5-a) broker hangs

Jordan Redner 06/07/2013
Terms of Use; Privacy Policy; Copyright ©2013-2015 (revision 20150226.965aeb8)
 
 
Close
loading
Please Confirm
Close