[mq-users] Re: High memory usage on brokers during port scan.

  • From: Amy Kang <amy.kang@...>
  • To: Sandeep Pournami <sandeep.p@...>
  • Cc: users@...
  • Subject: [mq-users] Re: High memory usage on brokers during port scan.
  • Date: Mon, 03 Mar 2014 22:11:35 -0800

Hi Sandeep,

The stacktrace you showed below is on 'jms' service with 'shared' thread pool. The StreamCorruptedException was expected since your scanner was sending garbage data to broker. However the 'shared' thread pool does not appear to handle the StreamCorruptedException properly which likely caused the thread not released. Please file a jira issue attaching complete broker log including low memory warning log message or OOM exception and exceptions that leading to it. Please note that the 'shared' thread pool in < 5.0 (the one your broker uses now) has been replaced by new implementation in >= 5.0.

>Is there any work around possible to prevent this from happening and causing production impact ?

There are 2 possible options - 'dedicated' thread pool or broker SSL client authentication - unfortunately the former requires MQ-349 fix and broker SSL client authentication is only available >= 5.0.

Thanks,
amy

On 03/03/2014 05:01 PM, Sandeep Pournami wrote:
We have another issue on our brokers when our internal vulnerability detector scanner runs on the openmq ports.

We are seeing that the broker is going to a low memory state during this time.. sometimes it goes out of memory as well. It recovers automatically when the scan is stopped.

Not sure why these bad packets are causing memory to spike like this. Shouldn’t it be just dropping these connections once it sees the bad packet numbers?
All the broker ports are being scanned from multiple threads during this time.
Is there any work around possible to prevent this from happening and causing production impact ?

XXXXXXXXX  - this is the scanner tool ip.

[03/Mar/2014:14:50:29 PST] WARNING [B2070]: Data on connection ???@XXXXXXXXX:42090 is corrupted, closing connection:
java.io.StreamCorruptedException: Bad packet magic number: 369295617. Expecting: 469754818
at com.sun.messaging.jmq.io.Packet.parseFixedBuffer(Packet.java:362)
        at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1178)
at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readInPacket(IMQIPConnection.java:1166)
at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1233)
at com.sun.messaging.jmq.jmsserver.service.imq.group.ReadThread.process(ReadThread.java:113)
at com.sun.messaging.jmq.jmsserver.service.imq.group.SelectThread.processThread(SelectThread.java:433)
at com.sun.messaging.jmq.jmsserver.service.imq.group.GroupRunnable.process(GroupRunnable.java:150)
at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
        at java.lang.Thread.run(Thread.java:662)
[03/Mar/2014:14:50:29 PST] WARNING Last good packet received from connection ???@XXXXXXXXX:42090: type = 0, size = 0
[03/Mar/2014:14:50:29 PST] [B1066]: Closing: ???@XXXXXXXXXX:0->jms:0 because "java.io.StreamCorruptedException: Bad packet magic number: 369295617. Expecting: 469754818". Count: service=0 broker=78
[03/Mar/2014:14:50:29 PST] WARNING [B2070]: Data on connection ???@XXXXXXXXXXX:42091 is corrupted, closing connection:
java.io.StreamCorruptedException: Bad packet magic number: 369295360. Expecting: 469754818
at com.sun.messaging.jmq.io.Packet.parseFixedBuffer(Packet.java:362)
        at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1178)
at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readInPacket(IMQIPConnection.java:1166)
at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1233)
at com.sun.messaging.jmq.jmsserver.service.imq.group.ReadThread.process(ReadThread.java:113)
at com.sun.messaging.jmq.jmsserver.service.imq.group.SelectThread.processThread(SelectThread.java:433)
at com.sun.messaging.jmq.jmsserver.service.imq.group.GroupRunnable.process(GroupRunnable.java:150)
at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
        at java.lang.Thread.run(Thread.java:662)
[03/Mar/2014:14:50:29 PST] WARNING Last good packet received from connection ???@XXXXXXXXXXX:42091: type = 0, size = 0
[03/Mar/2014:14:50:29 PST] [B1066]: Closing: ???@XXXXXXXXXX:0->jms:0 because "java.io.StreamCorruptedException: Bad packet magic number: 369295360. Expecting: 469754818". Count: service=0 broker=78
[03/Mar/2014:14:50:30 PST] [B1089]: In low memory condition, Broker is attempting to free up resources
[03/Mar/2014:14:50:30 PST] [B1088]: Entering Memory State RED from previous state ORANGE - allocated memory is 7879889K, 98% of total memory used






[mq-users] High memory usage on brokers during port scan.

Sandeep Pournami 03/04/2014

[mq-users] Re: High memory usage on brokers during port scan.

Amy Kang 03/04/2014

[mq-users] Re: High memory usage on brokers during port scan.

Sandeep Pournami 03/07/2014

[mq-users] Re: High memory usage on brokers during port scan.

Amy Kang 03/07/2014
Terms of Use; Privacy Policy; Copyright ©2013-2015 (revision 20150226.965aeb8)
 
 
Close
loading
Please Confirm
Close