[mq-users] Re: High memory usage on brokers during port scan.

  • From: Sandeep Pournami <sandeep.p@...>
  • To: Amy Kang <amy.kang@...>
  • Cc: users@...
  • Subject: [mq-users] Re: High memory usage on brokers during port scan.
  • Date: Fri, 07 Mar 2014 10:13:47 -0800

Amy,

we did some more detailed analysis and it looks like the memory shoots up 
when a particular port is being scanned. It is NOT the JMS,Admin or cluster 
ports. It is some dynamic port. ssladmin or ssljms ? I don’t know how to 
check what ports they are listening for sure.
Is there a way we can disable ssl services?

Sandeep.

On Mar 3, 2014, at 10:11 PM, Amy Kang <amy.kang@...> wrote:

> Hi Sandeep,
> 
> The stacktrace you showed below is on 'jms' service with 'shared' thread 
> pool.   The StreamCorruptedException was expected since your scanner was 
> sending garbage data to broker.   However the  'shared' thread pool does 
> not appear to handle the StreamCorruptedException properly which likely 
> caused the thread not released.  Please file a jira issue attaching 
> complete broker log including low memory warning log message or OOM 
> exception and exceptions that leading to it.   Please note that the 
> 'shared' thread pool in < 5.0 (the one your broker uses now) has been 
> replaced by new implementation in >= 5.0. 
> 
> >Is there any work around possible to prevent this from happening and 
> >causing production impact ?
> 
> There are 2 possible options - 'dedicated' thread pool or broker SSL client 
> authentication - unfortunately the former requires MQ-349 fix and broker 
> SSL client authentication is only available >= 5.0.
> 
> Thanks,
> amy
> 
> On 03/03/2014 05:01 PM, Sandeep Pournami wrote:
>> We have another issue on our brokers when our internal vulnerability 
>> detector scanner runs on the openmq ports.
>
>> We are seeing that the broker is going to a low memory state during this 
>> time.. sometimes it goes out of memory as well. It recovers automatically 
>> when the scan is stopped.
>
>> Not sure why these bad packets are causing memory to spike like this. 
>> Shouldn’t it be just dropping these connections once it sees the bad 
>> packet numbers?
>> All the broker ports are being scanned from multiple threads during this 
>> time.
>> Is there any work around possible to prevent this from happening and 
>> causing production impact ?
>
>> XXXXXXXXX  - this is the scanner tool ip.
>
>> [03/Mar/2014:14:50:29 PST] WARNING [B2070]: Data on connection 
>> ???@XXXXXXXXX:42090 is corrupted, closing connection:
>> java.io.StreamCorruptedException: Bad packet magic number: 369295617. 
>> Expecting: 469754818
>>         at 
>> com.sun.messaging.jmq.io.Packet.parseFixedBuffer(Packet.java:362)
>>         at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1178)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readInPacket(IMQIPConnection.java:1166)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1233)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.ReadThread.process(ReadThread.java:113)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.SelectThread.processThread(SelectThread.java:433)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.GroupRunnable.process(GroupRunnable.java:150)
>>         at 
>> com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
>>         at java.lang.Thread.run(Thread.java:662)
>> [03/Mar/2014:14:50:29 PST] WARNING Last good packet received from 
>> connection ???@XXXXXXXXX:42090: type = 0, size = 0
>> [03/Mar/2014:14:50:29 PST] [B1066]:   Closing: ???@XXXXXXXXXX:0->jms:0 
>> because "java.io.StreamCorruptedException: Bad packet magic number: 
>> 369295617. Expecting: 469754818". Count: service=0 broker=78
>> [03/Mar/2014:14:50:29 PST] WARNING [B2070]: Data on connection 
>> ???@XXXXXXXXXXX:42091 is corrupted, closing connection:
>> java.io.StreamCorruptedException: Bad packet magic number: 369295360. 
>> Expecting: 469754818
>>         at 
>> com.sun.messaging.jmq.io.Packet.parseFixedBuffer(Packet.java:362)
>>         at com.sun.messaging.jmq.io.Packet.readPacket(Packet.java:1178)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readInPacket(IMQIPConnection.java:1166)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1233)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.ReadThread.process(ReadThread.java:113)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.SelectThread.processThread(SelectThread.java:433)
>>         at 
>> com.sun.messaging.jmq.jmsserver.service.imq.group.GroupRunnable.process(GroupRunnable.java:150)
>>         at 
>> com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
>>         at java.lang.Thread.run(Thread.java:662)
>> [03/Mar/2014:14:50:29 PST] WARNING Last good packet received from 
>> connection ???@XXXXXXXXXXX:42091: type = 0, size = 0
>> [03/Mar/2014:14:50:29 PST] [B1066]:   Closing: ???@XXXXXXXXXX:0->jms:0 
>> because "java.io.StreamCorruptedException: Bad packet magic number: 
>> 369295360. Expecting: 469754818". Count: service=0 broker=78
>> [03/Mar/2014:14:50:30 PST] [B1089]: In low memory condition, Broker is 
>> attempting to free up resources
>> [03/Mar/2014:14:50:30 PST] [B1088]: Entering Memory State  RED  from 
>> previous state ORANGE - allocated memory is 7879889K, 98% of total memory 
>> used
>
>
>
> 



[mq-users] High memory usage on brokers during port scan.

Sandeep Pournami 03/04/2014

[mq-users] Re: High memory usage on brokers during port scan.

Amy Kang 03/04/2014

[mq-users] Re: High memory usage on brokers during port scan.

Sandeep Pournami 03/07/2014

[mq-users] Re: High memory usage on brokers during port scan.

Amy Kang 03/07/2014
Terms of Use; Privacy Policy; Copyright ©2013-2016 (revision 20151030.c1dd42a)
 
 
Close
loading
Please Confirm
Close