I can't reproduce the described issue. The expected RuntimeException appeared. There is no infinite loop.
But I found other issue in my testing. When sending a message from the standalone java client to queue, the MDB behaves as expected and retries 5 times, we can see 6 RuntimeExceptions. Then the message was moved to DMQ. It looks everything works fine at this point. But when I restart domain and server, the MDB throws 6 RuntimeExceptions again. The message also enters queue and marked as "UnAck". Each time I retart domain, the MDB still throws 6 RuntimeExceptions. It looks to me the behavior is not correct. The message has been moved to the DMQ, why it is delivered again and again when domain restart?