This issue was originally reported by a user on the GlassFish developer list. Here is the thread:
The issue can be summarised as follows: a MDB consumes a message from an inbound queue, updates a database then invokes a session bean which sends a message to an outbound queue. If 40 messages are placed on the inbound queue then we see a variety of messages in the server log (see that thread), including this one:
625+0000|SEVERE|glassfish3.1|javax.resourceadapter.mqjmsra.outbound.connection|_ThreadID=36;_ThreadName=Thread-1;|commitTransaction (XA) on JMSService:jmsdirect failed for connectionId:950872495901869318 and onePhase:false due to Unknown JMSService server error ERROR: com.sun.messaging.jmq.jmsserver.util.BrokerException: Bad transaction state transition. Cannot perform operation COMMIT_TRANSACTION(46) (XAFlag=0x0:TMNOFLAGS) on a transaction in state COMPLETE(4).|#]
The error occurs ONLY if the session bean caches its JMS connection, session and producer in a field of the bean. This is valid, though it is contrary to the conventional practice which is to create the connection, use it, and close the connection every time the session bean is invoked. If these objects are not cached then this bug is NOT seen. This bug therefore has a workaround.
The issue can be reproduced using JMS only (i.e. not using a database), though to see exactly the same errors as the user reported it is necessary to force the use of two-phase commits.
A simple NetBeans application is attached which demonstrates the issue. This consists of a Enterprise Application "TransactionTests" which is composed of a ejb application "TransactionTests-ejb" and a web application "TransactionTests-war".
Steps to reproduce:
1. Install the latest version of GlassFish 3.1 (I used build 37)
2. Before starting GlassFish, edit domain.xml to set the JVM option -Dimq.jmsra.isSameRMAllowed=false . This is needed to force two-phase transactions to be used. (If this is not done the application will still fail but you will get different errors).
3. Use NetBeans to build the application (which is an ear cotaining an ejb and a web app) and deploy it in GlassFish.
4. Visit http://localhost:8080/TransactionTests-war/ and click on "Run MDB Test 1". This causes a servlet to send 40 messages to the inbound queue.
5. Inspect the server log for errors