AIX machines, run richAccess stress test with new SDK from IBM with a fix for the bug 16707. Three instances in a cluster, three machines, one instance plus DAS on one machine, then one instance per a machine. Problems that were described in the bug 16707 appeared almost immediately after the test was started. Within 3 days of running I did not see any issues. Then I saw in server.log files of all instances such warnings:
[#|2011-11-07T00:01:37.191-0800|WARNING|glassfish3.1.2|javax.jms|_ThreadID=14;_ThreadName=Thread-9;|[I500]: Caught JVM Exception: java.io.EOFException: Trying to read 72 bytes. Already read 0 bytes.|#]
Later more severe errors appeared in server.log of one instance and for that instance 35 requests failed. The new severe error messages started with that message.
[#|2011-11-07T18:47:28.195-0800|SEVERE|glassfish3.1.2|com.s1as.e2e.richAccess.servlet.sendorder|_ThreadID=15;_ThreadName=Thread-9;|Exception e java.lang.NullPointerException|#]
I've attached a server.log for that instance.