XAResource.setTransactionTimeout(encompassingJTATransactionTImeoutValue) is not being called on resources enlisted in a Glassfish transaction.
Javadoc for the XAResource.setTransactionTimeout: "Once set, this timeout value is effective until setTransactionTimeout is invoked again with a different value." Which means that if an XAResource has a predefined timeout less than the timeout set on the UserTransaction, the transaction manager has no control (and should not have any according to the text above) over the XAResource behavior.
The javadoc does not state this as a requirement. Note that the javadoc does say "Sets the current transaction timeout value for this |XAResource| instance. " suggesting, as is the case, that each XAResource in question is an instance related to a transaction.
However, in all other application servers, XAResource.setTransactionTimeout(encompassingJTATransactionTImeoutValue) is the default behavior (in WebLogic there is an override option but even that is not suggested best practice) and it makes logical sense as the XAResource is always a part of a JTA transaction and thus the timeout values are directly related.
Glassfish should make Glassfish xa-resource-timeout dynamic in future releases.