(This problem was already discussed with Jakub Podlesak of the Jersey team. He has a GF3Fail.war which proofs the failure).
When a client does not care of the returned document type, it does not send an Accept: header. So the server is free to decide for a type that it thinks fits best. In case there is NO @Produces annotation found in a resource, and the application containing that resource has registered its own context provider HAVING a @Produces annotation, it is obvious that THIS custom provider shall be used to marshal the returned entity AND to define the use Content-Type: header when responding.
Actually this is not working in GFv3.1.1_b11 as the contained Jersey engine does NOT pick that custom context provider, but instead ALWAYS asks its own JSON context provider to marshal the result. This is working when the declared return type of the resource method is the custom JAXB class (actually it would be more smart to FIRST give the application-defined provider a chance to marshal the response before picking a Jersey-addon, but this is a different story).
But it is NOT WORKING (JSON context says it does not know that entity class) in case the declared return type is javax.ws.rs.core.Response and the actual entity class is to be retrieved from the dynamically provided entity given to Response.ok(CustomClass)! In that case the JSON provider is picked, which says it does not know that class.
As long as the JSON provider is picked by Jersey, it MUST add the dynamically the class found in the actual dynamic response.
A far better solution obviously would be to FIRST ask the application-provided providers to do the marshalling as this obviously is what the application programmer tries to achieve (as he doesn't know of the JSON provider at all).
There are two possible workarounds, which both render unuseable in case neither the code of the client not the code of the WAR is available to the GlassFish v3 administrator. As a result, this is a showstopper for non-programmers and such must get fixed ASAP.
(A) Change the client to always provide Accept: header – Only possible if the GlassFish administrator has control over all users, what is ridiculous in case of public anonymous services. Also this does not reflect the typical web scenario, where a client can safely assume to get back anything as long as the server can send anything (which this server could if not having this bug).
(B) Add @Provides(contentType) to the resource method – Only possible if the GlassFish administrator has control over the WAR's source code, what is ridiculous in case of running third party off-the-shelf applications provides by large ISVs ("We tested with application server XYZ and will not change our WAR just to make it run in your buggy server!").