In Glassfish 4.0, one of the causes of the 40% regression we have in remote EJB performance comes from JDKBridge.loadClass – and in particular, when that calls the process classNotFoundInJDKClassLoader. That class in turn calls org.glassfish.pfl.basic.proxy.CompositeInvocationHandlerImpl.invoke -> org.glassfish.pfl.basic.logex.WrapperGenerator$2.invoke -> org.glassfish.pfl.basic.logex.WrapperGenerator.handleFullLogging -> org.glassfish.pfl.basic.logex.WrapperGenerator.trimStackTrace (which is enormously expensive because it walks the stack and interns dozens of strings along the way).
This all occurs when creating the stub while reading the cdr stream (CDRInputStream.read_Object -> StubFactoryFactoryDybamicBase.creaetStubFactory).
I guess that normal classloading semantics are in place here – it is looking first in the JDK classloader, and then will look in the app classloader. But while looking up the class in the JDK classloader will throw a ClassNotFoundException, we shouldn't be spending any time processing that exception for logging: the exception is never printed to the log (default logging settings are in place). So perhaps it is as simple along the way as making sure not to call into the logging code (including formatting the parameters for a call into the logger) unless the logging is really going to happen. Though in general, it would be better if the classloader could cache lookups like this so we don't have to repeatedly execute the same failed lookups.