jersey
  1. jersey
  2. JERSEY-1905

Injecting SecurityContext into a simple @GET method reduces performance from a few thousand requests per second to about 150 max.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.3
    • Component/s: core
    • Labels:
      None

      Description

      This is the method in question.

          @GET
          @Path( "/checkJobStatus" )
          @Produces( "text/plain; charset=UTF-8" )
          public String checkJobStatus( @QueryParam( "JobRequestID" ) final String jobRequestID, @Context final SecurityContext securityContext )
          {
              return "INVALID";
          }
      

      If I remove the

      @Context final SecurityContext securityContext

      parameter, the method performs as expects.

      Here is a thread dump captured while 50 client threads make requests to this path.

      It seems there is contention for a lock inside CGLIB.. It seems strange that CGLIB would be used while injecting something like a SecurityContext..

      "Camel (camelContext) thread #102 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea3c08c800 nid=0x426e waiting for monitor entry [0x00007fe9ad0ce000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fa26f918> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #101 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fe9bc04e000 nid=0x426d waiting on condition [0x00007fe9ad1d0000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e5973268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #100 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x0000000001b35800 nid=0x426c waiting on condition [0x00007fe9ad2d1000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e5973268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #99 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea4d10c800 nid=0x426b waiting for monitor entry [0x00007fe9ad3d1000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fa3579c8> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #98 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea3c08a800 nid=0x426a runnable [0x00007fe9ad4d2000]
         java.lang.Thread.State: RUNNABLE
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:227)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fa742de8> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #97 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fe9bc04d000 nid=0x4269 runnable [0x00007fe9ad5d4000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e5973268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #96 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x0000000001b39000 nid=0x4268 waiting for monitor entry [0x00007fe9ad6d4000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fbb11518> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #95 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea4d10b000 nid=0x4267 waiting on condition [0x00007fe9ad7d6000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e5973268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #94 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea3c08a000 nid=0x4266 waiting on condition [0x00007fe9ad8d7000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e5973268> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #93 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fe9bc04f000 nid=0x4265 waiting for monitor entry [0x00007fe9ad9d7000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fb20bac0> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #92 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x0000000001b38800 nid=0x4264 waiting for monitor entry [0x00007fe9adad8000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fbbf79c0> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #91 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea4c997800 nid=0x4263 waiting for monitor entry [0x00007fe9adbd9000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fa5a9a18> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #90 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fea3c086800 nid=0x4262 waiting for monitor entry [0x00007fe9adcda000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000f9e04be8> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #89 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x00007fe9bc04b000 nid=0x4261 waiting for monitor entry [0x00007fe9adddb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fba4fe98> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      "Camel (camelContext) thread #88 - wsgw://0.0.0.0:8282" daemon prio=10 tid=0x0000000001b2d000 nid=0x4260 waiting for monitor entry [0x00007fe9adedc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:190)
      	- waiting to lock <0x00000000e220dc10> (a net.sf.cglib.core.AbstractClassGenerator$Source)
      	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
      	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:285)
      	at org.jvnet.hk2.internal.Utilities$6.run(Utilities.java:2117)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at org.jvnet.hk2.internal.Utilities.secureCreate(Utilities.java:2106)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2157)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000fb867498> (a java.lang.Object)
      	at org.glassfish.jersey.internal.inject.ContextInjectionResolver.resolve(ContextInjectionResolver.java:104)
      	at org.glassfish.jersey.server.internal.inject.DelegatedInjectionValueFactoryProvider$1.provide(DelegatedInjectionValueFactoryProvider.java:96)
      	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParameterValues(ParameterValueHelper.java:81)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$AbstractMethodParamInvoker.getParamValues(JavaResourceMethodDispatcherProvider.java:121)
      	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:195)
      	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:91)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:346)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:341)
      	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:101)
      	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:224)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
      	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
      	at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
      	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
      	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:198)
      	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:946)
      	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:323)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:372)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
      	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:218)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:681)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:452)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:529)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1085)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:445)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      ......
      

        Issue Links

          Activity

          Hide
          Jakub Podlesak added a comment -

          Current fix for JERSEY-1967 and JERSEY-1761 is in fact just a workaround and does not ensure the performance issues are gone (dynamic proxies are still being created under the cover). I need to add some more performance tests to cover this properly.

          Show
          Jakub Podlesak added a comment - Current fix for JERSEY-1967 and JERSEY-1761 is in fact just a workaround and does not ensure the performance issues are gone (dynamic proxies are still being created under the cover). I need to add some more performance tests to cover this properly.
          Hide
          Jakub Podlesak added a comment -

          Once the improvement in HK2 is available, we can rid of working around the issue by unwrapping already created dynamic proxy instance.

          Show
          Jakub Podlesak added a comment - Once the improvement in HK2 is available, we can rid of working around the issue by unwrapping already created dynamic proxy instance.
          Hide
          Jakub Podlesak added a comment - - edited

          This should get fixed in 2.3 version.

          Actual TODO:
          1) refactor the workaround
          2) past internal review process

          Show
          Jakub Podlesak added a comment - - edited This should get fixed in 2.3 version. Actual TODO: 1) refactor the workaround 2) past internal review process
          Hide
          Jakub Podlesak added a comment -

          recent continuous performance test results

          Show
          Jakub Podlesak added a comment - recent continuous performance test results
          Hide
          Jakub Podlesak added a comment -

          This was fixed in the master branch (see attached performance test result).

          The fix relies on current HK2 runtime internals.
          Jersey adds an artificial active descriptor to control proxy generation
          for method parameters
          in core-server/src/main/java/org/glassfish/jersey/server/internal/inject/DelegatedInjectionValueFactoryProvider.java

          Going to submit a new task so that a better fix is implemented,
          once HK2-128 improvement is ready.

          Show
          Jakub Podlesak added a comment - This was fixed in the master branch (see attached performance test result). The fix relies on current HK2 runtime internals. Jersey adds an artificial active descriptor to control proxy generation for method parameters in core-server/src/main/java/org/glassfish/jersey/server/internal/inject/DelegatedInjectionValueFactoryProvider.java Going to submit a new task so that a better fix is implemented, once HK2-128 improvement is ready.

            People

            • Assignee:
              Jakub Podlesak
              Reporter:
              aaronjwhiteside
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 3 hours Original Estimate - 3 hours
                3h
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 day, 5 hours
                1d 5h