jersey
  1. jersey
  2. JERSEY-1991

Processing of most container requests block in HK2 ServiceLocatorImpl

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1
    • Fix Version/s: 2.4
    • Component/s: core
    • Labels:
      None
    • Environment:

      OS X, Debian

      Description

      While working on a Jersey container implementation for Netty 4 i noticed all threads of my Netty executor group ( running the Jersey handler i implemented ) are very often blocked on one of two methods in hk2's org.jvnet.hk2.internal.ServiceLocatorImpl ( resolveContext, internalGetDescriptor ) that both synchronise on the same lock.

      Has anyone experienced anything similar ?
      Here are the two stack traces i'm getting

      Stacks at 03:51:49 PM (uptime 7m 46s)

      defaultEventExecutorGroup-4-15 [BLOCKED] CPU time: 12s
      org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(Injectee, Type, String, Unqualified, Annotation[])
      org.jvnet.hk2.internal.ServiceLocatorImpl.getService(Type, Annotation[])
      org.jvnet.hk2.internal.IterableProviderImpl.get()
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.getMethodRouter(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.access$400(MethodSelectingRouter, ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter$4.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.RoutingStage._apply(ContainerRequest, Router)<4 recursive calls>
      org.glassfish.jersey.server.internal.routing.RoutingStage.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.RoutingStage.apply(Object)
      org.glassfish.jersey.process.internal.Stages.process(Object, Stage, Ref)
      org.glassfish.jersey.server.ServerRuntime$1.run()
      org.glassfish.jersey.internal.Errors$1.call()<2 recursive calls>
      org.glassfish.jersey.internal.Errors.process(Callable, boolean)
      org.glassfish.jersey.internal.Errors.process(Producer, boolean)
      org.glassfish.jersey.internal.Errors.process(Runnable)
      org.glassfish.jersey.process.internal.RequestScope.runInScope(Runnable)
      org.glassfish.jersey.server.ServerRuntime.process(ContainerRequest)
      org.glassfish.jersey.server.ApplicationHandler.handle(ContainerRequest)
      andras.jersey.JerseyNettyHandler.channelRead0(ChannelHandlerContext, FullHttpRequest)
      andras.jersey.JerseyNettyHandler.channelRead0(ChannelHandlerContext, Object)
      io.netty.channel.SimpleChannelInboundHandler.channelRead(ChannelHandlerContext, Object)
      io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(Object)
      io.netty.channel.DefaultChannelHandlerContext.access$800(DefaultChannelHandlerContext, Object)
      io.netty.channel.DefaultChannelHandlerContext$9.run()
      io.netty.util.concurrent.DefaultEventExecutor.run()
      io.netty.util.concurrent.SingleThreadEventExecutor$2.run()
      java.lang.Thread.run()

      Stacks at 03:51:49 PM (uptime 7m 46s)

      defaultEventExecutorGroup-4-16 [BLOCKED] CPU time: 12s
      org.jvnet.hk2.internal.ServiceLocatorImpl.resolveContext(Class)
      org.jvnet.hk2.internal.Utilities.createService(ActiveDescriptor, Injectee, ServiceLocatorImpl, ServiceHandle, Class)
      org.jvnet.hk2.internal.ServiceLocatorImpl.getService(Type, Annotation[])
      org.jvnet.hk2.internal.IterableProviderImpl.get()
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.getMethodRouter(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.access$400(MethodSelectingRouter, ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter$4.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.MethodSelectingRouter.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.RoutingStage._apply(ContainerRequest, Router)<4 recursive calls>
      org.glassfish.jersey.server.internal.routing.RoutingStage.apply(ContainerRequest)
      org.glassfish.jersey.server.internal.routing.RoutingStage.apply(Object)
      org.glassfish.jersey.process.internal.Stages.process(Object, Stage, Ref)
      org.glassfish.jersey.server.ServerRuntime$1.run()
      org.glassfish.jersey.internal.Errors$1.call()<2 recursive calls>
      org.glassfish.jersey.internal.Errors.process(Callable, boolean)
      org.glassfish.jersey.internal.Errors.process(Producer, boolean)
      org.glassfish.jersey.internal.Errors.process(Runnable)
      org.glassfish.jersey.process.internal.RequestScope.runInScope(Runnable)
      org.glassfish.jersey.server.ServerRuntime.process(ContainerRequest)
      org.glassfish.jersey.server.ApplicationHandler.handle(ContainerRequest)
      andras.jersey.JerseyNettyHandler.channelRead0(ChannelHandlerContext, FullHttpRequest)
      andras.jersey.JerseyNettyHandler.channelRead0(ChannelHandlerContext, Object)
      io.netty.channel.SimpleChannelInboundHandler.channelRead(ChannelHandlerContext, Object)
      io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(Object)
      io.netty.channel.DefaultChannelHandlerContext.access$800(DefaultChannelHandlerContext, Object)
      io.netty.channel.DefaultChannelHandlerContext$9.run()
      io.netty.util.concurrent.DefaultEventExecutor.run()
      io.netty.util.concurrent.SingleThreadEventExecutor$2.run()
      java.lang.Thread.run()

      Attached is a junit test suite to ( very easily ) reproduce this. I couldnt find an easy way to write an assertion for this so reproducing this is a 3 step process:

      • run the test with -Djersey.messages=100000 or a similarly high number to keep the VM busy
      • take a thread dump of the running VM
      • the test itself will output some Metrics data on latency as experienced by the test framework's client ( high quantiles, median, mean )

      I couldnt find a way to attach files to this issue, so the test is at https://github.com/pgn84/jersey-performance

      thanks!
      Andras

        Issue Links

          Activity

          Hide
          Jakub Podlesak added a comment -

          This needs to be fixed in HK2, set fix version to 2.4, as it is unlikely this would get fixed in 2.3. Actual fix version depends on HK2 fix.

          Show
          Jakub Podlesak added a comment - This needs to be fixed in HK2, set fix version to 2.4, as it is unlikely this would get fixed in 2.3. Actual fix version depends on HK2 fix.
          Hide
          Jakub Podlesak added a comment -

          This has been fixed in the Jersey master branch: HK2 version upgraded to 2.2.0-b21
          where locking strategy has been relaxed to avoid above described blocking issues.

          Show
          Jakub Podlesak added a comment - This has been fixed in the Jersey master branch: HK2 version upgraded to 2.2.0-b21 where locking strategy has been relaxed to avoid above described blocking issues.

            People

            • Assignee:
              Jakub Podlesak
              Reporter:
              andras84
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 6 hours
                6h