glassfish
  1. glassfish
  2. GLASSFISH-18377

Glassfish threads hang on native method calls under high load

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2_b23
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Glassfish threads hang on native method calls under high load, CPU is pegged to 400% (I have four cores).

      On both JDK 1.6.0_29 and 1.7.0_3.

      I was testing using "ab -n 1000 -c 10" against an Jersey resource that talks to an OSGi service to write to a database using JPA2 (via Spring to manage JTA transactions). The ab command fails about half way through most of the time, because glassfish has stopped responding.

      No matter how many times I run jstack, those threads with the large stacks are always in some native method, it never changes even after half an hour.

      By which time glassfish tries to interrupt the http-thread-pool threads for being idle, but fails to do so and continuously prints out the warning that it's interrupting the threads in the server.log...

      I understand that his won't be easy to diagnose and I am willing to provide any support I can.

      I tried attaching jvisualvm to get some performance stats, but it won't start profiling and fails with an empty error message pop-up window.

      Jconsole doesn't want to attach either.

      2012-02-17 19:08:13
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode):
      
      "Attach Listener" daemon prio=10 tid=0x00007f54d0001000 nid=0x2ce5 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "http-thread-pool-8080(5)" daemon prio=10 tid=0x00007f5498018000 nid=0x2ca6 runnable [0x00007f54753d0000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Throwable.fillInStackTrace(Native Method)
      	- locked <0x00000000ffd61918> (a java.lang.ClassCastException)
      	at java.lang.Throwable.<init>(Throwable.java:181)
      	at java.lang.Exception.<init>(Exception.java:29)
      	at java.lang.RuntimeException.<init>(RuntimeException.java:32)
      	at java.lang.ClassCastException.<init>(ClassCastException.java:29)
      	at java.lang.Class.cast(Class.java:2990)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:149)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:93)
      	at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:126)
      	at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      	at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      	at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      	at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
      	at org.jvnet.hk2.component.Habitat.getBy(Habitat.java:1056)
      	at org.jvnet.hk2.component.Habitat.getByType(Habitat.java:1037)
      	at org.jvnet.hk2.component.Habitat.getComponent(Habitat.java:781)
      	at com.sun.enterprise.transaction.TransactionNamingProxy$UserTransactionProxy.create(TransactionNamingProxy.java:150)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:507)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at org.glassfish.osgijta.JTAExtender$MyInvocationHandler.invoke(JTAExtender.java:95)
      	at $Proxy92.begin(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      	at $Proxy205.begin(Unknown Source)
      	at org.springframework.transaction.jta.JtaTransactionManager.doJtaBegin(JtaTransactionManager.java:863)
      	at org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:820)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
      	at com.mm.common.db.RuleBasedTransactionTemplate.execute(RuleBasedTransactionTemplate.java:77)
      	at com.mm.service.optin.impl.OptInServiceImpl.initiateOptIn(OptInServiceImpl.java:159)
      	at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.glassfish.osgicdi.impl.OSGiServiceFactory$DynamicInvocationHandler.invoke(OSGiServiceFactory.java:234)
      	at $Proxy211.initiateOptIn(Unknown Source)
      	at com.mm.ws.optin.OptInResource.initiate(OptInResource.java:57)
      	at com.mm.ws.optin.OptInResource$Proxy$_$$_WeldClientProxy.initiate(OptInResource$Proxy$_$$_WeldClientProxy.java)
      	at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
      	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
      	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
      	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
      	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
      	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
      	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
      	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
      	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "http-thread-pool-8080(4)" daemon prio=10 tid=0x00007f5498016000 nid=0x2ca5 runnable [0x00007f54754d1000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Object.notifyAll(Native Method)
      	at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:494)
      	- locked <0x00000000e4954860> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
      	at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:386)
      	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1015)
      	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:731)
      	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:668)
      	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:601)
      	at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:560)
      	at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:717)
      	at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:778)
      	at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:457)
      	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1081)
      	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:844)
      	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1040)
      	at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:418)
      	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1128)
      	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2871)
      	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1516)
      	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1498)
      	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1463)
      	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:485)
      	at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:773)
      	at com.mm.common.db.dao.AbstractDao.getSingleResultOrNull(AbstractDao.java:145)
      	at com.mm.service.optin.dao.OptInDaoImpl.find(OptInDaoImpl.java:30)
      	at com.mm.service.optin.impl.OptInServiceImpl.doInitiateOptIn(OptInServiceImpl.java:189)
      	at com.mm.service.optin.impl.OptInServiceImpl$2.doInTransaction(OptInServiceImpl.java:164)
      	at com.mm.service.optin.impl.OptInServiceImpl$2.doInTransaction(OptInServiceImpl.java:160)
      	at com.mm.common.db.RuleBasedTransactionTemplate.execute(RuleBasedTransactionTemplate.java:80)
      	at com.mm.service.optin.impl.OptInServiceImpl.initiateOptIn(OptInServiceImpl.java:159)
      	at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.glassfish.osgicdi.impl.OSGiServiceFactory$DynamicInvocationHandler.invoke(OSGiServiceFactory.java:234)
      	at $Proxy211.initiateOptIn(Unknown Source)
      	at com.mm.ws.optin.OptInResource.initiate(OptInResource.java:57)
      	at com.mm.ws.optin.OptInResource$Proxy$_$$_WeldClientProxy.initiate(OptInResource$Proxy$_$$_WeldClientProxy.java)
      	at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
      	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
      	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
      	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
      	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
      	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
      	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
      	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
      	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "http-thread-pool-8080(3)" daemon prio=10 tid=0x00007f5498014800 nid=0x2ca4 runnable [0x00007f54755d2000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Throwable.fillInStackTrace(Native Method)
      	- locked <0x00000000ffb01ba0> (a java.lang.ClassCastException)
      	at java.lang.Throwable.<init>(Throwable.java:181)
      	at java.lang.Exception.<init>(Exception.java:29)
      	at java.lang.RuntimeException.<init>(RuntimeException.java:32)
      	at java.lang.ClassCastException.<init>(ClassCastException.java:29)
      	at java.lang.Class.cast(Class.java:2990)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:149)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:93)
      	at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:126)
      	at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      	at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      	at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      	at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
      	at org.jvnet.hk2.component.Habitat.getBy(Habitat.java:1056)
      	at org.jvnet.hk2.component.Habitat.getByType(Habitat.java:1037)
      	at org.jvnet.hk2.component.Habitat.getComponent(Habitat.java:781)
      	at com.sun.enterprise.transaction.TransactionNamingProxy$UserTransactionProxy.create(TransactionNamingProxy.java:150)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:507)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at org.glassfish.osgijta.JTAExtender$MyInvocationHandler.invoke(JTAExtender.java:95)
      	at $Proxy92.getStatus(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor345.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      	at $Proxy205.getStatus(Unknown Source)
      	at org.springframework.transaction.jta.JtaTransactionManager.doRollback(JtaTransactionManager.java:1033)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
      	at com.mm.common.db.RuleBasedTransactionTemplate.rollbackOnException(RuleBasedTransactionTemplate.java:111)
      	at com.mm.common.db.RuleBasedTransactionTemplate.execute(RuleBasedTransactionTemplate.java:85)
      	at com.mm.service.optin.impl.OptInServiceImpl.initiateOptIn(OptInServiceImpl.java:159)
      	at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.glassfish.osgicdi.impl.OSGiServiceFactory$DynamicInvocationHandler.invoke(OSGiServiceFactory.java:234)
      	at $Proxy211.initiateOptIn(Unknown Source)
      	at com.mm.ws.optin.OptInResource.initiate(OptInResource.java:57)
      	at com.mm.ws.optin.OptInResource$Proxy$_$$_WeldClientProxy.initiate(OptInResource$Proxy$_$$_WeldClientProxy.java)
      	at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
      	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
      	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
      	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
      	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
      	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
      	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
      	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
      	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "http-thread-pool-8080(2)" daemon prio=10 tid=0x00007f5498013800 nid=0x2ca3 runnable [0x00007f54756d3000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Throwable.fillInStackTrace(Native Method)
      	- locked <0x00000000ffcade40> (a java.lang.ClassCastException)
      	at java.lang.Throwable.<init>(Throwable.java:181)
      	at java.lang.Exception.<init>(Exception.java:29)
      	at java.lang.RuntimeException.<init>(RuntimeException.java:32)
      	at java.lang.ClassCastException.<init>(ClassCastException.java:29)
      	at java.lang.Class.cast(Class.java:2990)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:149)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:93)
      	at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:126)
      	at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      	at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      	at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      	at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
      	at org.jvnet.hk2.component.Habitat.getBy(Habitat.java:1056)
      	at org.jvnet.hk2.component.Habitat.getByType(Habitat.java:1037)
      	at org.jvnet.hk2.component.Habitat.getComponent(Habitat.java:781)
      	at com.sun.enterprise.transaction.TransactionNamingProxy$UserTransactionProxy.create(TransactionNamingProxy.java:150)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:507)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at org.glassfish.osgijta.JTAExtender$MyInvocationHandler.invoke(JTAExtender.java:95)
      	at $Proxy92.begin(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      	at $Proxy205.begin(Unknown Source)
      	at org.springframework.transaction.jta.JtaTransactionManager.doJtaBegin(JtaTransactionManager.java:863)
      	at org.springframework.transaction.jta.JtaTransactionManager.doBegin(JtaTransactionManager.java:820)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
      	at com.mm.common.db.RuleBasedTransactionTemplate.execute(RuleBasedTransactionTemplate.java:77)
      	at com.mm.service.optin.impl.OptInServiceImpl.initiateOptIn(OptInServiceImpl.java:159)
      	at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.glassfish.osgicdi.impl.OSGiServiceFactory$DynamicInvocationHandler.invoke(OSGiServiceFactory.java:234)
      	at $Proxy211.initiateOptIn(Unknown Source)
      	at com.mm.ws.optin.OptInResource.initiate(OptInResource.java:57)
      	at com.mm.ws.optin.OptInResource$Proxy$_$$_WeldClientProxy.initiate(OptInResource$Proxy$_$$_WeldClientProxy.java)
      	at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
      	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
      	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
      	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
      	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
      	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
      	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
      	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
      	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "http-thread-pool-8080(1)" daemon prio=10 tid=0x00007f5498003000 nid=0x2ca2 runnable [0x00007f5477bf8000]
         java.lang.Thread.State: RUNNABLE
      	at java.lang.Throwable.fillInStackTrace(Native Method)
      	- locked <0x00000000ffbaf790> (a java.lang.ClassCastException)
      	at java.lang.Throwable.<init>(Throwable.java:181)
      	at java.lang.Exception.<init>(Exception.java:29)
      	at java.lang.RuntimeException.<init>(RuntimeException.java:32)
      	at java.lang.ClassCastException.<init>(ClassCastException.java:29)
      	at java.lang.Class.cast(Class.java:2990)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:149)
      	at org.jvnet.hk2.component.InjectionManager.inject(InjectionManager.java:93)
      	at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:126)
      	at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      	at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      	at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      	at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
      	at org.jvnet.hk2.component.Habitat.getBy(Habitat.java:1056)
      	at org.jvnet.hk2.component.Habitat.getByType(Habitat.java:1037)
      	at org.jvnet.hk2.component.Habitat.getComponent(Habitat.java:781)
      	at com.sun.enterprise.transaction.TransactionNamingProxy$UserTransactionProxy.create(TransactionNamingProxy.java:150)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:507)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:455)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at javax.naming.InitialContext.lookup(InitialContext.java:392)
      	at org.glassfish.osgijta.JTAExtender$MyInvocationHandler.invoke(JTAExtender.java:95)
      	at $Proxy92.getStatus(Unknown Source)
      	at sun.reflect.GeneratedMethodAccessor345.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      	at $Proxy205.getStatus(Unknown Source)
      	at org.springframework.transaction.jta.JtaTransactionManager.doRollback(JtaTransactionManager.java:1033)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:845)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:822)
      	at com.mm.common.db.RuleBasedTransactionTemplate.rollbackOnException(RuleBasedTransactionTemplate.java:111)
      	at com.mm.common.db.RuleBasedTransactionTemplate.execute(RuleBasedTransactionTemplate.java:85)
      	at com.mm.service.optin.impl.OptInServiceImpl.initiateOptIn(OptInServiceImpl.java:159)
      	at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.glassfish.osgicdi.impl.OSGiServiceFactory$DynamicInvocationHandler.invoke(OSGiServiceFactory.java:234)
      	at $Proxy211.initiateOptIn(Unknown Source)
      	at com.mm.ws.optin.OptInResource.initiate(OptInResource.java:57)
      	at com.mm.ws.optin.OptInResource$Proxy$_$$_WeldClientProxy.initiate(OptInResource$Proxy$_$$_WeldClientProxy.java)
      	at sun.reflect.GeneratedMethodAccessor349.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
      	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
      	at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1542)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
      	at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
      	at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
      	at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
      	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
      	at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:849)
      	at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:746)
      	at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1045)
      	at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:228)
      	at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
      	at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
      	at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
      	at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
      	at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
      	at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "OptInServiceScheduler_QuartzSchedulerThread" prio=10 tid=0x00007f549007c800 nid=0x2ca1 in Object.wait() [0x00007f54757d6000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e411c8e0> (a java.lang.Object)
      	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:430)
      	- locked <0x00000000e411c8e0> (a java.lang.Object)
      
      "OptInServiceScheduler_Worker-10" prio=10 tid=0x00007f5490074000 nid=0x2ca0 in Object.wait() [0x00007f54758d7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410e7e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410e7e0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-9" prio=10 tid=0x00007f5490072000 nid=0x2c9f in Object.wait() [0x00007f54759d8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410e530> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410e530> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-8" prio=10 tid=0x00007f5490065800 nid=0x2c9e in Object.wait() [0x00007f5475ad9000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410e280> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410e280> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-7" prio=10 tid=0x00007f54900ac000 nid=0x2c9d in Object.wait() [0x00007f5475bda000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410dfd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410dfd0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-6" prio=10 tid=0x00007f54900ab800 nid=0x2c9c in Object.wait() [0x00007f5475cdb000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410dd20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410dd20> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-5" prio=10 tid=0x00007f549004c800 nid=0x2c9b in Object.wait() [0x00007f5475ddc000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410da70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410da70> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-4" prio=10 tid=0x00007f54900b4000 nid=0x2c99 in Object.wait() [0x00007f5475edd000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410d7c0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410d7c0> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-3" prio=10 tid=0x00007f5490097800 nid=0x2c98 in Object.wait() [0x00007f5475fde000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410d510> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410d510> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-2" prio=10 tid=0x00007f5490062000 nid=0x2c97 in Object.wait() [0x00007f54761e0000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e410d260> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e410d260> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "OptInServiceScheduler_Worker-1" prio=10 tid=0x00007f5490057000 nid=0x2c96 in Object.wait() [0x00007f54f81ea000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e4104f88> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:552)
      	- locked <0x00000000e4104f88> (a org.quartz.simpl.SimpleThreadPool$WorkerThread)
      
      "connector-timer-proxy" daemon prio=10 tid=0x00007f5490045800 nid=0x2c95 in Object.wait() [0x00007f54d41bc000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e41a25a0> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:509)
      	- locked <0x00000000e41a25a0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "Timer-2" daemon prio=10 tid=0x00007f5490041000 nid=0x2c94 in Object.wait() [0x00007f5476dec000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e41a2a68> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e41a2a68> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "p: thread-pool-1; w: 2" daemon prio=10 tid=0x00007f54ac23d000 nid=0x2c82 in Object.wait() [0x00007f5477cfb000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e38b9e60> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
      	at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:124)
      	- locked <0x00000000e38b9e60> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
      	at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:518)
      
      "p: thread-pool-1; w: 1" daemon prio=10 tid=0x00007f54ac2df800 nid=0x2c81 in Object.wait() [0x00007f54d4ece000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e38b9e60> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
      	at com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl.requestWork(WorkQueueImpl.java:124)
      	- locked <0x00000000e38b9e60> (a com.sun.corba.ee.impl.orbutil.threadpool.WorkQueueImpl)
      	at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:518)
      
      "dol-jar-scanner" daemon prio=10 tid=0x00007f54ac0f9000 nid=0x2c80 waiting on condition [0x00007f54d53d3000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e38ba370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "dol-jar-scanner" daemon prio=10 tid=0x00007f54ac2da800 nid=0x2c7f waiting on condition [0x00007f5476ae9000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e38ba370> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Timer-1" daemon prio=10 tid=0x00007f54ac467000 nid=0x2c63 in Object.wait() [0x00007f54760df000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e2d377d8> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:509)
      	- locked <0x00000000e2d377d8> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "admin-thread-pool-4848(6)" daemon prio=10 tid=0x00007f549c01f000 nid=0x2c62 in Object.wait() [0x00007f54768e7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "ContainerBackgroundProcessor[StandardEngine[glassfish-web].StandardHost[__asadmin].StandardContext[]]" daemon prio=10 tid=0x00007f54ac400000 nid=0x2c61 waiting on condition [0x00007f54769e8000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1771)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Timer-0" daemon prio=10 tid=0x00007f54a802d000 nid=0x2c5f in Object.wait() [0x00007f54762e1000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1f1f3a8> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e1f1f3a8> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "deployment-jar-scanner" daemon prio=10 tid=0x00007f54ac00e800 nid=0x2c5e waiting on condition [0x00007f54763e2000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0e502c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "admin-thread-pool-4848(5)" daemon prio=10 tid=0x00007f549c01d000 nid=0x2c5d in Object.wait() [0x00007f54764e3000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "admin-thread-pool-4848(4)" daemon prio=10 tid=0x00007f549c01b000 nid=0x2c5c in Object.wait() [0x00007f54765e4000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "admin-thread-pool-4848(3)" daemon prio=10 tid=0x00007f549c01f800 nid=0x2c5b in Object.wait() [0x00007f54766e5000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "admin-thread-pool-4848(2)" daemon prio=10 tid=0x00007f549c014000 nid=0x2c5a in Object.wait() [0x00007f54767e6000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-16-thread-1" daemon prio=10 tid=0x00007f54fd10b000 nid=0x2c51 waiting on condition [0x00007f54770ef000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0c9e428> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:163)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-15-thread-1" daemon prio=10 tid=0x00007f54fd185800 nid=0x2c50 waiting on condition [0x00007f54771f0000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e1920ab8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:163)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-14-thread-1" daemon prio=10 tid=0x00007f54fce9f000 nid=0x2c4f waiting on condition [0x00007f54772f1000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0f4b250> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:163)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-13-thread-1" daemon prio=10 tid=0x00007f54fd0f2000 nid=0x2c4e waiting on condition [0x00007f54773f2000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0c9df98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:163)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-12-thread-1" daemon prio=10 tid=0x00007f54fc1b4000 nid=0x2c4d waiting on condition [0x00007f54774f3000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0c9e198> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at org.jvnet.hk2.config.Transactions$Notifier$1.run(Transactions.java:163)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Grizzly" daemon prio=10 tid=0x00007f54fce44000 nid=0x2c4c waiting on condition [0x00007f54775f4000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e1fc2de0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
      	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "admin-thread-pool-4848(1)" daemon prio=10 tid=0x00007f549c013000 nid=0x2c4b in Object.wait() [0x00007f54776f5000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.SyncThreadPool$SyncThreadWorker.getTask(SyncThreadPool.java:374)
      	- locked <0x00000000e1ecc508> (a java.lang.Object)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:524)
      	at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
      	at java.lang.Thread.run(Thread.java:662)
      
      "ContainerBackgroundProcessor[StandardEngine[glassfish-web].StandardHost[server].StandardContext[/web-services]]" daemon prio=10 tid=0x00007f54b8319000 nid=0x2c49 waiting on condition [0x00007f54777f6000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1771)
      	at java.lang.Thread.run(Thread.java:662)
      
      "deployment-jar-scanner" daemon prio=10 tid=0x00007f54e400a800 nid=0x2c46 waiting on condition [0x00007f54778f7000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0e502c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "deployment-jar-scanner" daemon prio=10 tid=0x00007f54e400d000 nid=0x2c45 waiting on condition [0x00007f54779f8000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0e502c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "deployment-jar-scanner" daemon prio=10 tid=0x00007f54b87a5000 nid=0x2c44 waiting on condition [0x00007f5477af9000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0e502c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Spring DM Context Creation Timer" daemon prio=10 tid=0x00007f54a0172000 nid=0x2c3c in Object.wait() [0x00007f5477efd000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1c118e0> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e1c118e0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "Gemini Blueprint context shutdown thread" daemon prio=10 tid=0x00007f54a017a000 nid=0x2c3b in Object.wait() [0x00007f5477ffe000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1c1bec0> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e1c1bec0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "Spring DM Context Creation Timer" daemon prio=10 tid=0x00007f54a00d7000 nid=0x2c39 in Object.wait() [0x00007f54d42bd000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1c1c2d0> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e1c1c2d0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "Gemini Blueprint context shutdown thread" daemon prio=10 tid=0x00007f54a0051800 nid=0x2c38 in Object.wait() [0x00007f54d47c7000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e1c1c6f8> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e1c1c6f8> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "telnetconsole.Listener" daemon prio=10 tid=0x00007f54b817e800 nid=0x2c34 runnable [0x00007f54d45c5000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.PlainSocketImpl.socketAccept(Native Method)
      	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      	- locked <0x00000000e16a6e10> (a java.net.SocksSocketImpl)
      	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      	at java.net.ServerSocket.accept(ServerSocket.java:430)
      	at org.apache.felix.shell.remote.Listener$Acceptor.run(Listener.java:150)
      	at java.lang.Thread.run(Thread.java:662)
      
      "pool-6-thread-1" prio=10 tid=0x00007f54b814c000 nid=0x2c31 waiting on condition [0x00007f54d46c6000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e1846020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "SCR Component Actor" daemon prio=10 tid=0x00007f54b81a5800 nid=0x2c26 in Object.wait() [0x00007f54d48c8000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e17edd00> (a java.util.LinkedList)
      	at java.lang.Object.wait(Object.java:485)
      	at org.apache.felix.scr.impl.ComponentActorThread.run(ComponentActorThread.java:74)
      	- locked <0x00000000e17edd00> (a java.util.LinkedList)
      	at java.lang.Thread.run(Thread.java:662)
      
      "ContainerBackgroundProcessor[StandardEngine[glassfish-web]]" daemon prio=10 tid=0x00007f54b8163800 nid=0x2c1d waiting on condition [0x00007f54d49c9000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1771)
      	at java.lang.Thread.run(Thread.java:662)
      
      "ContainerBackgroundProcessor[StandardEngine[glassfish-web].StandardHost[server].StandardContext[]]" daemon prio=10 tid=0x00007f54b80ef000 nid=0x2c1b waiting on condition [0x00007f54d4aca000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1771)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Configuration Updater" daemon prio=10 tid=0x00007f54b8197000 nid=0x2c11 in Object.wait() [0x00007f54d4bcb000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e17e9240> (a java.util.LinkedList)
      	at java.lang.Object.wait(Object.java:485)
      	at org.apache.felix.cm.impl.UpdateThread.run(UpdateThread.java:67)
      	- locked <0x00000000e17e9240> (a java.util.LinkedList)
      
      "fileinstall-/home/awhiteside/Projects/mtrust/deploy/glassfish-mtrust-package/target/glassfish3/glassfish/domains/domain1/autodeploy/bundles/" daemon prio=10 tid=0x00007f54b81ab800 nid=0x2c10 in Object.wait() [0x00007f54d4ccc000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e17cd718> (a org.apache.felix.fileinstall.internal.DirectoryWatcher)
      	at org.apache.felix.fileinstall.internal.DirectoryWatcher.run(DirectoryWatcher.java:268)
      	- locked <0x00000000e17cd718> (a org.apache.felix.fileinstall.internal.DirectoryWatcher)
      
      "DestroyJavaVM" prio=10 tid=0x00007f54fc00a800 nid=0x2bdf waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "GlassFish Kernel Main Thread" prio=10 tid=0x00007f54fd18d000 nid=0x2c0f in Object.wait() [0x00007f54d4dcd000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e18a3870> (a com.sun.enterprise.v3.server.AppServerStartup$1)
      	at java.lang.Object.wait(Object.java:485)
      	at com.sun.enterprise.v3.server.AppServerStartup$1.run(AppServerStartup.java:163)
      	- locked <0x00000000e18a3870> (a com.sun.enterprise.v3.server.AppServerStartup$1)
      
      "DynamicReloader" daemon prio=10 tid=0x00007f54fd119000 nid=0x2c0e in Object.wait() [0x00007f54d57d7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e18a3be0> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:509)
      	- locked <0x00000000e18a3be0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "RMI RenewClean-[127.0.0.1:8686]" daemon prio=10 tid=0x00007f548801c000 nid=0x2c0c in Object.wait() [0x00007f54d4fcf000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e19806b8> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      	- locked <0x00000000e19806b8> (a java.lang.ref.ReferenceQueue$Lock)
      	at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
      	at java.lang.Thread.run(Thread.java:662)
      
      "RMI Scheduler(0)" daemon prio=10 tid=0x00007f5478012800 nid=0x2c0b waiting on condition [0x00007f54d50d0000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0f687f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
      	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "transaction-manager" daemon prio=10 tid=0x00007f54fd139000 nid=0x2c0a in Object.wait() [0x00007f54d52d2000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e175e738> (a java.util.TaskQueue)
      	at java.lang.Object.wait(Object.java:485)
      	at java.util.TimerThread.mainLoop(Timer.java:483)
      	- locked <0x00000000e175e738> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "GC Daemon" daemon prio=10 tid=0x00007f5478018000 nid=0x2c08 in Object.wait() [0x00007f54d54d4000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0f4b768> (a sun.misc.GC$LatencyLock)
      	at sun.misc.GC$Daemon.run(GC.java:100)
      	- locked <0x00000000e0f4b768> (a sun.misc.GC$LatencyLock)
      
      "RMI Reaper" prio=10 tid=0x00007f5478016000 nid=0x2c07 in Object.wait() [0x00007f54d55d5000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0f3bc98> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      	- locked <0x00000000e0f3bc98> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      	at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
      	at java.lang.Thread.run(Thread.java:662)
      
      "RMI TCP Accept-8686" daemon prio=10 tid=0x00007f5478009800 nid=0x2c06 runnable [0x00007f54d56d6000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.PlainSocketImpl.socketAccept(Native Method)
      	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
      	- locked <0x00000000e0ee5228> (a java.net.SocksSocketImpl)
      	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
      	at java.net.ServerSocket.accept(ServerSocket.java:430)
      	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
      	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
      	at java.lang.Thread.run(Thread.java:662)
      
      "PingService" daemon prio=10 tid=0x00007f54fd0eb800 nid=0x2c04 in Object.wait() [0x00007f54d58d8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0ef4fd8> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:509)
      	- locked <0x00000000e0ef4fd8> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "AutoDeployer" daemon prio=10 tid=0x00007f54fd0d9000 nid=0x2c02 in Object.wait() [0x00007f54d59d9000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0ef53d0> (a java.util.TaskQueue)
      	at java.util.TimerThread.mainLoop(Timer.java:509)
      	- locked <0x00000000e0ef53d0> (a java.util.TaskQueue)
      	at java.util.TimerThread.run(Timer.java:462)
      
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x00007f548c001000 nid=0x2bfe runnable [0x00007f54f82eb000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
      	- locked <0x00000000e0eb3a90> (a sun.nio.ch.Util$2)
      	- locked <0x00000000e0eb3a80> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e0eb3838> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
      	at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
      	at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
      	at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Thread-18" prio=10 tid=0x00007f54fd0d2000 nid=0x2bfd in Object.wait() [0x00007f54f83ec000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0ec4e90> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
      	- locked <0x00000000e0ec4e90> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.start(Controller.java:869)
      	at com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
      	- locked <0x00000000e0eb34a0> (a java.lang.Object)
      	at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
      	at com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
      	at com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:140)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
      
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x00007f5494006800 nid=0x2bfc runnable [0x00007f54f84ed000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
      	- locked <0x00000000e0e9a270> (a sun.nio.ch.Util$2)
      	- locked <0x00000000e0e9a260> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e0e9a018> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
      	at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
      	at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
      	at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Thread-15" prio=10 tid=0x00007f54fd0d0800 nid=0x2bfb in Object.wait() [0x00007f54f85ee000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0eb2b78> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
      	- locked <0x00000000e0eb2b78> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.start(Controller.java:869)
      	at com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
      	- locked <0x00000000e0e99c80> (a java.lang.Object)
      	at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
      	at com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
      	at com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:140)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
      
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x00007f54a0001000 nid=0x2bfa runnable [0x00007f54f86ef000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
      	- locked <0x00000000e0e775d8> (a sun.nio.ch.Util$2)
      	- locked <0x00000000e0e775c8> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e0e77380> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
      	at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
      	at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
      	at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x00007f54a4001000 nid=0x2bf9 runnable [0x00007f54f87f0000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
      	- locked <0x00000000e0e929b0> (a sun.nio.ch.Util$2)
      	- locked <0x00000000e0e929a0> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e0e92758> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
      	at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
      	at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
      	at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Grizzly-kernel-thread(1)" daemon prio=10 tid=0x00007f54ac00c800 nid=0x2bf8 runnable [0x00007f54f88f1000]
         java.lang.Thread.State: RUNNABLE
      	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
      	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
      	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
      	- locked <0x00000000e0e2f218> (a sun.nio.ch.Util$2)
      	- locked <0x00000000e0e2f208> (a java.util.Collections$UnmodifiableSet)
      	- locked <0x00000000e0e2efc0> (a sun.nio.ch.EPollSelectorImpl)
      	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
      	at com.sun.grizzly.TCPSelectorHandler.select(TCPSelectorHandler.java:513)
      	at com.sun.grizzly.SelectorHandlerRunner.doSelect(SelectorHandlerRunner.java:190)
      	at com.sun.grizzly.SelectorHandlerRunner.run(SelectorHandlerRunner.java:132)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Thread-10" prio=10 tid=0x00007f54fd0cf800 nid=0x2bf7 in Object.wait() [0x00007f54f89f2000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0eb8d58> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
      	- locked <0x00000000e0eb8d58> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.start(Controller.java:869)
      	at com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
      	- locked <0x00000000e0e923d0> (a java.lang.Object)
      	at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
      	at com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
      	at com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
      
      "Thread-6" prio=10 tid=0x00007f54fd0b6800 nid=0x2bf6 in Object.wait() [0x00007f54f8af3000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0e91ad8> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
      	- locked <0x00000000e0e91ad8> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.start(Controller.java:869)
      	at com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
      	- locked <0x00000000e0e76ff8> (a java.lang.Object)
      	at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
      	at com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
      	at com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
      
      "Thread-4" prio=10 tid=0x00007f54fd0bd800 nid=0x2bf5 in Object.wait() [0x00007f54f8bf4000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0e76560> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.waitUntilSelectorHandlersStop(Controller.java:1119)
      	- locked <0x00000000e0e76560> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.sun.grizzly.Controller.start(Controller.java:869)
      	at com.sun.grizzly.http.SelectorThread.startListener(SelectorThread.java:1285)
      	- locked <0x00000000e0e2ebe8> (a java.lang.Object)
      	at com.sun.grizzly.http.SelectorThread.run(SelectorThread.java:1123)
      	at com.sun.grizzly.http.SelectorThread.startEndpoint(SelectorThread.java:1212)
      	at com.sun.enterprise.v3.services.impl.GrizzlyListener.startEndpoint(GrizzlyListener.java:142)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy$2.run(GrizzlyProxy.java:342)
      
      "Thread-2" prio=10 tid=0x00007f54fd084000 nid=0x2bf4 waiting on condition [0x00007f54f8cf5000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0ec8b38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
      	at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
      	at com.sun.enterprise.server.logging.GFFileHandler.log(GFFileHandler.java:665)
      	at com.sun.enterprise.server.logging.GFFileHandler$1.run(GFFileHandler.java:179)
      
      "pool-1-thread-1" daemon prio=10 tid=0x00007f54fd077000 nid=0x2bf3 waiting on condition [0x00007f54f8df6000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000e0ec7ae0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
      	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
      	at java.lang.Thread.run(Thread.java:662)
      
      "FelixFrameworkWiring" daemon prio=10 tid=0x00007f54fce32800 nid=0x2bef in Object.wait() [0x00007f54f8ffe000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e114b498> (a java.util.ArrayList)
      	at java.lang.Object.wait(Object.java:485)
      	at org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:156)
      	- locked <0x00000000e114b498> (a java.util.ArrayList)
      	at java.lang.Thread.run(Thread.java:662)
      
      "FelixStartLevel" daemon prio=10 tid=0x00007f54fc287000 nid=0x2bee in Object.wait() [0x00007f5500b19000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e0a16ec8> (a java.util.ArrayList)
      	at java.lang.Object.wait(Object.java:485)
      	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:272)
      	- locked <0x00000000e0a16ec8> (a java.util.ArrayList)
      	at java.lang.Thread.run(Thread.java:662)
      
      "FelixDispatchQueue" daemon prio=10 tid=0x00007f54b800e800 nid=0x2bed in Object.wait() [0x00007f5500a18000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e08c5d50> (a java.util.ArrayList)
      	at java.lang.Object.wait(Object.java:485)
      	at org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:1063)
      	- locked <0x00000000e08c5d50> (a java.util.ArrayList)
      	at org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
      	at org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:101)
      	at java.lang.Thread.run(Thread.java:662)
      
      "Low Memory Detector" daemon prio=10 tid=0x00007f54fc129000 nid=0x2bea runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "C2 CompilerThread1" daemon prio=10 tid=0x00007f54fc126800 nid=0x2be9 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "C2 CompilerThread0" daemon prio=10 tid=0x00007f54fc108000 nid=0x2be8 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Signal Dispatcher" daemon prio=10 tid=0x00007f54fc0b7000 nid=0x2be7 runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "Finalizer" daemon prio=10 tid=0x00007f54fc09a000 nid=0x2be6 in Object.wait() [0x00007f550119b000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e081e2b8> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
      	- locked <0x00000000e081e2b8> (a java.lang.ref.ReferenceQueue$Lock)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
      	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0x00007f54fc098800 nid=0x2be5 in Object.wait() [0x00007f550129c000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0x00000000e081e250> (a java.lang.ref.Reference$Lock)
      	at java.lang.Object.wait(Object.java:485)
      	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      	- locked <0x00000000e081e250> (a java.lang.ref.Reference$Lock)
      
      "VM Thread" prio=10 tid=0x00007f54fc091800 nid=0x2be4 runnable 
      
      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f54fc01e000 nid=0x2be0 runnable 
      
      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f54fc020000 nid=0x2be1 runnable 
      
      "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f54fc021800 nid=0x2be2 runnable 
      
      "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f54fc023800 nid=0x2be3 runnable 
      
      "VM Periodic Task Thread" prio=10 tid=0x00007f54fc133800 nid=0x2beb waiting on condition 
      
      JNI global references: 1562
      
      

        Activity

        Hide
        aaronjwhiteside added a comment -

        Although I cannot provide all of the source code for the application (IP issues), I can probably provide a self contained ZIP of glassfish + the application + embedded db pre-configured, that reproduces the problem.

        Then all one would have to do is unzip it, start the domain and run ab against it.

        Show
        aaronjwhiteside added a comment - Although I cannot provide all of the source code for the application (IP issues), I can probably provide a self contained ZIP of glassfish + the application + embedded db pre-configured, that reproduces the problem. Then all one would have to do is unzip it, start the domain and run ab against it.
        Hide
        aaronjwhiteside added a comment -

        So just FYI, I can reproduce this on every machine I try, so it's not something specific to mine.

        If you want the zip let me know where I can upload it (it's about 90mb).

        Also I should have mentioned earlier I am running on Linux (Fedora) to be specific.

        Show
        aaronjwhiteside added a comment - So just FYI, I can reproduce this on every machine I try, so it's not something specific to mine. If you want the zip let me know where I can upload it (it's about 90mb). Also I should have mentioned earlier I am running on Linux (Fedora) to be specific.
        Hide
        aaronjwhiteside added a comment -

        bump...

        Show
        aaronjwhiteside added a comment - bump...

          People

          • Assignee:
            shreedhar_ganapathy
            Reporter:
            aaronjwhiteside
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: