glassfish
  1. glassfish
  2. GLASSFISH-20582

Deadlock while running quicklook tests (test_gd_security profile)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: logging
    • Labels:
      None
    • Environment:

      svn r. 62108

      Description

      On hudson I have a job which does clean svn co of GF trunk, builds it, integrates custom Metro build and runs set of tests on built binary - QL secure, QL, ws devtests, CTS smoke.

      Today during QL secure run GF hung and jstack reports:

      Found one Java-level deadlock:
      =============================
      "RunLevelControllerThread-1369715622963":
        waiting to lock monitor 0x00007f4c0c02db50 (object 0x00000000fce39bb8, a java.util.logging.LogManager$LoggerContext),
        which is held by "RunLevelControllerThread-1369715622644"
      "RunLevelControllerThread-1369715622644":
        waiting to lock monitor 0x00007f4c0c02edb0 (object 0x00000000fce2d658, a java.util.logging.LogManager),
        which is held by "RunLevelControllerThread-1369715622636"
      "RunLevelControllerThread-1369715622636":
        waiting to lock monitor 0x00007f4c0c02db50 (object 0x00000000fce39bb8, a java.util.logging.LogManager$LoggerContext),
        which is held by "RunLevelControllerThread-1369715622644"
      
      Java stack information for the threads listed above:
      ===================================================
      "RunLevelControllerThread-1369715622963":
      	at java.util.logging.LogManager$LoggerContext.findLogger(LogManager.java:489)
      	- waiting to lock <0x00000000fce39bb8> (a java.util.logging.LogManager$LoggerContext)
      	at java.util.logging.LogManager.getLogger(LogManager.java:910)
      	at java.util.logging.LogManager.demandLogger(LogManager.java:400)
      	at java.util.logging.Logger.demandLogger(Logger.java:317)
      	at java.util.logging.Logger.getLogger(Logger.java:361)
      	at org.jboss.logging.JDKLogger.<init>(JDKLogger.java:37)
      	at org.jboss.logging.JDKLoggerProvider.getLogger(JDKLoggerProvider.java:28)
      	at org.jboss.logging.LoggerProviders.find(LoggerProviders.java:37)
      	at org.jboss.logging.LoggerProviders.<clinit>(LoggerProviders.java:32)
      	at org.jboss.logging.Logger.getLogger(Logger.java:2163)
      	at org.jboss.logging.Logger.getMessageLogger(Logger.java:2259)
      	at org.jboss.logging.Logger.getMessageLogger(Logger.java:2214)
      	at org.hibernate.validator.internal.util.logging.LoggerFactory.make(LoggerFactory.java:29)
      	at org.hibernate.validator.internal.util.Version.<clinit>(Version.java:27)
      	at org.hibernate.validator.internal.engine.ConfigurationImpl.<clinit>(ConfigurationImpl.java:65)
      	at org.hibernate.validator.HibernateValidator.createGenericConfiguration(HibernateValidator.java:41)
      	at javax.validation.Validation$GenericBootstrapImpl.configure(Validation.java:276)
      	at javax.validation.Validation.buildDefaultValidatorFactory(Validation.java:110)
      	at org.jvnet.hk2.config.WriteableView.<init>(WriteableView.java:106)
      	at org.jvnet.hk2.config.ConfigSupport.getWriteableView(ConfigSupport.java:213)
      	at org.jvnet.hk2.config.ConfigSupport.getWriteableView(ConfigSupport.java:235)
      	at org.jvnet.hk2.config.ConfigSupport._apply(ConfigSupport.java:159)
      	at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:135)
      	at org.jvnet.hk2.config.ConfigSupport.apply(ConfigSupport.java:114)
      	at com.sun.enterprise.config.modularity.parser.ModuleConfigurationLoader.createConfigBeanForType(ModuleConfigurationLoader.java:97)
      	- locked <0x00000000e16b3350> (a com.sun.enterprise.config.modularity.ConfigModularityUtils)
      	at com.sun.enterprise.config.modularity.ExtensionPatternInvocationImpl.handleExtension(ExtensionPatternInvocationImpl.java:91)
      	at org.jvnet.hk2.config.Dom.invokeConfigExtensionMethod(Dom.java:1136)
      	at org.jvnet.hk2.config.Dom.invoke(Dom.java:1077)
      	at org.glassfish.config.support.TranslatedConfigView.invoke(TranslatedConfigView.java:129)
      	at com.sun.proxy.$Proxy36.getExtensionByType(Unknown Source)
      	at com.sun.enterprise.config.modularity.ConfigModularityJustInTimeInjectionResolver.justInTimeResolution(ConfigModularityJustInTimeInjectionResolver.java:101)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.secondChanceResolve(ServiceLocatorImpl.java:421)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetInjecteeDescriptor(ServiceLocatorImpl.java:480)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.getInjecteeDescriptor(ServiceLocatorImpl.java:492)
      	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:69)
      	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:191)
      	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:214)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:311)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:180)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e222f268> (a java.lang.Object)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:734)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:719)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      "RunLevelControllerThread-1369715622644":
      	at java.util.logging.LogManager.drainLoggerRefQueueBounded(LogManager.java:811)
      	- waiting to lock <0x00000000fce2d658> (a java.util.logging.LogManager)
      	at java.util.logging.LogManager$LoggerContext.addLocalLogger(LogManager.java:511)
      	- locked <0x00000000fce39bb8> (a java.util.logging.LogManager$LoggerContext)
      	at java.util.logging.LogManager.addLogger(LogManager.java:848)
      	at java.util.logging.LogManager.demandLogger(LogManager.java:405)
      	at java.util.logging.Logger.demandLogger(Logger.java:317)
      	at java.util.logging.Logger.getLogger(Logger.java:361)
      	at org.glassfish.grizzly.Grizzly.logger(Grizzly.java:69)
      	at org.glassfish.grizzly.nio.transport.TCPNIOServerConnection.<clinit>(TCPNIOServerConnection.java:66)
      	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.obtainServerNIOConnection(TCPNIOTransport.java:625)
      	at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bindToChannelAndAddress(TCPNIOBindingHandler.java:136)
      	at org.glassfish.grizzly.nio.transport.TCPNIOBindingHandler.bind(TCPNIOBindingHandler.java:87)
      	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:450)
      	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:439)
      	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:95)
      	at org.glassfish.grizzly.config.GenericGrizzlyListener.start(GenericGrizzlyListener.java:168)
      	at com.sun.enterprise.v3.services.impl.GlassfishNetworkListener.start(GlassfishNetworkListener.java:94)
      	at com.sun.enterprise.v3.services.impl.GrizzlyProxy.start(GrizzlyProxy.java:232)
      	at com.sun.enterprise.v3.services.impl.GrizzlyService.createNetworkProxy(GrizzlyService.java:484)
      	- locked <0x00000000fcf15ab0> (a com.sun.enterprise.v3.services.impl.GrizzlyService)
      	at com.sun.enterprise.v3.services.impl.GrizzlyService.postConstruct(GrizzlyService.java:407)
      	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:281)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:328)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:180)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2242360> (a java.lang.Object)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:734)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:719)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      "RunLevelControllerThread-1369715622636":
      	at java.util.logging.LogManager$LoggerContext.addLocalLogger(LogManager.java:505)
      	- waiting to lock <0x00000000fce39bb8> (a java.util.logging.LogManager$LoggerContext)
      	at java.util.logging.LogManager.addLogger(LogManager.java:848)
      	at com.sun.logging.LogDomains.addLoggerToLogManager(LogDomains.java:471)
      	- locked <0x00000000fce2d658> (a java.util.logging.LogManager)
      	- locked <0x00000000e0d86878> (a java.lang.Class for java.util.logging.Logger)
      	at com.sun.logging.LogDomains.getLogger(LogDomains.java:446)
      	- locked <0x00000000e1744660> (a java.lang.Class for com.sun.logging.LogDomains)
      	at com.sun.enterprise.resource.pool.PoolManagerImpl.<clinit>(PoolManagerImpl.java:103)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
      	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1091)
      	at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:244)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:319)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2243290> (a java.lang.Object)
      	at org.jvnet.hk2.internal.IterableProviderImpl$MyIterator.next(IterableProviderImpl.java:205)
      	at org.glassfish.api.invocation.InvocationManagerImpl.<init>(InvocationManagerImpl.java:96)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
      	at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1091)
      	at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:244)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:319)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2243378> (a java.lang.Object)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:558)
      	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
      	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:191)
      	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:214)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:311)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2243468> (a java.lang.Object)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:558)
      	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
      	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:191)
      	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:214)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:311)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2243558> (a java.lang.Object)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:558)
      	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
      	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:191)
      	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:214)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:311)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e2243648> (a java.lang.Object)
      	at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:558)
      	at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
      	at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:191)
      	at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:214)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:311)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.jvnet.hk2.internal.SingletonContext.findOrCreate(SingletonContext.java:118)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e1f8be50> (a java.lang.Object)
      	at org.glassfish.internal.data.EngineInfo.getContainer(EngineInfo.java:86)
      	at org.glassfish.kernel.javaee.WebContainerStarter.startWebContainer(WebContainerStarter.java:241)
      	at org.glassfish.kernel.javaee.WebContainerStarter.postConstruct(WebContainerStarter.java:180)
      	at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:281)
      	at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:328)
      	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:448)
      	at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:180)
      	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2204)
      	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
      	- locked <0x00000000e22437a8> (a java.lang.Object)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:734)
      	at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:719)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:722)
      
      Found 1 deadlock.
      
      

      Note that this issue seems to be random.

        Activity

        Hide
        Lukas Jungmann added a comment -

        full thread dump

        Show
        Lukas Jungmann added a comment - full thread dump
        Hide
        rajendra_inamdar added a comment -

        Checked with Jim who confirmed that this is a JDK bug which has been fixed in 7u25 and above.

        See:
        https://jbs.oracle.com/bugs/browse/JDK-8010939

        Show
        rajendra_inamdar added a comment - Checked with Jim who confirmed that this is a JDK bug which has been fixed in 7u25 and above. See: https://jbs.oracle.com/bugs/browse/JDK-8010939
        Hide
        rajendra_inamdar added a comment -

        Closing this bug as this is not a GF bug, but a bug in JDK libraries which will be fixes in 7u25 and above. Since GF 4.0 will be certified with 7u21, this bug should be documented in the Java EE 7 RI/SDK release notes.

        Show
        rajendra_inamdar added a comment - Closing this bug as this is not a GF bug, but a bug in JDK libraries which will be fixes in 7u25 and above. Since GF 4.0 will be certified with 7u21, this bug should be documented in the Java EE 7 RI/SDK release notes.
        Hide
        Gail Risdal added a comment -

        Added the following to the release notes:

        Deadlock while running quicklook tests (test_gd_security profile) (20582)

        Description
        GlassFish Server 4.0 is certified with Java HotSpot 7u21. Due to an issue with the JDK logging libraries, JDK logging code could deadlock in certain cases. This issue has been fixed in Java HotSpot 7u25 and later.

        Workaround
        If you encounter this issue, use Java HotSpot 7u25 or later.
        For the complete report about this issue, see GLASSFISH-20582 (http://java.net/jira/browse/GLASSFISH-20582).

        Show
        Gail Risdal added a comment - Added the following to the release notes: Deadlock while running quicklook tests (test_gd_security profile) (20582) Description GlassFish Server 4.0 is certified with Java HotSpot 7u21. Due to an issue with the JDK logging libraries, JDK logging code could deadlock in certain cases. This issue has been fixed in Java HotSpot 7u25 and later. Workaround If you encounter this issue, use Java HotSpot 7u25 or later. For the complete report about this issue, see GLASSFISH-20582 ( http://java.net/jira/browse/GLASSFISH-20582 ).

          People

          • Assignee:
            rajendra_inamdar
            Reporter:
            Lukas Jungmann
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: