glassfish
  1. glassfish
  2. GLASSFISH-18699

Thread hang at ConcurrencyManager.releaseDeferredLock()

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Blocker Blocker
    • Resolution: Unresolved
    • Affects Version/s: v2.1
    • Fix Version/s: None
    • Component/s: entity-persistence
    • Labels:
      None
    • Environment:

      SPARC Solaris 5.8, Java(TM) SE Runtime Environment (build 1.6.0_30-b12), Toplink Essentials 2.1-b31g-fcs (10/19/2009), Glassfish v2.1-b60e-sunos

      Description

      We have a solution running in the client's Production environment. Under heavy load, the entire Glassfish Application Server instance will come to a halt with > 1000 threads in TIMED_WAITING (sleeping) state. The solution we have at the moment is to restart the entire application server which is a major problem in a Production environment.

      Symptoms of the problem:
      1) With nightly Glassfish restart:
      a) Thread hang happens once every fortnight, or even up to 3 weeks.
      2) Without nightly Glassfish restart:
      a) Thread hang happens minimally twice a day, which translates to twice downtime during business hours
      3) When system comes to a halt, a JVM head dump shows many threads of the following (with stacktrace):

      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at oracle.toplink.essentials.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:429)
      at oracle.toplink.essentials.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:373)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:614)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:451)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:421)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:387)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.processItem(ReportQueryResult.java:220)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.buildResult(ReportQueryResult.java:182)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.<init>(ReportQueryResult.java:98)
      at oracle.toplink.essentials.queryframework.ReportQuery.buildObject(ReportQuery.java:594)
      at oracle.toplink.essentials.queryframework.ReportQuery.buildObjects(ReportQuery.java:643)
      at oracle.toplink.essentials.queryframework.ReportQuery.executeDatabaseQuery(ReportQuery.java:804)
      at oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:628)
      at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:692)
      at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:746)
      at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2248)
      at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
      at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:924)
      at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:367)
      at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.getResultList(EJBQueryImpl.java:478)
      at com.sun.enterprise.util.QueryWrapper.getResultList(QueryWrapper.java:196)

      Information on Software components being used:
      1) Glassfish v2.1-b60e-sunos
      2) Toplink essentials in $GLASSFISH_HOME/lib - 2.1-b60e-fcs (12/23/2008)
      3) Toplink essentials packaged in our applications lib - 2.1-b31g-fcs (10/19/2009) <-- extracted from GFv2.1.1

      Workarounds we have tried:
      1) Referred to similar issue at http://java.net/jira/browse/GLASSFISH-4689
      2) Implemented a SessionCustomizer that sets setCacheTransactionIsolation to CONCURRENT_READ_WRITE, the thread hang still happens

      Would like to know if anyone is facing similar issue as above. We have so far been unable to re-produce in our test environments. We have also considered upgrading to EclipseLink but without the ability to re-produce we are rather sceptical that the problem would be addressed as we have no way to confirm it.

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            tware
            Reporter:
            kokwei
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: