glassfish
  1. glassfish
  2. GLASSFISH-13277

Monitoring bootstrap start-up performance problem

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1
    • Fix Version/s: 3.1_b25
    • Component/s: monitoring
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • Issuezilla Id:
      13,277

      Description

      Monitoring bootstrap code is probing any directory under
      lib/install/applications to resource adapters that may or may not contain
      flashlight probes. This cause the connector class loader to be created and cause
      a long list of modules to be resolved.

      Solution : we should only probe RAs when they are deployed (used) rather than at
      startup.

      Possible extended solution : we should extend that mechanism to all deployments
      so that any Java EE deployed application can ship probes and they would become
      automatically registered with our monitoring/REST tree. I am already drooling at
      the possibilities...
      For that, it might be as simple as writing a ProbeDeployer.

        Issue Links

          Activity

          Hide
          Nazrul added a comment -

          We need a fix for this to make progress with GlassFish developer benchmark.

          Show
          Nazrul added a comment - We need a fix for this to make progress with GlassFish developer benchmark.
          Hide
          shreedhar_ganapathy added a comment -

          Mahesh is busy with HA work.
          Vince is beginning to look into this and working with Mahesh on this.

          Could we get some idea about the % or absolute numbers in terms of impact of the currrent issue, that the
          suggested solution is expected to overcome ?

          Show
          shreedhar_ganapathy added a comment - Mahesh is busy with HA work. Vince is beginning to look into this and working with Mahesh on this. Could we get some idea about the % or absolute numbers in terms of impact of the currrent issue, that the suggested solution is expected to overcome ?
          Hide
          Tom Mueller added a comment -

          The overall performance regression measurements are here:

          http://javaperf.sfbay.sun.com/JSPWiki/Wiki.jsp?page=V3.1StartupAndDeploymentPerformance

          I don't know if all accesses of the lib/install/applications directory are due to the monitoring
          bootstrap code, but here are some numbers based on analyzing the truss output of a DAS start:

          1534 calls to system calls with lib/install/applications

          • 102 opens
          • 320 resolvepath
          • 1112 stat64

          Based on truss timings, the code surrounding these calls accounts for about 3.1 seconds of a 44
          second DAS startup time, or about 7%.

          More data is available upon request.

          Show
          Tom Mueller added a comment - The overall performance regression measurements are here: http://javaperf.sfbay.sun.com/JSPWiki/Wiki.jsp?page=V3.1StartupAndDeploymentPerformance I don't know if all accesses of the lib/install/applications directory are due to the monitoring bootstrap code, but here are some numbers based on analyzing the truss output of a DAS start: 1534 calls to system calls with lib/install/applications 102 opens 320 resolvepath 1112 stat64 Based on truss timings, the code surrounding these calls accounts for about 3.1 seconds of a 44 second DAS startup time, or about 7%. More data is available upon request.
          Hide
          vince kraemer added a comment -

          Created an attachment (id=5094)
          a first attempt at a fix. i am sure it could be improved

          Show
          vince kraemer added a comment - Created an attachment (id=5094) a first attempt at a fix. i am sure it could be improved
          Hide
          vince kraemer added a comment -

          note on patch: QL passes

          Show
          vince kraemer added a comment - note on patch: QL passes
          Hide
          vince kraemer added a comment -

          jagadish is making code changes that address this

          Show
          vince kraemer added a comment - jagadish is making code changes that address this
          Hide
          Jagadish added a comment -
          • Made changes such that ClassLoaderHierarchy is not injected in
            MonitoringBootstrap, removed using connector-class-loader.
          • Made changes in connector-runtime to register a system-RAR for probes only
            when the RAR is bootstrapped.

          This seems to give an improvement of around ~100-125 ms.
          without-fix : 4.66 seconds (5 times start-stop domain avg)
          with-fix : 4.53 seconds (5 times start-stop domain avg)

          FIX INFORMATION :
          svn log -v -r 41680
          https://glassfish-svn.dev.java.net/servlets/ReadMsg?list=commits&msgNo=24234

          Fix will be available in 3.1-b25 / 13th-Oct-2010 nightly onwards

          Show
          Jagadish added a comment - Made changes such that ClassLoaderHierarchy is not injected in MonitoringBootstrap, removed using connector-class-loader. Made changes in connector-runtime to register a system-RAR for probes only when the RAR is bootstrapped. This seems to give an improvement of around ~100-125 ms. without-fix : 4.66 seconds (5 times start-stop domain avg) with-fix : 4.53 seconds (5 times start-stop domain avg) FIX INFORMATION : svn log -v -r 41680 https://glassfish-svn.dev.java.net/servlets/ReadMsg?list=commits&msgNo=24234 Fix will be available in 3.1-b25 / 13th-Oct-2010 nightly onwards

            People

            • Assignee:
              Jagadish
              Reporter:
              Tom Mueller
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: