glassfish
  1. glassfish
  2. GLASSFISH-16560

very slow EAR deployment (annotation scanning seems to be very slow)

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1
    • Fix Version/s: future release
    • Component/s: deployment
    • Labels:
      None
    • Environment:

      Independent of OS and HW. Seen on Linux and Windows, 32 and 64 bit.

      Description

      Using the following EAR file:

      18 jar files with 383 Stateless EJB's an 65 MDB in the root directory of the EAR archive.
      19 war files with 66 Servlets in the root directory of the EAR archive.
      152 jar files in the ./lib directory of the EAR file.
      Size of EAR file is 75 MByte.

      The EAR file deploys in GF 2.1.1 in 40 seconds. With GF 3.0, 3.1 (b43) and 3.2-SNAPSHOT it deploys in >400 seconds. So it deploys 10 times slower than in GF 2.1.1.

      We use a lot of self developed annotations (>70) which reside in jar files in the ./lib directory of the EAR file.

      The profiling shows us that the time gets consumed in the method com.sun.enterprise.deployment.annotation.impl.ModuleScanner.addScanURI(). This method is called from the method addLibraryJars() in the same class. It seems, that for all found annotation, the whole amount of jar files get always scanned for annotaions. This task takes very long time and is very cpu intensive. In my profile sample, the method addScanURI() is called many thousand times from the method addLibraryJars(). I think the result of already scanned classes for annotations should be remembered and reused.

      I've attached a JVisualVM profiling file.

      1. gf31_slow_ear_deployment.nps
        48 kB
        fmeili
      2. prof_with_dol_patch.nps
        55 kB
        fmeili

        Activity

        Hide
        Hong Zhang added a comment -

        I just attached the patch in the issue. I think it should match with that 3.1.1 build, but if you see any problem, I can provide a patch for the 3.2 too (I plan to backport the change there too). You could do a baseline measurement before you install the patch. And then reinstall the build, and install the patch and see what difference it makes.

        Show
        Hong Zhang added a comment - I just attached the patch in the issue. I think it should match with that 3.1.1 build, but if you see any problem, I can provide a patch for the 3.2 too (I plan to backport the change there too). You could do a baseline measurement before you install the patch. And then reinstall the build, and install the patch and see what difference it makes.
        Hide
        fmeili added a comment -

        I've done some tests with the patch you've provided. The deployment time with the dol patch was shrinking from 400 seconds to 170 seconds, which is a big improvement.

        Glassfish 3.1.1 b04 original: 400 s
        Glassfish 3.1.1 with dol patch: 170 s
        Glassfish 2.1.1 b31g-fcs: 40 s

        Now the EAR deployment is not longer 10 times slower compared with GF 2.1.1, but is still 4 times slower. I've done some profiling with the dol patch, but sadly I couldn't find an additional single CPU hot spot (like in the version without the dol patch). As far as I can see, the most CPU time get consumed in annotation processing, but not at in a single method. There are a bunch of methods involved. I've attached a profiler snapshot with dol patch active. Maybe you can find something else which may be problematic while deploying.

        Thank for your fast response and patch.
        Greetings, Frank

        Show
        fmeili added a comment - I've done some tests with the patch you've provided. The deployment time with the dol patch was shrinking from 400 seconds to 170 seconds, which is a big improvement. Glassfish 3.1.1 b04 original: 400 s Glassfish 3.1.1 with dol patch: 170 s Glassfish 2.1.1 b31g-fcs: 40 s Now the EAR deployment is not longer 10 times slower compared with GF 2.1.1, but is still 4 times slower. I've done some profiling with the dol patch, but sadly I couldn't find an additional single CPU hot spot (like in the version without the dol patch). As far as I can see, the most CPU time get consumed in annotation processing, but not at in a single method. There are a bunch of methods involved. I've attached a profiler snapshot with dol patch active. Maybe you can find something else which may be problematic while deploying. Thank for your fast response and patch. Greetings, Frank
        Hide
        Hong Zhang added a comment -

        Frank thanks for verifying the patch! Glad this patch had made big improvement (and thanks again for reporting this and providing the profiling data pointing us to where we could improve significantly). I have checked in the changes into both 3.1.1 branch and the trunk (3.2) now.

        For the next step, I will probably take a look to see if we can save the processed results for the libraries and re-use it for each module instead of doing this for each module. But please keep in mind, as we are now additionally processing these library jars while we didn't in 2.1, there will be always time differences compared to 2.1 (especially when the application contains large amount of library jars).

        Show
        Hong Zhang added a comment - Frank thanks for verifying the patch! Glad this patch had made big improvement (and thanks again for reporting this and providing the profiling data pointing us to where we could improve significantly). I have checked in the changes into both 3.1.1 branch and the trunk (3.2) now. For the next step, I will probably take a look to see if we can save the processed results for the libraries and re-use it for each module instead of doing this for each module. But please keep in mind, as we are now additionally processing these library jars while we didn't in 2.1, there will be always time differences compared to 2.1 (especially when the application contains large amount of library jars).
        Hide
        Hong Zhang added a comment -

        Could not find any more simple set of the changes which could improve significantly. I am keeping this as a RFE for improvement in future release.

        Show
        Hong Zhang added a comment - Could not find any more simple set of the changes which could improve significantly. I am keeping this as a RFE for improvement in future release.
        Hide
        Hong Zhang added a comment -

        Scrubbing RFEs for GlassFish 4.0.

        Show
        Hong Zhang added a comment - Scrubbing RFEs for GlassFish 4.0.

          People

          • Assignee:
            Hong Zhang
            Reporter:
            fmeili
          • Votes:
            9 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated: