Using the following EAR file:
18 jar files with 383 Stateless EJB's an 65 MDB in the root directory of the EAR archive.
19 war files with 66 Servlets in the root directory of the EAR archive.
152 jar files in the ./lib directory of the EAR file.
Size of EAR file is 75 MByte.
The EAR file deploys in GF 2.1.1 in 40 seconds. With GF 3.0, 3.1 (b43) and 3.2-SNAPSHOT it deploys in >400 seconds. So it deploys 10 times slower than in GF 2.1.1.
We use a lot of self developed annotations (>70) which reside in jar files in the ./lib directory of the EAR file.
The profiling shows us that the time gets consumed in the method com.sun.enterprise.deployment.annotation.impl.ModuleScanner.addScanURI(). This method is called from the method addLibraryJars() in the same class. It seems, that for all found annotation, the whole amount of jar files get always scanned for annotaions. This task takes very long time and is very cpu intensive. In my profile sample, the method addScanURI() is called many thousand times from the method addLibraryJars(). I think the result of already scanned classes for annotations should be remembered and reused.
I've attached a JVisualVM profiling file.