[GLASSFISH-19940] Investigate modifying META-INF/services handling to reduce performance regression Created: 19/Mar/13  Updated: 22/Apr/13  Resolved: 22/Apr/13

Status: Closed
Project: glassfish
Component/s: hk2
Affects Version/s: 4.0_b79
Fix Version/s: 4.0

Type: Task Priority: Critical
Reporter: Tom Mueller Assignee: mtaube
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Tags: devx_web

 Description   

Part of the startup time for GlassFish is to read in the META-INF/services/* files from the OSGi bundles (76 out of the 289 have these) and then to consult the content of these files during certain class loader operations. Since HK2 doesn't use these services files, it is not clear that we need to spend this time during startup. Eliminating this processing may help reduce the performance regression for GF 4.0.

Here is some data.

From the profile, the loadMetadata method spends about half of its time loading the service descriptors. This shows that there are 76 descriptors read from 289 bundles. Only 34 of the bundles have any descriptors at all.

3.0% - 962 ms - 289 inv. org.jvnet.hk2.osgiadapter.OSGiModuleDefinition$BundleJar.loadMetadata
-1.6% - 527 ms - 289 inv. org.jvnet.hk2.osgiadapter.OSGiModuleDefinition$BundleJar.parseServiceDescriptors
--1.0% - 308 ms - 34 inv. org.osgi.framework.Bundle.getEntryPaths
--0.6% - 196 ms - 76 inv. com.sun.enterprise.module.ModuleMetadata.load
--0.0% - 11,643 µs - 365 inv. org.osgi.framework.Bundle.getEntry
--0.0% - 9,557 µs - 76 inv. java.net.URL.openStream
--0.0% - 1,329 µs - 76 inv. java.util.Enumeration.nextElement
--0.0% - 51 µs - 76 inv. java.lang.String.substring
--0.0% - 4 µs - 110 inv. java.util.Enumeration.hasMoreElements
--0.0% - 1 µs - 76 inv. java.io.InputStream.close

The OSGiModuleDefinition$BundleJar.parseServiceDescriptors method reads these files into a ModuleMetadata class. The ModuleMetadata is also written by AbstractModulesRegistryImpl.add (via OSGIFactoryImpl.createModulesRegistry).

The ModuleMetadata services entries information is read during startup by APIClassLoaderServiceImpl$APIClassLoader to resolve JAR Service references for the following classes:

javax.xml.stream.XMLInputFactory (* this one is found in the woodstox-core-asl.jar file)
javax.ws.rs.ext.MessageBodyReader
javax.json.spi.JsonProvider
javax.xml.parsers.DocumentBuilderFactory
javax.xml.transform.TransformerFactory
com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager

Only the first one (as indicated) is actually found in any of the GlassFish OSGi bundles.

Maybe the annotation processor that generates the services entries for GlassFish could be modified so that entries are not generated for @Contract implementations. This would reduce the number of JARs that actually have services. Also, the APIClassLoader class could be improved so that can do a more efficient lookup for META-INF/services references (rather than looping through all 289 bundles).

The purpose of this issue to to investigate whether a performance improvement is possible. If it is possible, then this issue should be changed to a "bug" to reflect that changes actually need to be made to the system. The changes would be needed for 4.0 to meet the release criteria for this benchmark.



 Comments   
Comment by jwells [ 19/Mar/13 ]

So I don't think this comment is accurate:

Maybe the annotation processor that generates the services entries for GlassFish could be modified so that entries are not generated for @Contract implementations.

I do not believe any META-INF/services are being generated specifically for hk2 contracts. A list of files under META-INF/services (glassfish/modules directory) is below. There are a few things HK2 related (PopulatorPostProcessor etc) but those things are needed there as they have to be there before hk2 services are populated...

batch-config.properties
batch-services.properties
com.ibm.jbatch.tck.spi.BatchContainerServiceProvider
com.sun.enterprise.admin.util.cache.DataProvider
com.sun.enterprise.ee.cms.core.GroupManagementService
com.sun.enterprise.mgmt.transport.NetworkManager
com.sun.faces.spi.FacesConfigResourceProvider
com.sun.faces.spi.injectionprovider
com.sun.faces.spi.injectionprovider
com.sun.security.sasl.preview.SaslClientFactory
com.sun.tools.xjc.Plugin
com.sun.xml.ws.api.config.management.ManagedEndpointFactory
com.sun.xml.ws.api.pipe.TubelineAssemblerFactory
com.sun.xml.ws.api.policy.PolicyResolverFactory
com.sun.xml.ws.api.server.EndpointReferenceExtensionContributor
com.sun.xml.ws.api.wsdl.parser.MetadataResolverFactory
com.sun.xml.ws.policy.jaxws.spi.PolicyFeatureConfigurator
com.sun.xml.ws.policy.jaxws.spi.PolicyMapConfigurator
com.sun.xml.ws.policy.spi.PolicyAssertionCreator
com.sun.xml.ws.policy.spi.PolicyAssertionValidator
com.sun.xml.ws.policy.spi.PrefixMapper
com.sun.xml.ws.security.spi.AlternativeSelector
com.sun.xml.ws.spi.db.BindingContextFactory
faces-config.xml
javax.annotation.processing.Processor
javax.batch.operations.JobOperator
javax.ejb.spi.EJBContainerProvider
javax.enterprise.inject.spi.CDIProvider
javax.enterprise.inject.spi.Extension
javax.enterprise.inject.spi.Extension
javax.enterprise.inject.spi.Extension
javax.enterprise.inject.spi.Extension
javax.enterprise.inject.spi.Extension
javax.enterprise.inject.spi.Extension
javax.management.remote.JMXConnectorProvider
javax.management.remote.JMXConnectorServerProvider
javax.persistence.spi.PersistenceProvider
javax.servlet.ServletContainerInitializer
javax.servlet.ServletContainerInitializer
javax.servlet.ServletContainerInitializer
javax.servlet.ServletContainerInitializer
javax.servlet.ServletContainerInitializer
javax.servlet.ServletContainerInitializer
javax.validation.spi.ValidationProvider
javax.websocket.ContainerProvider
javax.websocket.server.ServerContainerProvider
javax.websocket.server.ServerEndpointConfig$Configurator
javax.ws.rs.ext.RuntimeDelegate
javax.xml.bind.JAXBContext
javax.xml.soap.MessageFactory
javax.xml.soap.MetaFactory
javax.xml.soap.SOAPConnectionFactory
javax.xml.soap.SOAPFactory
javax.xml.stream.XMLEventFactory
javax.xml.stream.XMLInputFactory
javax.xml.stream.XMLOutputFactory
javax.xml.ws.spi.Provider
org.codehaus.stax2.validation.XMLValidationSchemaFactory.dtd
org.codehaus.stax2.validation.XMLValidationSchemaFactory.relaxng
org.codehaus.stax2.validation.XMLValidationSchemaFactory.w3c
org.glassfish.embeddable.spi.RuntimeBuilder
org.glassfish.faces.integration.GlassFishInjectionProvider
org.glassfish.hk2.bootstrap.PopulatorPostProcessor
org.glassfish.hk2.bootstrap.PopulatorPostProcessor
org.glassfish.hk2.bootstrap.PopulatorPostProcessor
org.glassfish.hk2.extension.ServiceLocatorGenerator
org.glassfish.jersey.internal.spi.AutoDiscoverable
org.glassfish.jersey.internal.spi.AutoDiscoverable
org.glassfish.jersey.internal.spi.AutoDiscoverable
org.glassfish.jersey.server.spi.ComponentProvider
org.glassfish.jersey.server.spi.ContainerProvider
org.glassfish.jersey.servlet.spi.AsyncContextDelegateProvider
org.glassfish.tyrus.spi.ComponentProvider
org.iso_relax.verifier.VerifierFactoryLoader
org.relaxng.datatype.DatatypeLibraryFactory

Comment by Sanjeeb Sahoo [ 19/Mar/13 ]

Looks like a regression during HK2 2.x effort. The metadata should be coming from the cache as opposed to from bundles during subsequent restart.

Comment by mtaube [ 19/Mar/13 ]

After look at this I agree with Sahoo, if ModuleMetadata is present in the cache the call to loadMetadata should not be happening. I am investigating this.

Comment by Tom Mueller [ 19/Mar/13 ]

I added some timing statements within the APIClassLoader.getResource where it looks through all of the modules. Here are the results from a server startup:

getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1120
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1285
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1006
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1240
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 860
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 825
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 866
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 868
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1393
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1059
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 878
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1231
getResource: META-INF/services/javax.xml.stream.XMLInputFactory 1339
getResource: META-INF/services/javax.xml.parsers.DocumentBuilderFactory 1800
getResource: META-INF/services/javax.ws.rs.ext.RuntimeDelegate 1494
getResource: META-INF/services/javax.xml.transform.TransformerFactory 1682
getResource: META-INF/services/com.sun.org.apache.xalan.internal.xsltc.dom.XSLTCDTMManager 1172
getResource: META-INF/services/javax.xml.stream.XMLOutputFactory 875

The third column is the time taken in usec. The total time is 21 msec. Given that the total server startup time on the same system is almost 4000 ms, this is probably not work fixing.

Comment by Tom Mueller [ 19/Mar/13 ]

Question: some bundles include a services file manually. (Sorry for the mistake about the annotation process). For example, the nucleus/admin/util module has a services file here:

main/nucleus/admin/util/src/main/resources/META-INF/services/com.sun.enterprise.admin.util.cache.DataProvider:
com.sun.enterprise.admin.util.cache.StringDataProvider
com.sun.enterprise.admin.util.cache.ByteArrayDataProvider
com.sun.enterprise.admin.util.cache.CommandModelDataProvider

Is this file needed? Based on looking at the code that uses DataProvider, it appears that it isn't needed anymore.

Are any of the services files created by GlassFish needed?
If this is a case-by-case situation, I don't know if it is worth it to go through each and every JAR to see if these are needed.

Comment by mtaube [ 19/Mar/13 ]

WRT caching, Tom confirmed that the loadMetadata call happens on a cold boot of a domain but is avoided on subsequent starts (when there are cache hits).

Comment by Sanjeeb Sahoo [ 21/Mar/13 ]

Since cache is consulted during subsequent server startup even for META-INF/services file, why is this bug still open?

Comment by Tom Mueller [ 21/Mar/13 ]

One reason is that we still may have META-INF/services files being included in GlassFish JARs that are not necessary. For example, admin-util.jar had one, but Martin has recently removed it. At the time I created this issue, there were 34 JAR files that have 76 different META-INF/services files. John provided the list of the META-INF/services files above, but not the list of JAR files that contain them. Here's the list:

autostart/osgi-cdi.jar
bean-validator.jar
com.ibm.jbatch-runtime-all.jar
ejb-container.jar
gf-jms-injection.jar
glassfish.jar
hk2-locator.jar
javax.faces.jar
javax.servlet.jsp.jar
jaxb-extra-osgi.jar
jaxb-osgi.jar
jersey-bean-validation.jar
jersey-container-grizzly2-http.jar
jersey-container-servlet.jar
jersey-gf-ejb.jar
jersey-media-json-processing.jar
jersey-server.jar
jmxremote_optional-repackaged.jar
jsf-connector.jar
kernel.jar
ldapbp-repackaged.jar
org.eclipse.persistence.jpa.jar
rest-service.jar
shoal-gms-impl.jar
tyrus-client.jar
tyrus-container-glassfish-cdi.jar
tyrus-container-servlet.jar
tyrus-server.jar
web-sse.jar
webservices-osgi.jar
weld-integration-fragment.jar
weld-integration.jar
woodstox-core-asl.jar

Many of these come from other projects, so we may not be able to eliminate the services files there, but some such as glassfish.jar, kernel.jar and ejb-container.jar probably do not need them.

It is not clear that there is a significant performance gain by fixing this, but we may get to the point where every little bit counts.

Regarding kernel.jar, the source code doesn't have any META-INF/services file, but one is generated. The content is:

$ cat META-INF/services/org.glassfish.hk2.bootstrap.PopulatorPostProcessor
org.glassfish.kernel.embedded.EmbeddedInhabitantsParser

The EmbeddedInhabitantsParser class has the @MetaInfServices annotation which causes the services file to be generated. Does the HK2 PopulatorPostProcessor contract use the JAR service locator mechanism, or does it use hk2-locator information? If it is the latter, then these annotations are not necessary.

The @MetaInfServices annotation is used in three GF JAR files (kernel.jar, glassfish.jar, and rest-service.jar).

Comment by Sanjeeb Sahoo [ 21/Mar/13 ]

No of Java EE technologies still rely on META-INF/services mechanism for them to be plugged into the appserver runtime. e.g., ServletContainerInitializers, Various XML Parser factories, persistence providers, etc. So, it would be very difficult to change them.

glassfish.jar uses META-INF/services to look up GlassFishRuntimeBuilders. So that file will remain that way.

I am not sure about kernel.jar or rest-service.jar.

Generated at Sat Sep 05 17:30:01 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.