Performance regression in server startup
[GLASSFISH-20206] Boot GlassFish startup services in parallel Created: 05/Apr/13 Updated: 19/Sep/14 Resolved: 18/Jun/13
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
We could boot GlassFish startup services in parallel to get better startup performance
|Comment by Darious3 [ 05/Apr/13 ]|
This would be the most welcome feature ever!
Once GlassFish was said to be the fastest starting server, but on my system JBoss starts faster. Don't they startup stuff in parallel as well?
|Comment by TangYong [ 06/Apr/13 ]|
I also agree with you that the feature is very cool!
"JBoss AS 7 on the other hand starts and deploys all services in parallel. This complex problem is solved by our new service container, the JBoss Modular Service Container. MSC is essentially an advanced concurrent state machine. It analyzes the dependencies between all services on the fly and attempts to start as many as it can at the same time, while still adhering to the relationship requirements. This means that not only is startup blazing fast, but you also now can deploy multiple deployments in parallel."
To some extent, HK2 is similar to Modular Service Container in addition that hk2 implements jsr330.
So, we should believe that implementing "boot GlassFish startup services in parallel" must improve gf startup performance greatly!
In the other hand,
>Once GlassFish was said to be the fastest starting server
We have seen that JBOSS's Module Subsystem is a relatively advanced thing and if you want to use jboss7 cli, you will find that in order to use it, you must use the following,
Noting that after you execute 1), you can use jboss gui not starting cli subsystem. Thus, 1)'s startup perfermance will be very fast because it needs not to start other subsystems at this time point.
Backing to glassfish, after starting glassfish, admin cli has been available.
So, maybe this is different desgin philosophy between gf and jboss, however, this has some impacts on what you said "GlassFish was said to be the fastest starting server" ,
In any way, I also wish gf to be the fastest starting server in the future!
|Comment by jwells [ 08/Apr/13 ]|
HK2 is an implementation of JSR-330, but it has many other features. One of them is the RunLevelService, which allows certain services to be part of a boot or shutdown sequence. The underlying technology now allows for these boot services to be done in parallel. However, if any service depends on another service (like via @Inject) then those will be started serially.
GlassFish itself has about 45 startup services, and another good thing about getting this functionality in is that we can now start to analyze how those services work when booting in parallel to further help boot performance.
|Comment by jwells [ 10/Apr/13 ]|
What is the impact on the customer of the bug?
They will see a performance improvement during startup. They will also be able to configure the number of threads used during the parallel boot of services.
What is the cost/risk of fixing the bug?
The actual change to GlassFish code is not large, but I'd consider this change to be fairly high risk, as code that did not previously run on separate threads will now run on separate threads. We have done a lot of testing, but multi-threaded issues can sometimes only be found with soak tests as opposed to devtests.
Is there an impact on documentation or message strings?
Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
Which is the targeted build of 4.0 for this fix?
If this an integration of a new version of a component from another project,
|Comment by Tom Mueller [ 10/Apr/13 ]|
Approved for 4.0.
|Comment by jwells [ 12/Apr/13 ]|
Change 61389 uptook hk2 and modified AppServerStartup to use the 4 threads by default for startup services. Experiments with Oracle Solaris Studio have shown we get pretty good parallelism (though it could be better in some areas).
|Comment by jwells [ 12/Apr/13 ]|
Was backed out with 61391 due to upstream Hudson failures. Investigating...
|Comment by jwells [ 12/Apr/13 ]|
The multithreaded boot code is in again at 61406, but the default is currently set to USE_NO_THREADS. So to turn on the use of parallel boot services you must set the java system properly
As with the previous checkin, you can control the number of threads it will use with
The default for that parameter is 4. If you set it < 1 you will get infinite maximum threads
|Comment by jwells [ 13/Apr/13 ]|
This was backed out again due to a discovered deadlock in hk2, which has subsequently been fixed. It has been put back in at change 61417.
The names of the parameters controlling the behavior have also been changed and are now
|Comment by jwells [ 15/Apr/13 ]|
The default was put back to using multiple threads in the server (with a maximum number of threads at 4) at change 61427
|Comment by jwells [ 16/Apr/13 ]|
There are two deadlocks that have been found. Backing out the default change until the deadlocks can be fixed.
|Comment by David Zhao [ 22/Apr/13 ]|
Revision 61428 causes WARNING messages logged in server.log, which is a regression.
It is easy to reproduce:
1. start glassfish
Then WARNING appears in server.log,
[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING]  [javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system] [tid: _ThreadID=133 _ThreadName=Thread-29] [timeMillis: 1366600428973] [levelValue: 900] [[
loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel ], State = [READY],9394331)
[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [ra.stop.failed] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428973] [levelValue: 900] [[
RAR8053: RA [ jmsra ] stop failed, java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(
[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [INFO]  [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 800] [[
[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [WARNING] [ra.stop-unsuccessful] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 900] [[
|Comment by jwells [ 24/Apr/13 ]|
Backing it out... again.
Default policy is USE_NO_THREADS at change 61605
|Comment by jwells [ 18/Jun/13 ]|
We are currently running startup services in parallel. If bugs/deadlocks are found please enter individual JIRA issues