Performance regression in server startup (GLASSFISH-16460)

[GLASSFISH-20206] Boot GlassFish startup services in parallel Created: 05/Apr/13  Updated: 19/Sep/14  Resolved: 18/Jun/13

Status: Closed
Project: glassfish
Component/s: performance
Affects Version/s: None
Fix Version/s: 4.1

Type: Sub-task Priority: Major
Reporter: jwells Assignee: jwells
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
depends on GLASSFISH-20300 config modularity write transactions ... Resolved
depends on HK2-98 The Run Level Service could start ser... Resolved
blocks GLASSFISH-18941 [PERF] additional regression in start... Open
blocks GLASSFISH-20299 ResourceManager run level 2 slows dow... Reopened
blocks GLASSFISH-16460 Performance regression in server startup Resolved
Tags: 4_0-approved, devx_web


We could boot GlassFish startup services in parallel to get better startup performance

Comment by Darious3 [ 05/Apr/13 ]

This would be the most welcome feature ever!

Once GlassFish was said to be the fastest starting server, but on my system JBoss starts faster. Don't they startup stuff in parallel as well?

Comment by TangYong [ 06/Apr/13 ]


I also agree with you that the feature is very cool!
However, as far as I know, JBOSS 7's desgin architecture has a litter different from glassfish although it is based on module architecture too. JBOSS 7's core design philosophy is Module Subsystem and Modular Service Container, seeing [1]. By seeing the following,

"JBoss AS 7 on the other hand starts and deploys all services in parallel. This complex problem is solved by our new service container, the JBoss Modular Service Container. MSC is essentially an advanced concurrent state machine. It analyzes the dependencies between all services on the fly and attempts to start as many as it can at the same time, while still adhering to the relationship requirements. This means that not only is startup blazing fast, but you also now can deploy multiple deployments in parallel."

To some extent, HK2 is similar to Modular Service Container in addition that hk2 implements jsr330.

So, we should believe that implementing "boot GlassFish startup services in parallel" must improve gf startup performance greatly!

In the other hand,

>Once GlassFish was said to be the fastest starting server
Objectively speaking, apart from improving hk2 layer, we can not ignore the following,

We have seen that JBOSS's Module Subsystem is a relatively advanced thing and if you want to use jboss7 cli, you will find that in order to use it, you must use the following,

2)jboss-cli.bat --connect

Noting that after you execute 1), you can use jboss gui not starting cli subsystem. Thus, 1)'s startup perfermance will be very fast because it needs not to start other subsystems at this time point.

Backing to glassfish, after starting glassfish, admin cli has been available.

So, maybe this is different desgin philosophy between gf and jboss, however, this has some impacts on what you said "GlassFish was said to be the fastest starting server" ,

In any way, I also wish gf to be the fastest starting server in the future!



Comment by jwells [ 08/Apr/13 ]

HK2 is an implementation of JSR-330, but it has many other features. One of them is the RunLevelService, which allows certain services to be part of a boot or shutdown sequence. The underlying technology now allows for these boot services to be done in parallel. However, if any service depends on another service (like via @Inject) then those will be started serially.

GlassFish itself has about 45 startup services, and another good thing about getting this functionality in is that we can now start to analyze how those services work when booting in parallel to further help boot performance.

Comment by jwells [ 10/Apr/13 ]

What is the impact on the customer of the bug?

They will see a performance improvement during startup. They will also be able to configure the number of threads used during the parallel boot of services.

What is the cost/risk of fixing the bug?

The actual change to GlassFish code is not large, but I'd consider this change to be fairly high risk, as code that did not previously run on separate threads will now run on separate threads. We have done a lot of testing, but multi-threaded issues can sometimes only be found with soak tests as opposed to devtests.

Is there an impact on documentation or message strings?


Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

EJB, Admin

Which is the targeted build of 4.0 for this fix?


If this an integration of a new version of a component from another project,
what are the changes that are being brought in? This might be list of
Jira issues from that project or a list of revision messages.

Comment by Tom Mueller [ 10/Apr/13 ]

Approved for 4.0.

Comment by jwells [ 12/Apr/13 ]

Change 61389 uptook hk2 and modified AppServerStartup to use the 4 threads by default for startup services. Experiments with Oracle Solaris Studio have shown we get pretty good parallelism (though it could be better in some areas).

Comment by jwells [ 12/Apr/13 ]

Was backed out with 61391 due to upstream Hudson failures. Investigating...

Comment by jwells [ 12/Apr/13 ]

The multithreaded boot code is in again at 61406, but the default is currently set to USE_NO_THREADS. So to turn on the use of parallel boot services you must set the java system properly

As with the previous checkin, you can control the number of threads it will use with

The default for that parameter is 4. If you set it < 1 you will get infinite maximum threads

Comment by jwells [ 13/Apr/13 ]

This was backed out again due to a discovered deadlock in hk2, which has subsequently been fixed. It has been put back in at change 61417.

The names of the parameters controlling the behavior have also been changed and are now


Comment by jwells [ 15/Apr/13 ]

The default was put back to using multiple threads in the server (with a maximum number of threads at 4) at change 61427

Comment by jwells [ 16/Apr/13 ]

There are two deadlocks that have been found. Backing out the default change until the deadlocks can be fixed.

Comment by David Zhao [ 22/Apr/13 ]

Revision 61428 causes WARNING messages logged in server.log, which is a regression.

It is easy to reproduce:

1. start glassfish
2. asadmin jms-ping --target server
3. shut down glassfish

Then WARNING appears in server.log,

[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [] [] [tid: _ThreadID=133 _ThreadName=Thread-29] [timeMillis: 1366600428973] [levelValue: 900] [[
Error occurs when shutting down JMSRA: Service was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(

loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)

[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [ra.stop.failed] [] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428973] [levelValue: 900] [[
RAR8053: RA [ jmsra ] stop failed, java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Service was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(







loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)

[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [INFO] [] [] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 800] [[
shutdown of RA [ jmsra ] is either already complete or already cancelled]]

[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [WARNING] [ra.stop-unsuccessful] [] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 900] [[
RAR7095: jmsra shutdown unsuccessful. Please refer the server and/or resource adapter logs for more information.]]

Comment by jwells [ 24/Apr/13 ]

Backing it out... again.

Default policy is USE_NO_THREADS at change 61605

Comment by jwells [ 18/Jun/13 ]

We are currently running startup services in parallel. If bugs/deadlocks are found please enter individual JIRA issues

Generated at Sun Feb 26 01:37:26 UTC 2017 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.