Issue Details (XML | Word | Printable)

Key: GLASSFISH-20206
Type: Sub-task Sub-task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: jwells
Reporter: jwells
Votes: 1
Watchers: 4
Operations

If you were logged in you would be able to see more operations.
glassfish
GLASSFISH-16460

Boot GlassFish startup services in parallel

Created: 05/Apr/13 07:59 PM   Updated: 18/Jun/13 10:07 AM   Resolved: 18/Jun/13 10:07 AM
Component/s: performance
Affects Version/s: None
Fix Version/s: 4.0.1

Time Tracking:
Not Specified

Issue Links:
Dependency

Tags: devx_web 4_0-approved
Participants: Darious3, David Zhao, jwells, TangYong and Tom Mueller


 Description  « Hide

We could boot GlassFish startup services in parallel to get better startup performance



Darious3 added a comment - 05/Apr/13 10:38 PM

This would be the most welcome feature ever!

Once GlassFish was said to be the fastest starting server, but on my system JBoss starts faster. Don't they startup stuff in parallel as well?


TangYong added a comment - 06/Apr/13 03:11 PM

Darious3,

I also agree with you that the feature is very cool!
However, as far as I know, JBOSS 7's desgin architecture has a litter different from glassfish although it is based on module architecture too. JBOSS 7's core design philosophy is Module Subsystem and Modular Service Container, seeing [1]. By seeing the following,

"JBoss AS 7 on the other hand starts and deploys all services in parallel. This complex problem is solved by our new service container, the JBoss Modular Service Container. MSC is essentially an advanced concurrent state machine. It analyzes the dependencies between all services on the fly and attempts to start as many as it can at the same time, while still adhering to the relationship requirements. This means that not only is startup blazing fast, but you also now can deploy multiple deployments in parallel."

To some extent, HK2 is similar to Modular Service Container in addition that hk2 implements jsr330.

So, we should believe that implementing "boot GlassFish startup services in parallel" must improve gf startup performance greatly!

In the other hand,

>Once GlassFish was said to be the fastest starting server
Objectively speaking, apart from improving hk2 layer, we can not ignore the following,

We have seen that JBOSS's Module Subsystem is a relatively advanced thing and if you want to use jboss7 cli, you will find that in order to use it, you must use the following,

1)standalone.bat
2)jboss-cli.bat --connect

Noting that after you execute 1), you can use jboss gui not starting cli subsystem. Thus, 1)'s startup perfermance will be very fast because it needs not to start other subsystems at this time point.

Backing to glassfish, after starting glassfish, admin cli has been available.

So, maybe this is different desgin philosophy between gf and jboss, however, this has some impacts on what you said "GlassFish was said to be the fastest starting server" ,

In any way, I also wish gf to be the fastest starting server in the future!

[1]: http://planet.jboss.org/post/why_is_jboss_as_7_so_fast

Thanks
--Tang


jwells added a comment - 08/Apr/13 06:42 PM

HK2 is an implementation of JSR-330, but it has many other features. One of them is the RunLevelService, which allows certain services to be part of a boot or shutdown sequence. The underlying technology now allows for these boot services to be done in parallel. However, if any service depends on another service (like via @Inject) then those will be started serially.

GlassFish itself has about 45 startup services, and another good thing about getting this functionality in is that we can now start to analyze how those services work when booting in parallel to further help boot performance.


jwells added a comment - 10/Apr/13 08:06 PM

What is the impact on the customer of the bug?

They will see a performance improvement during startup. They will also be able to configure the number of threads used during the parallel boot of services.

What is the cost/risk of fixing the bug?

The actual change to GlassFish code is not large, but I'd consider this change to be fairly high risk, as code that did not previously run on separate threads will now run on separate threads. We have done a lot of testing, but multi-threaded issues can sometimes only be found with soak tests as opposed to devtests.

Is there an impact on documentation or message strings?

No

Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

EJB, Admin

Which is the targeted build of 4.0 for this fix?

b88

If this an integration of a new version of a component from another project,
what are the changes that are being brought in? This might be list of
Jira issues from that project or a list of revision messages.

http://gf-hudson.us.oracle.com/hudson/job/promote-hk2/169/changes#detail0


Tom Mueller added a comment - 10/Apr/13 08:10 PM

Approved for 4.0.


jwells added a comment - 12/Apr/13 12:39 PM

Change 61389 uptook hk2 and modified AppServerStartup to use the 4 threads by default for startup services. Experiments with Oracle Solaris Studio have shown we get pretty good parallelism (though it could be better in some areas).


jwells added a comment - 12/Apr/13 02:10 PM

Was backed out with 61391 due to upstream Hudson failures. Investigating...


jwells added a comment - 12/Apr/13 08:34 PM

The multithreaded boot code is in again at 61406, but the default is currently set to USE_NO_THREADS. So to turn on the use of parallel boot services you must set the java system properly

com.oracle.glassfish.startupThreadPolicy=FULLY_THREADED

As with the previous checkin, you can control the number of threads it will use with

com.oracle.glassfish.maxStartupThreads=4

The default for that parameter is 4. If you set it < 1 you will get infinite maximum threads


jwells added a comment - 13/Apr/13 07:08 PM

This was backed out again due to a discovered deadlock in hk2, which has subsequently been fixed. It has been put back in at change 61417.

The names of the parameters controlling the behavior have also been changed and are now

org.glassfish.startupThreadPolicy=[FULLY_THREADED|USE_NO_THREADS]
org.glassfish.maxStartupThreads=4


jwells added a comment - 15/Apr/13 03:40 PM

The default was put back to using multiple threads in the server (with a maximum number of threads at 4) at change 61427


jwells added a comment - 16/Apr/13 12:33 PM

There are two deadlocks that have been found. Backing out the default change until the deadlocks can be fixed.


David Zhao added a comment - 22/Apr/13 04:39 AM

Revision 61428 causes WARNING messages logged in server.log, which is a regression.

It is easy to reproduce:

1. start glassfish
2. asadmin jms-ping --target server
3. shut down glassfish

Then WARNING appears in server.log,

[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [] [javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system] [tid: _ThreadID=133 _ThreadName=Thread-29] [timeMillis: 1366600428973] [levelValue: 900] [[
Error occurs when shutting down JMSRA: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(
implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher}
scope=org.glassfish.hk2.runlevel.RunLevel
qualifiers={}
descriptorType=CLASS
descriptorVisibility=NORMAL
metadata=runLevelValue={10},Bundle-SymbolicName={org.glassfish.main.core.kernel},Bundle-Version={4.0.0.SNAPSHOT}
rank=50
loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)
proxiable=null
analysisName=null
id=690
locatorId=0
identityHashCode=16925195
reified=true)]]

[2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [ra.stop.failed] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428973] [levelValue: 900] [[
RAR8053: RA [ jmsra ] stop failed, java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(
implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher}
scope=org.glassfish.hk2.runlevel.RunLevel
qualifiers={}
descriptorType=CLASS
descriptorVisibility=NORMAL
metadata=runLevelValue={10},Bundle-SymbolicName={org.glassfish.main.core.kernel},Bundle-Version={4.0.0.SNAPSHOT}
rank=50
loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)
proxiable=null
analysisName=null
id=690
locatorId=0
identityHashCode=16925195
reified=true)]]

[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [INFO] [] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 800] [[
shutdown of RA [ jmsra ] is either already complete or already cancelled]]

[2013-04-22T11:13:48.974+0800] [glassfish 4.0] [WARNING] [ra.stop-unsuccessful] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 900] [[
RAR7095: jmsra shutdown unsuccessful. Please refer the server and/or resource adapter logs for more information.]]


jwells added a comment - 24/Apr/13 02:22 AM

Backing it out... again.

Default policy is USE_NO_THREADS at change 61605


jwells added a comment - 18/Jun/13 10:07 AM

We are currently running startup services in parallel. If bugs/deadlocks are found please enter individual JIRA issues