Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1
    • Component/s: performance
    • Labels:
      None

      Description

      We could boot GlassFish startup services in parallel to get better startup performance

        Issue Links

          Activity

          Hide
          Darious3 added a comment -

          This would be the most welcome feature ever!

          Once GlassFish was said to be the fastest starting server, but on my system JBoss starts faster. Don't they startup stuff in parallel as well?

          Show
          Darious3 added a comment - This would be the most welcome feature ever! Once GlassFish was said to be the fastest starting server, but on my system JBoss starts faster. Don't they startup stuff in parallel as well?
          Hide
          TangYong added a comment -

          Darious3,

          I also agree with you that the feature is very cool!
          However, as far as I know, JBOSS 7's desgin architecture has a litter different from glassfish although it is based on module architecture too. JBOSS 7's core design philosophy is Module Subsystem and Modular Service Container, seeing [1]. By seeing the following,

          "JBoss AS 7 on the other hand starts and deploys all services in parallel. This complex problem is solved by our new service container, the JBoss Modular Service Container. MSC is essentially an advanced concurrent state machine. It analyzes the dependencies between all services on the fly and attempts to start as many as it can at the same time, while still adhering to the relationship requirements. This means that not only is startup blazing fast, but you also now can deploy multiple deployments in parallel."

          To some extent, HK2 is similar to Modular Service Container in addition that hk2 implements jsr330.

          So, we should believe that implementing "boot GlassFish startup services in parallel" must improve gf startup performance greatly!

          In the other hand,

          >Once GlassFish was said to be the fastest starting server
          Objectively speaking, apart from improving hk2 layer, we can not ignore the following,

          We have seen that JBOSS's Module Subsystem is a relatively advanced thing and if you want to use jboss7 cli, you will find that in order to use it, you must use the following,

          1)standalone.bat
          2)jboss-cli.bat --connect

          Noting that after you execute 1), you can use jboss gui not starting cli subsystem. Thus, 1)'s startup perfermance will be very fast because it needs not to start other subsystems at this time point.

          Backing to glassfish, after starting glassfish, admin cli has been available.

          So, maybe this is different desgin philosophy between gf and jboss, however, this has some impacts on what you said "GlassFish was said to be the fastest starting server" ,

          In any way, I also wish gf to be the fastest starting server in the future!

          [1]: http://planet.jboss.org/post/why_is_jboss_as_7_so_fast

          Thanks
          --Tang

          Show
          TangYong added a comment - Darious3, I also agree with you that the feature is very cool! However, as far as I know, JBOSS 7's desgin architecture has a litter different from glassfish although it is based on module architecture too. JBOSS 7's core design philosophy is Module Subsystem and Modular Service Container, seeing [1] . By seeing the following, "JBoss AS 7 on the other hand starts and deploys all services in parallel. This complex problem is solved by our new service container, the JBoss Modular Service Container. MSC is essentially an advanced concurrent state machine. It analyzes the dependencies between all services on the fly and attempts to start as many as it can at the same time, while still adhering to the relationship requirements. This means that not only is startup blazing fast, but you also now can deploy multiple deployments in parallel." To some extent, HK2 is similar to Modular Service Container in addition that hk2 implements jsr330. So, we should believe that implementing "boot GlassFish startup services in parallel" must improve gf startup performance greatly! In the other hand, >Once GlassFish was said to be the fastest starting server Objectively speaking, apart from improving hk2 layer, we can not ignore the following, We have seen that JBOSS's Module Subsystem is a relatively advanced thing and if you want to use jboss7 cli, you will find that in order to use it, you must use the following, 1)standalone.bat 2)jboss-cli.bat --connect Noting that after you execute 1), you can use jboss gui not starting cli subsystem. Thus, 1)'s startup perfermance will be very fast because it needs not to start other subsystems at this time point. Backing to glassfish, after starting glassfish, admin cli has been available. So, maybe this is different desgin philosophy between gf and jboss, however, this has some impacts on what you said "GlassFish was said to be the fastest starting server" , In any way, I also wish gf to be the fastest starting server in the future! [1] : http://planet.jboss.org/post/why_is_jboss_as_7_so_fast Thanks --Tang
          Hide
          jwells added a comment -

          HK2 is an implementation of JSR-330, but it has many other features. One of them is the RunLevelService, which allows certain services to be part of a boot or shutdown sequence. The underlying technology now allows for these boot services to be done in parallel. However, if any service depends on another service (like via @Inject) then those will be started serially.

          GlassFish itself has about 45 startup services, and another good thing about getting this functionality in is that we can now start to analyze how those services work when booting in parallel to further help boot performance.

          Show
          jwells added a comment - HK2 is an implementation of JSR-330, but it has many other features. One of them is the RunLevelService, which allows certain services to be part of a boot or shutdown sequence. The underlying technology now allows for these boot services to be done in parallel. However, if any service depends on another service (like via @Inject) then those will be started serially. GlassFish itself has about 45 startup services, and another good thing about getting this functionality in is that we can now start to analyze how those services work when booting in parallel to further help boot performance.
          Hide
          jwells added a comment -

          What is the impact on the customer of the bug?

          They will see a performance improvement during startup. They will also be able to configure the number of threads used during the parallel boot of services.

          What is the cost/risk of fixing the bug?

          The actual change to GlassFish code is not large, but I'd consider this change to be fairly high risk, as code that did not previously run on separate threads will now run on separate threads. We have done a lot of testing, but multi-threaded issues can sometimes only be found with soak tests as opposed to devtests.

          Is there an impact on documentation or message strings?

          No

          Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

          EJB, Admin

          Which is the targeted build of 4.0 for this fix?

          b88

          If this an integration of a new version of a component from another project,
          what are the changes that are being brought in? This might be list of
          Jira issues from that project or a list of revision messages.

          http://gf-hudson.us.oracle.com/hudson/job/promote-hk2/169/changes#detail0

          Show
          jwells added a comment - What is the impact on the customer of the bug? They will see a performance improvement during startup. They will also be able to configure the number of threads used during the parallel boot of services. What is the cost/risk of fixing the bug? The actual change to GlassFish code is not large, but I'd consider this change to be fairly high risk, as code that did not previously run on separate threads will now run on separate threads. We have done a lot of testing, but multi-threaded issues can sometimes only be found with soak tests as opposed to devtests. Is there an impact on documentation or message strings? No Which tests should QA (re)run to verify the fix did not destabilize GlassFish? EJB, Admin Which is the targeted build of 4.0 for this fix? b88 If this an integration of a new version of a component from another project, what are the changes that are being brought in? This might be list of Jira issues from that project or a list of revision messages. http://gf-hudson.us.oracle.com/hudson/job/promote-hk2/169/changes#detail0
          Hide
          Tom Mueller added a comment -

          Approved for 4.0.

          Show
          Tom Mueller added a comment - Approved for 4.0.
          Hide
          jwells added a comment -

          Change 61389 uptook hk2 and modified AppServerStartup to use the 4 threads by default for startup services. Experiments with Oracle Solaris Studio have shown we get pretty good parallelism (though it could be better in some areas).

          Show
          jwells added a comment - Change 61389 uptook hk2 and modified AppServerStartup to use the 4 threads by default for startup services. Experiments with Oracle Solaris Studio have shown we get pretty good parallelism (though it could be better in some areas).
          Hide
          jwells added a comment -

          Was backed out with 61391 due to upstream Hudson failures. Investigating...

          Show
          jwells added a comment - Was backed out with 61391 due to upstream Hudson failures. Investigating...
          Hide
          jwells added a comment -

          The multithreaded boot code is in again at 61406, but the default is currently set to USE_NO_THREADS. So to turn on the use of parallel boot services you must set the java system properly

          com.oracle.glassfish.startupThreadPolicy=FULLY_THREADED

          As with the previous checkin, you can control the number of threads it will use with

          com.oracle.glassfish.maxStartupThreads=4

          The default for that parameter is 4. If you set it < 1 you will get infinite maximum threads

          Show
          jwells added a comment - The multithreaded boot code is in again at 61406, but the default is currently set to USE_NO_THREADS. So to turn on the use of parallel boot services you must set the java system properly com.oracle.glassfish.startupThreadPolicy=FULLY_THREADED As with the previous checkin, you can control the number of threads it will use with com.oracle.glassfish.maxStartupThreads=4 The default for that parameter is 4. If you set it < 1 you will get infinite maximum threads
          Hide
          jwells added a comment -

          This was backed out again due to a discovered deadlock in hk2, which has subsequently been fixed. It has been put back in at change 61417.

          The names of the parameters controlling the behavior have also been changed and are now

          org.glassfish.startupThreadPolicy=[FULLY_THREADED|USE_NO_THREADS]
          org.glassfish.maxStartupThreads=4

          Show
          jwells added a comment - This was backed out again due to a discovered deadlock in hk2, which has subsequently been fixed. It has been put back in at change 61417. The names of the parameters controlling the behavior have also been changed and are now org.glassfish.startupThreadPolicy= [FULLY_THREADED|USE_NO_THREADS] org.glassfish.maxStartupThreads=4
          Hide
          jwells added a comment -

          The default was put back to using multiple threads in the server (with a maximum number of threads at 4) at change 61427

          Show
          jwells added a comment - The default was put back to using multiple threads in the server (with a maximum number of threads at 4) at change 61427
          Hide
          jwells added a comment -

          There are two deadlocks that have been found. Backing out the default change until the deadlocks can be fixed.

          Show
          jwells added a comment - There are two deadlocks that have been found. Backing out the default change until the deadlocks can be fixed.
          Hide
          David Zhao added a comment -

          Revision 61428 causes WARNING messages logged in server.log, which is a regression.

          It is easy to reproduce:

          1. start glassfish
          2. asadmin jms-ping --target server
          3. shut down glassfish

          Then WARNING appears in server.log,

          [2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [] [javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system] [tid: _ThreadID=133 _ThreadName=Thread-29] [timeMillis: 1366600428973] [levelValue: 900] [[
          Error occurs when shutting down JMSRA: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(
          implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
          contracts=

          {com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher}
          scope=org.glassfish.hk2.runlevel.RunLevel
          qualifiers={}
          descriptorType=CLASS
          descriptorVisibility=NORMAL
          metadata=runLevelValue={10},Bundle-SymbolicName={org.glassfish.main.core.kernel},Bundle-Version={4.0.0.SNAPSHOT}
          rank=50
          loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)
          proxiable=null
          analysisName=null
          id=690
          locatorId=0
          identityHashCode=16925195
          reified=true)]]

          [2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [ra.stop.failed] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428973] [levelValue: 900] [[
          RAR8053: RA [ jmsra ] stop failed, java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor(
          implementation=com.sun.enterprise.v3.services.impl.GrizzlyService
          contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher}

          scope=org.glassfish.hk2.runlevel.RunLevel
          qualifiers={}
          descriptorType=CLASS
          descriptorVisibility=NORMAL
          metadata=runLevelValue=

          {10}

          ,Bundle-SymbolicName=

          {org.glassfish.main.core.kernel}

          ,Bundle-Version=

          {4.0.0.SNAPSHOT}

          rank=50
          loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192]], State = [READY],9394331)
          proxiable=null
          analysisName=null
          id=690
          locatorId=0
          identityHashCode=16925195
          reified=true)]]

          [2013-04-22T11:13:48.974+0800] [glassfish 4.0] [INFO] [] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 800] [[
          shutdown of RA [ jmsra ] is either already complete or already cancelled]]

          [2013-04-22T11:13:48.974+0800] [glassfish 4.0] [WARNING] [ra.stop-unsuccessful] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 900] [[
          RAR7095: jmsra shutdown unsuccessful. Please refer the server and/or resource adapter logs for more information.]]

          Show
          David Zhao added a comment - Revision 61428 causes WARNING messages logged in server.log, which is a regression. It is easy to reproduce: 1. start glassfish 2. asadmin jms-ping --target server 3. shut down glassfish Then WARNING appears in server.log, [2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [] [javax.enterprise.resource.jms.com.sun.enterprise.connectors.jms.system] [tid: _ThreadID=133 _ThreadName=Thread-29] [timeMillis: 1366600428973] [levelValue: 900] [[ Error occurs when shutting down JMSRA: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor( implementation=com.sun.enterprise.v3.services.impl.GrizzlyService contracts= {com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher} scope=org.glassfish.hk2.runlevel.RunLevel qualifiers={} descriptorType=CLASS descriptorVisibility=NORMAL metadata=runLevelValue={10},Bundle-SymbolicName={org.glassfish.main.core.kernel},Bundle-Version={4.0.0.SNAPSHOT} rank=50 loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192] ], State = [READY] ,9394331) proxiable=null analysisName=null id=690 locatorId=0 identityHashCode=16925195 reified=true)]] [2013-04-22T11:13:48.973+0800] [glassfish 4.0] [WARNING] [ra.stop.failed] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428973] [levelValue: 900] [[ RAR8053: RA [ jmsra ] stop failed, java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.IllegalStateException: Service com.sun.enterprise.v3.services.impl.GrizzlyService was started at level 0 but it has a run level of 10. The full descriptor is SystemDescriptor( implementation=com.sun.enterprise.v3.services.impl.GrizzlyService contracts={com.sun.enterprise.v3.services.impl.GrizzlyService,org.glassfish.api.container.RequestDispatcher} scope=org.glassfish.hk2.runlevel.RunLevel qualifiers={} descriptorType=CLASS descriptorVisibility=NORMAL metadata=runLevelValue= {10} ,Bundle-SymbolicName= {org.glassfish.main.core.kernel} ,Bundle-Version= {4.0.0.SNAPSHOT} rank=50 loader=OsgiPopulatorPostProcessor.HK2Loader(OSGiModuleImpl:: Bundle = [org.glassfish.main.core.kernel [192] ], State = [READY] ,9394331) proxiable=null analysisName=null id=690 locatorId=0 identityHashCode=16925195 reified=true)]] [2013-04-22T11:13:48.974+0800] [glassfish 4.0] [INFO] [] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 800] [[ shutdown of RA [ jmsra ] is either already complete or already cancelled]] [2013-04-22T11:13:48.974+0800] [glassfish 4.0] [WARNING] [ra.stop-unsuccessful] [javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.service] [tid: _ThreadID=132 _ThreadName=Thread-28] [timeMillis: 1366600428974] [levelValue: 900] [[ RAR7095: jmsra shutdown unsuccessful. Please refer the server and/or resource adapter logs for more information.]]
          Hide
          jwells added a comment -

          Backing it out... again.

          Default policy is USE_NO_THREADS at change 61605

          Show
          jwells added a comment - Backing it out... again. Default policy is USE_NO_THREADS at change 61605
          Hide
          jwells added a comment -

          We are currently running startup services in parallel. If bugs/deadlocks are found please enter individual JIRA issues

          Show
          jwells added a comment - We are currently running startup services in parallel. If bugs/deadlocks are found please enter individual JIRA issues

            People

            • Assignee:
              jwells
              Reporter:
              jwells
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: