glassfish
  1. glassfish
  2. GLASSFISH-18267

Traffic loss during instance start between the time 8080 is up and application is loaded

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1.2
    • Fix Version/s: 4.1
    • Component/s: web_container
    • Labels:
      None
    • Environment:

      OEL6 + JRockit

      Description

      Build : 3.1.2 build 17 , JRockit VM 6u29., OEL6

      When running HA Stress tests we are observing traffic loss when a cluster instance is restarting. There is a time gap between the time when port 8080 is up and application loading is done. Loadbalancer probably detects http port is up and starts forwarding traffic. In this particular case, the time is approx. 11 sec and the request rate is 25 reqs/instance. 325 requests failed.

      The client receives a 404 during this time :

      SEVERE: 4xx or 5xx Response code - 404 HttpClientSession-47926
      Jan 17, 2012 10:58:01 PM com.sun.dft.glassfish.stress.ReadWriteTest$MyTestListener postInvoke
      SEVERE: <Unable to render embedded object: File (//www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><title>GlassFish v3 - Error report</title><style type="text/css"><) not found.--H1

      {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}

      H2

      {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}

      H3

      {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}

      BODY

      {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}

      B

      {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}

      P

      {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}

      A

      {color : black;}

      HR

      {color : #525D76;}

      --></style> </head><body><h1>HTTP Status 404 - </h1><hr/><p><b>type</b> Status report</p><p><b>message</b></p><p><b>description</b>The requested resource () is not available.</p><hr/><h3>GlassFish Server Open Source Edition 3.1.2-b17</h3></body></html>

      Server log :

      http://agni-1.us.oracle.com/net/asqe-logs.us.oracle.com/export1/3.1.2/Results/build17/ha/oel6_stress/stress/com.sun.dft.glassfish.stress.ReadWriteTest/testReadWrite/logs/st-cluster/instance101/logs/server.log

      [#|2012-01-17T22:58:00.873-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-1] on host/port [0.0.0.0:28080]|#]

      [#|2012-01-17T22:58:00.994-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [http-listener-2] on host/port [0.0.0.0:28181]|#]

      [#|2012-01-17T22:58:01.025-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0169: Created HTTP listener [admin-listener] on host/port [0.0.0.0:24848]|#]

      [#|2012-01-17T22:58:01.509-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0171: Created virtual server [server]|#]

      [#|2012-01-17T22:58:01.518-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0171: Created virtual server [__asadmin]|#]

      [#|2012-01-17T22:58:05.306-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0172: Virtual server [server] loaded default web module []|#]

      [#|2012-01-17T22:58:08.619-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.services.impl|_ThreadID=10;_ThreadName=Thread-2;|core.start_container_done|#]

      [#|2012-01-17T22:58:09.940-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1002: Security Manager is OFF.|#]

      [#|2012-01-17T22:58:10.115-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1010: Entering Security Startup Service|#]

      [#|2012-01-17T22:58:10.147-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1143: Loading policy provider com.sun.enterprise.security.provider.PolicyWrapper.|#]

      [#|2012-01-17T22:58:10.405-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [admin-realm] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#]

      [#|2012-01-17T22:58:10.417-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [file] of classtype [com.sun.enterprise.security.auth.realm.file.FileRealm] successfully created.|#]

      [#|2012-01-17T22:58:10.539-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security.auth.realm|_ThreadID=10;_ThreadName=Thread-2;|SEC1115: Realm [certificate] of classtype [com.sun.enterprise.security.auth.realm.certificate.CertificateRealm] successfully created.|#]

      [#|2012-01-17T22:58:10.591-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.security.com.sun.enterprise.security|_ThreadID=10;_ThreadName=Thread-2;|SEC1011: Security Service(s) Started Successfully|#]

      [#|2012-01-17T22:58:13.758-0800|INFO|glassfish3.1.2|ShoalLogger|_ThreadID=10;_ThreadName=Thread-2;|**GroupServiceProvider:: REGISTERED member event listeners for <group, instance> => <st-cluster, instance101>|#]

      [#|2012-01-17T22:58:14.061-0800|INFO|glassfish3.1.2|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=10;_ThreadName=Thread-2;|WEB0671: Loading application [ReadWriteServletTest] at [/ReadWriteServletTest]|#]

      [#|2012-01-17T22:58:14.076-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|CORE10010: Loading application ReadWriteServletTest done in 5,380 ms|#]

      [#|2012-01-17T22:58:14.085-0800|INFO|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=10;_ThreadName=Thread-2;|GlassFish Server Open Source Edition 3.1.2-b17 (17) startup time : Felix (8,857ms), startup services(21,300ms), total(30,157ms)|#]

      1. framework.log.1
        631 kB
        sonymanuel
      2. server.log
        245 kB
        sonymanuel

        Issue Links

          Activity

          Hide
          Gail Risdal added a comment -

          Actually, looks like the write-up was carried forward from the 3.1.2 Release Notes.

          Show
          Gail Risdal added a comment - Actually, looks like the write-up was carried forward from the 3.1.2 Release Notes.
          Hide
          oleksiys added a comment -

          @Gail

          can we change the description a bit like:

          Traffic loss during instance start between the time AJP (Apache) connector port is up and application is loaded (18267)

          Description
          A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error.

          Show
          oleksiys added a comment - @Gail can we change the description a bit like: Traffic loss during instance start between the time AJP (Apache) connector port is up and application is loaded (18267) Description A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error.
          Hide
          dlaudams added a comment - - edited

          A workaround:

          GlassFish will not start without an enabled listener. To get around this I add a secondary listener on a different port, disable the primary listener, restart/deploy, and then enabled the primary listener when running again.

          My deployment sequence looks like this:

          asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=false
          asadmin undeploy <appname>
          asadmin stop-domain domain1
          asadmin start-domain domain1
          asadmin deploy --name <appname> <earfile>
          asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=true

          Another benefit is that post-deployment checkout can be done against the secondary port before enabling the primary port.

          Show
          dlaudams added a comment - - edited A workaround: GlassFish will not start without an enabled listener. To get around this I add a secondary listener on a different port, disable the primary listener, restart/deploy, and then enabled the primary listener when running again. My deployment sequence looks like this: asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=false asadmin undeploy <appname> asadmin stop-domain domain1 asadmin start-domain domain1 asadmin deploy --name <appname> <earfile> asadmin set server.network-config.network-listeners.network-listener.http-listener-1.enabled=true Another benefit is that post-deployment checkout can be done against the secondary port before enabling the primary port.
          Hide
          Gail Risdal added a comment -

          Revised the Description in the release notes write-up to now read:

          A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when the AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error.

          I left the title as it was because that reflects the actual title of the JIRA issue.

          Show
          Gail Risdal added a comment - Revised the Description in the release notes write-up to now read: A traffic loss occurs when a clustered server instance is restarting. There is a time gap of a few seconds between when the AJP (Apache) connector port is running and when application loading is complete. During this gap client requests are denied with a 404 error. I left the title as it was because that reflects the actual title of the JIRA issue.
          Hide
          oleksiys added a comment -

          Ok, thank you Gail!

          Show
          oleksiys added a comment - Ok, thank you Gail!

            People

            • Assignee:
              oleksiys
              Reporter:
              sonymanuel
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: