glassfish
  1. glassfish
  2. GLASSFISH-15799

StackOverflowError in web container during instance startup.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.1_b40
    • Fix Version/s: 3.1_b41
    • Component/s: web_container
    • Labels:
      None
    • Environment:

      SuSE + iWS

      Description

      During instance startup in of the http failover tests, web container startup fails with StackOverflowError. After this, the instance is unresponsive to any asadmin commands and there are a lot of grizzly exceptions.

      From the error message it looks like there was some request right at the time of web container initialization.

      [#|2011-02-02T05:48:55.302-0800|INFO|glassfish3.1|ShoalLogger|_ThreadID=12;_ThreadName=Thread-1;|GMS1024: Adding Join member: instance105 group: st-cluster StartupState: INSTANCE_STARTUP |#]

      [#|2011-02-02T05:48:55.672-0800|INFO|glassfish3.1|javax.enterprise.system.container.web.com.sun.enterprise.web|_ThreadID=1;_ThreadName=Thread-1;|WEB0172: Virtual server [server] loaded default web module []|#]

      [#|2011-02-02T05:48:56.120-0800|SEVERE|glassfish3.1|org.apache.catalina.connector.CoyoteAdapter|_ThreadID=17;_ThreadName=Thread-1;|PWC3989: An exception or error occurred in the container during the request processing
      java.lang.StackOverflowError
      at java.lang.String.indexOf(String.java:1521)
      at java.net.URLStreamHandler.parseURL(URLStreamHandler.java:127)
      at java.net.URL.<init>(URL.java:596)
      at java.net.URL.<init>(URL.java:464)
      at sun.misc.URLClassPath$Loader.findResource(URLClassPath.java:458)
      at sun.misc.URLClassPath.findResource(URLClassPath.java:146)
      at java.net.URLClassLoader$2.run(URLClassLoader.java:385)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findResource(URLClassLoader.java:382)
      at java.lang.ClassLoader.getResource(ClassLoader.java:1003)
      at org.apache.felix.framework.ModuleImpl.doImplicitBootDelegation(ModuleImpl.java:1535)
      at org.apache.felix.framework.ModuleImpl.searchDynamicImports(ModuleImpl.java:1472)
      at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:734)
      at org.apache.felix.framework.ModuleImpl.getResourceByDelegation(ModuleImpl.java:652)
      at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.getResource(ModuleImpl.java:2020)
      at java.lang.ClassLoader.getResourceAsStream(ClassLoader.java:1193)
      at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2324)
      at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2309)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2308)
      at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1364)
      at java.util.ResourceBundle.findBundle(ResourceBundle.java:1328)
      at java.util.ResourceBundle.findBundle(ResourceBundle.java:1282)
      at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1224)
      at java.util.ResourceBundle.getBundle(ResourceBundle.java:952)
      at org.apache.catalina.util.StringManager.getStringInternal(StringManager.java:181)
      at org.apache.catalina.util.StringManager.getString(StringManager.java:156)
      at org.apache.catalina.util.StringManager.getString(StringManager.java:150)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:233)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)

      Grizzly errors after the stackoverflow. :
      [#|2011-02-02T05:48:56.329-0800|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=17;_ThreadName=Thread-1;|afterService unexpected exception:
      java.lang.StackOverflowError
      at java.lang.LinkageError.<init>(LinkageError.java:36)
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
      at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.findClass(ModuleImpl.java:1907)
      at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:727)
      at org.apache.felix.framework.ModuleImpl.access$400(ModuleImpl.java:71)
      at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1768)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
      at com.sun.grizzly.tcp.StaticResourcesAdapter.afterService(StaticResourcesAdapter.java:301)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.afterService(ContainerMapper.java:356)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:509)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)
      at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:267)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:227)
      at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:170)
      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:508)

        Activity

        Hide
        Shing Wai Chan added a comment -

        Can you provide more information/steps on how to reproduce the issue?

        Show
        Shing Wai Chan added a comment - Can you provide more information/steps on how to reproduce the issue?
        Hide
        sonymanuel added a comment -

        This issue happened during a Full HA run on Hudson.

        The specific test where instance startup failed is in http module with modified-session attribute : com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1

        See test logs :
        http://bigapp-oblade-10.us.oracle.com:1080/job/GF-HA-Functional-tests/38/artifact/results/http-modified-session/com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1/html/index.html

        Can you please check with Gopal how to re-run this scenario.

        I don't think the issue is specific to this test. Most of the tests after this started failing.

        http://bigapp-oblade-10.us.oracle.com:1080/job/GF-HA-Functional-tests/38/artifact/results/summary/overall-summary.html

        Show
        sonymanuel added a comment - This issue happened during a Full HA run on Hudson. The specific test where instance startup failed is in http module with modified-session attribute : com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1 See test logs : http://bigapp-oblade-10.us.oracle.com:1080/job/GF-HA-Functional-tests/38/artifact/results/http-modified-session/com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1/html/index.html Can you please check with Gopal how to re-run this scenario. I don't think the issue is specific to this test. Most of the tests after this started failing. http://bigapp-oblade-10.us.oracle.com:1080/job/GF-HA-Functional-tests/38/artifact/results/summary/overall-summary.html
        Hide
        Shing Wai Chan added a comment -

        The circular logic is from CoyoteAdapter as follows:
        if (context instanceof ContextRootInfo)

        { // this block of code will be invoked when an AJP request is intended // for an Adapter other than the CoyoteAdapter final Adapter toInvoke = ((ContextRootInfo) context).getAdapter(); toInvoke.service(req, res); toInvoke.afterService(req, res); return false; }

        This is added in r40454 for fixing issue https://glassfish.dev.java.net/issues/show_bug.cgi?id=12458

        Show
        Shing Wai Chan added a comment - The circular logic is from CoyoteAdapter as follows: if (context instanceof ContextRootInfo) { // this block of code will be invoked when an AJP request is intended // for an Adapter other than the CoyoteAdapter final Adapter toInvoke = ((ContextRootInfo) context).getAdapter(); toInvoke.service(req, res); toInvoke.afterService(req, res); return false; } This is added in r40454 for fixing issue https://glassfish.dev.java.net/issues/show_bug.cgi?id=12458
        Hide
        Ryan Lubke added a comment -

        When the ContainerMapper is initialized, it adds itself as the Adapter:

        protected synchronized void configureMapper() {
        mapper.setDefaultHostName(defaultHostName);
        mapper.addHost(defaultHostName,new String[]{},null);
        mapper.addContext(defaultHostName,ROOT,
        new ContextRootInfo(this,null),
        new String[]

        {"index.html","index.htm"}

        ,null);
        // Container deployed have the right to override the default setting.
        Mapper.setAllowReplacement(true);
        }

        Based on the stacktrace, at the time the request is being serviced, there is only one
        Adapter present (the CoyoteAdapter), so we remove any MappingData accumulated
        at this point and invoke the adapter directly.

        Because there isn't any mapping data, the CoyoteAdapter will attempt to map the request
        itself, which ends up returning the ContentRootInfo with the ContainerMapper as the adapter.

        The code Shing-Wai referenced at that point will invoke the ContainerMapper again, and then we
        cycle until we blow the stack.

        Since we shouldn't be re-invoking the ContainerMapper at all, we can fix this condition by
        checking the Adapter before invoking it.

        Show
        Ryan Lubke added a comment - When the ContainerMapper is initialized, it adds itself as the Adapter: protected synchronized void configureMapper() { mapper.setDefaultHostName(defaultHostName); mapper.addHost(defaultHostName,new String[]{},null); mapper.addContext(defaultHostName,ROOT, new ContextRootInfo(this,null), new String[] {"index.html","index.htm"} ,null); // Container deployed have the right to override the default setting. Mapper.setAllowReplacement(true); } Based on the stacktrace, at the time the request is being serviced, there is only one Adapter present (the CoyoteAdapter), so we remove any MappingData accumulated at this point and invoke the adapter directly. Because there isn't any mapping data, the CoyoteAdapter will attempt to map the request itself, which ends up returning the ContentRootInfo with the ContainerMapper as the adapter. The code Shing-Wai referenced at that point will invoke the ContainerMapper again, and then we cycle until we blow the stack. Since we shouldn't be re-invoking the ContainerMapper at all, we can fix this condition by checking the Adapter before invoking it.
        Hide
        Ryan Lubke added a comment -

        How bad is its impact? (Severity)
        Identify why the fix needs to occur now:

        • Likely to generate a customer support call
        • Significantly impacts the operation of a primary release driver feature

        How often does it happen? (Frequency)

        • the code that triggered this has been in place for several months.

        How much effort is required to fix it? (Cost)

        • minimal (single line check to prevent recursion)

        What is the risk of fixing it? (Risk)

        • minimal

        Does a work around for the issue exist? Can the workaround be reasonably employed by the end user?

        • no

        If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes?

        • n/a

        How long has the bug existed in the product?

        • several months

        Do regression tests exist for this issue?

        • Once the fix is in place the HA test shouldn't have issues going forward

        Which tests should QA (re)run to verify the fix did not destabilize GlassFish?

        • com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1

        When will a tested fix be ready for integration?

        • 2/3/2011
        Show
        Ryan Lubke added a comment - How bad is its impact? (Severity) Identify why the fix needs to occur now: Likely to generate a customer support call Significantly impacts the operation of a primary release driver feature How often does it happen? (Frequency) the code that triggered this has been in place for several months. How much effort is required to fix it? (Cost) minimal (single line check to prevent recursion) What is the risk of fixing it? (Risk) minimal Does a work around for the issue exist? Can the workaround be reasonably employed by the end user? no If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes? n/a How long has the bug existed in the product? several months Do regression tests exist for this issue? Once the fix is in place the HA test shouldn't have issues going forward Which tests should QA (re)run to verify the fix did not destabilize GlassFish? com.sun.dft.glassfish.http.failover.runtime.HttpFailoverRuntimeNewTest1 When will a tested fix be ready for integration? 2/3/2011
        Hide
        Chris Kasso added a comment -

        Approved for RC2.

        Show
        Chris Kasso added a comment - Approved for RC2.
        Hide
        Ryan Lubke added a comment -

        Changes applied:

        Revisions:
        ----------
        44884

        Modified Paths:
        ---------------
        branches/3.1/web/web-core/src/main/java/org/apache/catalina/connector/CoyoteAdapter.java

        Show
        Ryan Lubke added a comment - Changes applied: Revisions: ---------- 44884 Modified Paths: --------------- branches/3.1/web/web-core/src/main/java/org/apache/catalina/connector/CoyoteAdapter.java
        Hide
        Ryan Lubke added a comment -

        Changes applied to trunk: revision 44885.

        Show
        Ryan Lubke added a comment - Changes applied to trunk: revision 44885.

          People

          • Assignee:
            Ryan Lubke
            Reporter:
            sonymanuel
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved: