[GLASSFISH-17499] Uncaught NullPointerException during http.ProcessorTask.doProcess can break postProcess buffer cleanup Created: 27/Oct/11  Updated: 28/Nov/11  Resolved: 28/Oct/11

Status: Resolved
Project: glassfish
Component/s: web_container
Affects Version/s: 3.1.1_b12
Fix Version/s: 3.1.2_b08, 4.0

Type: Bug Priority: Major
Reporter: steve_cochran Assignee: Amy Roh
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

We are running the GlassFish Server version 3.1.1 build 12 plus patches loaded from the private Oracle repository on 8/16/2011.
During testing we noted that the following pair of errors occurred following a very large (over 8192 byte) URI sent to the server. Thereafter, each time that the thread was reused, it generated the same pair of errors, most likely because the second error was not caught and caused the thread to be released without the postProcess buffer cleanup 'recycle()'. With just a few retries of the long URI, we managed to disable all of the domain's request processing threads.

The error was triggered by the buffer overflow and then the unhandled null pointer error in the ErrorHandler caused the rest - the stack traces are shown below.

The actual error appears to be in the ErrorHandler property of the associated thread; however the grizzly http.ProcessorTask.doProcess probably should protect itself.

[#|2011-10-06T17:13:28.188-0400|SEVERE|oracle-glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=28;_ThreadName=Thread-3;|GRIZZLY0039: Request URI is too large.
java.nio.BufferOverflowException
at com.sun.grizzly.tcp.http11.InternalInputBuffer.fill(InternalInputBuffer.java:765)
at com.sun.grizzly.tcp.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:402)
at com.sun.grizzly.http.ProcessorTask.parseRequest(ProcessorTask.java:861)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:692)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]

[#|2011-10-06T17:13:28.188-0400|SEVERE|oracle-glassfish3.1.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=28;_ThreadName=Thread-3;|GRIZZLY0051: ProcessorTask exception.
java.lang.NullPointerException
at java.nio.CharBuffer.put(CharBuffer.java:896)
at com.sun.enterprise.web.accesslog.DefaultAccessLogFormatterImpl.appendRequestInfo(DefaultAccessLogFormatterImpl.java:494)
at com.sun.enterprise.web.accesslog.DefaultAccessLogFormatterImpl.appendLogEntry(DefaultAccessLogFormatterImpl.java:264)
at com.sun.enterprise.web.PEAccessLogValve.postInvoke(PEAccessLogValve.java:592)
at com.sun.enterprise.web.VirtualServer$2.onParsingError(VirtualServer.java:1698)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:709)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:662)

#]


 Comments   
Comment by oleksiys [ 27/Oct/11 ]

reassign to webcontainer to fix NPE.

Comment by Shing Wai Chan [ 27/Oct/11 ]

The NPE is due to the fact that hreq.getRequestURI() is null.
The URI is null because of the buffer overflow.

Assign to Amy to investigate what should be done in this case.
Note that one need to be able to identify this from the log later.

Comment by Amy Roh [ 28/Oct/11 ]

Fixed in svn 50549 & 50551.

Comment by gpambrozio [ 28/Nov/11 ]

Is there a workaround for this issue? Maybe adding a filter to every request? I started seeing this error today on my servers and I'm worried it might be some kind of DOS or that it might become one...

Generated at Sun Aug 28 14:57:48 UTC 2016 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.