Skip to main content

Faban hangs due to blocking socket read

  • From: Mukil Kesavan <mukilk@...>
  • To: issues@...
  • Subject: Faban hangs due to blocking socket read
  • Date: Fri, 6 Jul 2012 11:25:16 -0400

Hello,

This email is in the context of Olio + Faban. I've noticed that the workload threads sometimes don't terminate cleanly at all. The master ends up waiting forever for some agents after the ramp down is complete. I looked at the jvm stack of a non-terminating agent and found out that a ApacheHC3Transport::readURLcall blocks forever. Snippet:


"UIDriverAgent[0].847" daemon prio=10 tid=0x00007f046c62a800 nid=0x18ac runnable [0x00007f04392d4000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at com.sun.faban.driver.transport.util.TimedInputStream.read(TimedInputStream.java:171)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x00000000da638160> (a java.io.BufferedInputStream)
at org.apache.commons.httpclient.ChunkedInputStream.read(ChunkedInputStream.java:182)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:108)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.commons.httpclient.AutoCloseInputStream.read(AutoCloseInputStream.java:127)
at com.sun.faban.driver.transport.hc3.ApacheHC3Transport.readResponse(ApacheHC3Transport.java:906)
at com.sun.faban.driver.transport.hc3.ApacheHC3Transport.readURL(ApacheHC3Transport.java:302)
at com.sun.faban.driver.transport.hc3.ApacheHC3Transport.readURL(ApacheHC3Transport.java:434)
at org.apache.olio.workload.driver.UIDriver.doTagSearch(UIDriver.java:456)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.faban.driver.engine.TimeThread.doRun(TimeThread.java:166)
at com.sun.faban.driver.engine.AgentThread.run(AgentThread.java:202)

The constructor for the ApacheHC3Transport class seems to explicitly set a connection timeout but no read timeout. So after I set the SOTimeout parameter to 30 seconds (same value as connection timeout) the agents don't hang any more.

Could someone more familiar with the code please verify that this indeed is a valid fix? Are there any other implications?

Thanks,
Mukil


Faban hangs due to blocking socket read

Mukil Kesavan 07/06/2012
 
 
Close
loading
Please Confirm
Close