glassfish-corba
  1. glassfish-corba
  2. GLASSFISH_CORBA-6

Erroneous Timeout: IOP00410037: Timeout while reading data in buffer manager

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: current
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Windows 7, Glassfish 3.1, jdk1.6.0_26

      Description

      Sporadic error when transferring large byte arrays (e.g. 10 MB)

      Bug is "com.sun.corba.ee.impl.encoding.BufferManagerReadStream.java"

      The "underflow" method the "fragmentQueue.wait(orb.getORBData().fragmentReadTimeout())" returns unexplainable,
      but it is NOT the due to the timeout, since it returnes after some milliseconds.
      It looks like it get's notified without having the "fragmentQueue" properly enqueued, which I cannot explain.
      So there is this "IOP00410037: Timeout while reading data in buffer manager",
      but it is definitely no time out, also because it is thrown after some milliseconds, and this time out actually is 18 seconds.

      This can be easily reproduced with the following very simple test.
      It takes about a couple hundred tries (might take some minutes) until I get this error.
      Even when client and server run on the same machine.
      _____________________________

      package de.test.client;

      import javax.naming.Context;
      import javax.naming.InitialContext;
      import de.test.service.remote.TimeoutTestService;

      public class TimeoutTestClient {
      private static int tryNumber = 0;
      private static boolean running = true;

      public static void main(String[] args) throws Exception {
      Context initialContext = new InitialContext();
      TimeoutTestService timeoutTestService = (TimeoutTestService) initialContext.lookup(TimeoutTestService.class.getName());

      long startTime = 0;
      while (running) {
      try

      { tryNumber++; startTime = System.currentTimeMillis(); timeoutTestService.getBytes(10000000); }

      catch (Exception e)

      { System.out.println("Error occured at try number: " + tryNumber + " after " + (System.currentTimeMillis() - startTime) + " millis"); e.printStackTrace(); running = false; }

      }
      }
      }
      _____________________________

      package de.test.service.remote;

      import javax.ejb.Remote;

      @Remote
      public interface TimeoutTestService {
      byte[] getBytes(int size);
      }
      _____________________________

      package de.test;

      import javax.ejb.Stateless;
      import de.test.service.remote.TimeoutTestService;

      @Stateless
      public class TimeoutTestServiceBean implements TimeoutTestService {
      public byte[] getBytes(int size) {
      byte[] bytes = new byte[size];
      for (int i = 0; i < bytes.length; i++)

      { bytes[i] = 0x42; }

      return bytes;
      }
      }

        Activity

        Hide
        wigun added a comment - - edited

        Suggested solution: Check whether the bufferReadManagerTimeout is real.

        BufferManagerReadStream.java
        ...
        
        @Transport
        public ByteBufferWithInfo underflow (ByteBufferWithInfo bbwi)
        ...
        
        boolean interrupted = false;
        int fragmentReadTimeout = orb.getORBData().fragmentReadTimeout();
        long timeInMillisBeforeFragmentQueueWait = System.currentTimeMillis();
        try {
            // Bug 6372405
            fragmentQueue.wait(fragmentReadTimeout);
        } catch (InterruptedException e) {
            interrupted = true ;
        }
        
        // Bug 6372405
        if (!interrupted && fragmentQueue.size() == 0 && (System.currentTimeMillis() - timeInMillisBeforeFragmentQueueWait) >= fragmentReadTimeout) {
            throw wrapper.bufferReadManagerTimeout() ;
        }
        
        Show
        wigun added a comment - - edited Suggested solution: Check whether the bufferReadManagerTimeout is real. BufferManagerReadStream.java ... @Transport public ByteBufferWithInfo underflow (ByteBufferWithInfo bbwi) ... boolean interrupted = false ; int fragmentReadTimeout = orb.getORBData().fragmentReadTimeout(); long timeInMillisBeforeFragmentQueueWait = System .currentTimeMillis(); try { // Bug 6372405 fragmentQueue.wait(fragmentReadTimeout); } catch (InterruptedException e) { interrupted = true ; } // Bug 6372405 if (!interrupted && fragmentQueue.size() == 0 && ( System .currentTimeMillis() - timeInMillisBeforeFragmentQueueWait) >= fragmentReadTimeout) { throw wrapper.bufferReadManagerTimeout() ; }
        Hide
        wigun added a comment -

        Explanation: See java doc http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html:

        "A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup. While this will rarely occur in practice, applications must guard against it by testing for the condition that should have caused the thread to be awakened, and continuing to wait if the condition is not satisfied."

        Show
        wigun added a comment - Explanation: See java doc http://docs.oracle.com/javase/6/docs/api/java/lang/Object.html: "A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup. While this will rarely occur in practice, applications must guard against it by testing for the condition that should have caused the thread to be awakened, and continuing to wait if the condition is not satisfied."
        Hide
        wigun added a comment -

        Alternative (better?) solution:

        BufferManagerReadStream.java
        ...
                            int fragmentReadTimeout = orb.getORBData().fragmentReadTimeout();
                            long timeBeforeWait = System.currentTimeMillis();
                            boolean interrupted = false ;
                            try {
                                while (fragmentQueue.size() == 0 && System.currentTimeMillis() < (timeBeforeWait + fragmentReadTimeout)) {
                                    // Bug 6372405
                                    fragmentQueue.wait(fragmentReadTimeout);
                                }
                            } catch (InterruptedException e) {
                                interrupted = true ;
                            }
        ...
        
        Show
        wigun added a comment - Alternative (better?) solution: BufferManagerReadStream.java ... int fragmentReadTimeout = orb.getORBData().fragmentReadTimeout(); long timeBeforeWait = System .currentTimeMillis(); boolean interrupted = false ; try { while (fragmentQueue.size() == 0 && System .currentTimeMillis() < (timeBeforeWait + fragmentReadTimeout)) { // Bug 6372405 fragmentQueue.wait(fragmentReadTimeout); } } catch (InterruptedException e) { interrupted = true ; } ...
        Hide
        jthoennes added a comment -

        We have filed an Oracle Service Request (SR 3-5014082933)
        and believe we are also hit by this issues since we load large amounts of data every day.

        Sporadically, the download fails with the above error message.

        See also the old bug GLASSFISH-2872 were Ken Cavanaugh fixed an issue by guarding against such thread wakeups.

        So please: Any answer from development here?

        Show
        jthoennes added a comment - We have filed an Oracle Service Request (SR 3-5014082933) and believe we are also hit by this issues since we load large amounts of data every day. Sporadically, the download fails with the above error message. See also the old bug GLASSFISH-2872 were Ken Cavanaugh fixed an issue by guarding against such thread wakeups. So please: Any answer from development here?
        Hide
        boernd added a comment -

        We manually patched the glassfish-corba-orb.jar file with the "Alternative (better?) solution" which seems to work for us so far. If anybody wants to have the modified jar file, plz contact me.

        Regards
        Bernd

        Show
        boernd added a comment - We manually patched the glassfish-corba-orb.jar file with the "Alternative (better?) solution" which seems to work for us so far. If anybody wants to have the modified jar file, plz contact me. Regards Bernd
        Hide
        cwolf77 added a comment -

        We are facing this problem with glassfish-3.2.1.
        Is there any plan to fix this issue or is there an official patch available somewhere?

        Thx
        Kind Regards,
        Christian

        Show
        cwolf77 added a comment - We are facing this problem with glassfish-3.2.1. Is there any plan to fix this issue or is there an official patch available somewhere? Thx Kind Regards, Christian
        Hide
        Ed Bratt added a comment -

        If you have a support contract, please open a support case and reference this JIRA. Thank you.

        Show
        Ed Bratt added a comment - If you have a support contract, please open a support case and reference this JIRA. Thank you.

          People

          • Assignee:
            Unassigned
            Reporter:
            wigun
          • Votes:
            4 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated: