glassfish
  1. glassfish
  2. GLASSFISH-15523

Unable to call EJB on remote server after remote server has been restarted, if a call was attempted (and failed) while remote server was inaccessible

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v3.0.1
    • Fix Version/s: 3.1_b42
    • Component/s: orb
    • Labels:
      None
    • Environment:

      Glassfish 3.0.1, Windows 7, JDK 1.6.0u22

      Description

      If an application deployed on a Glassfish server attempts to access an EJB on a remote server, we get the following error:

      Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079689 No; nested exception is:
      org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
      at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
      at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
      at no.evote.service._MunicipalityService_Remote_DynamicStub.getMunicipalitiesWithStatus(no/evote/service/_MunicipalityService_Remote_DynamicStub.java)
      ... 114 more
      Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3431)
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3452)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:256)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:269)
      at com.sun.corba.ee.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:125)
      at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:188)
      at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:186)
      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:181)
      ... 117 more
      Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection refused: connect
      at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:340)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:239)
      ... 122 more
      Caused by: java.net.ConnectException: Connection refused: connect
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:529)
      at java.net.Socket.connect(Socket.java:478)
      at java.net.Socket.(Socket.java:375)
      at java.net.Socket.(Socket.java:189)
      at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:328)
      ... 123 more

      When the remote server is started again, subsequent attempts to access EJBs on the remote server gives the following error:

      Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
      org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
      at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
      at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
      at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
      at no.evote.service._ElectionEventService_Remote_DynamicStub.findById(no/evote/service/_ElectionEventService_Remote_DynamicStub.java)
      ... 120 more
      Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3603)
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3621)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1726)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1216)
      at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:492)
      at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)
      Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211 completed: No
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3687)
      at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3706)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1938)
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1651)
      ... 3 more
      Caused by: java.io.IOException: End-of-stream
      at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1927)
      ... 4 more

      I would expect the call to the remote EJB to work when the server was restarted.

      Note that this only happens when an attempt to use the remote EJB is done while the remote server is inaccessible. If the remote server is restarted without attempting to access the EJBs, everything works as normal.

        Activity

        Vetle Leinonen-Roeim created issue -
        Nazrul made changes -
        Field Original Value New Value
        Assignee Nazrul [ ai109478 ] haroldcarr [ haroldcarr ]
        Component/s orb [ 10610 ]
        Nazrul made changes -
        Assignee haroldcarr [ haroldcarr ] Ken Cavanaugh [ kcavanaugh ]
        Hide
        Ken Cavanaugh added a comment -

        Remote EJB functionality is well tested in many different internal tests.
        What are you doing that is different here?
        Is the initial test (with the connection refused) done in a mode where the
        server is inaccessible? That would be the expected error in that case.
        Is the subsequent test done after making the server accessible?

        Show
        Ken Cavanaugh added a comment - Remote EJB functionality is well tested in many different internal tests. What are you doing that is different here? Is the initial test (with the connection refused) done in a mode where the server is inaccessible? That would be the expected error in that case. Is the subsequent test done after making the server accessible?
        Hide
        Ken Cavanaugh added a comment -

        Please provide complete details on how to reproduce this problem, including a test case.
        Marking the issue incomplete until a test case is available.

        Show
        Ken Cavanaugh added a comment - Please provide complete details on how to reproduce this problem, including a test case. Marking the issue incomplete until a test case is available.
        Ken Cavanaugh made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Incomplete [ 4 ]
        Hide
        Vetle Leinonen-Roeim added a comment -

        I have attached a test case for this.

        The attachment consists of four Maven projects:

        • admin, the parent project
        • admin-backend, a WAR project containing the EJB we're trying to access
        • admin-common, containing the interface for the EJB
        • admin-frontend, containing a single servlet accessing the EJB

        I have also included the WAR files. This has been ripped out from a larger project, and I hope I have been able to remove any strange dependencies. Let me know if there are any problems.

        To reproduce the error, I do the following;

        1) deploy admin-backend on domain1 on a GF 3.0.1 instance
        2) deploy admin-frontend on domain2 on the same GF instance
        3) go to http://localhost:18080/status (domain2 is configured to listen to 18080), and see that both injected and looked up EJB show status OK
        4) stop domain1 (the domain running the EJB)
        5) go to http://localhost:18080/status and wait for timeout, which is expected
        6) start domain1
        7) go to http://localhost:18080/status and see that both EJBs are still unable to access the remote EJB

        I would expect at the very least the new lookup to work.

        Hope this helps.

        Show
        Vetle Leinonen-Roeim added a comment - I have attached a test case for this. The attachment consists of four Maven projects: admin, the parent project admin-backend, a WAR project containing the EJB we're trying to access admin-common, containing the interface for the EJB admin-frontend, containing a single servlet accessing the EJB I have also included the WAR files. This has been ripped out from a larger project, and I hope I have been able to remove any strange dependencies. Let me know if there are any problems. To reproduce the error, I do the following; 1) deploy admin-backend on domain1 on a GF 3.0.1 instance 2) deploy admin-frontend on domain2 on the same GF instance 3) go to http://localhost:18080/status (domain2 is configured to listen to 18080), and see that both injected and looked up EJB show status OK 4) stop domain1 (the domain running the EJB) 5) go to http://localhost:18080/status and wait for timeout, which is expected 6) start domain1 7) go to http://localhost:18080/status and see that both EJBs are still unable to access the remote EJB I would expect at the very least the new lookup to work. Hope this helps.
        Vetle Leinonen-Roeim made changes -
        Attachment testcase-GLASSFISH-15523.zip [ 27602 ]
        Ken Cavanaugh made changes -
        Tags 3_1-exclude
        Hide
        Ken Cavanaugh added a comment -

        Re-opening now that a test case is availab. Will investigate for 3.2 (3.1 RC1 build is 3 days aways,
        so it's too late for 3.1).

        Show
        Ken Cavanaugh added a comment - Re-opening now that a test case is availab. Will investigate for 3.2 (3.1 RC1 build is 3 days aways, so it's too late for 3.1).
        Ken Cavanaugh made changes -
        Resolution Incomplete [ 4 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Ken Cavanaugh made changes -
        Fix Version/s 3.2 [ 10969 ]
        Hide
        Vetle Leinonen-Roeim added a comment -

        A note on the test case: it's configured to connect remotely to the EJB server/domain, and I forgot to remove a hard coded host name from sun-web.xml in the admin-backend project.

        Either change the host name in sun-web.xml, or override "admin-backend" in your hosts file.

        Show
        Vetle Leinonen-Roeim added a comment - A note on the test case: it's configured to connect remotely to the EJB server/domain, and I forgot to remove a hard coded host name from sun-web.xml in the admin-backend project. Either change the host name in sun-web.xml, or override "admin-backend" in your hosts file.
        Hide
        Vetle Leinonen-Roeim added a comment -

        I've looked a bit more into this.

        If I remove any injection, add ejb-ref and ejb-local-ref for the EJB to web.xml, and only look up the EJB in the servlet's doGet method, it is able to recover after starting/stopping the backend server.

        However, if I look it up in a @PostConstruct method or in the servlet's constructor, it is unable to recover - even if I look it up again in the doGet method.

        Hope this helps.

        Show
        Vetle Leinonen-Roeim added a comment - I've looked a bit more into this. If I remove any injection, add ejb-ref and ejb-local-ref for the EJB to web.xml, and only look up the EJB in the servlet's doGet method, it is able to recover after starting/stopping the backend server. However, if I look it up in a @PostConstruct method or in the servlet's constructor, it is unable to recover - even if I look it up again in the doGet method. Hope this helps.
        Hide
        Vetle Leinonen-Roeim added a comment -

        Good news! I tried the test case in Glassfish 3.1 build 42, and I can't reproduce it, so feel free to close this one.

        Show
        Vetle Leinonen-Roeim added a comment - Good news! I tried the test case in Glassfish 3.1 build 42, and I can't reproduce it, so feel free to close this one.
        Hide
        Ken Cavanaugh added a comment -

        Apparently fixed in a later build. Closing at the request of the submitter.
        .

        Show
        Ken Cavanaugh added a comment - Apparently fixed in a later build. Closing at the request of the submitter. .
        Ken Cavanaugh made changes -
        Status Reopened [ 4 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        tcourant added a comment -

        Is it possible to have the patched code ?

        Thank you.

        Show
        tcourant added a comment - Is it possible to have the patched code ? Thank you.
        Hide
        Harshad Vilekar added a comment -

        Reopening to update the fix version.

        Show
        Harshad Vilekar added a comment - Reopening to update the fix version.
        Harshad Vilekar made changes -
        Resolution Fixed [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Hide
        Harshad Vilekar added a comment -

        Updated the fix version - since the bug was verified in 3.1_b42.

        Show
        Harshad Vilekar added a comment - Updated the fix version - since the bug was verified in 3.1_b42.
        Harshad Vilekar made changes -
        Fix Version/s 3.1_b42 [ 14632 ]
        Fix Version/s 3.2 [ 10969 ]
        Hide
        Harshad Vilekar added a comment -

        Was already closed at the request of the submitter.

        Show
        Harshad Vilekar added a comment - Was already closed at the request of the submitter.
        Harshad Vilekar made changes -
        Status Reopened [ 4 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Ken Cavanaugh
            Reporter:
            Vetle Leinonen-Roeim
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: