[GLASSFISH-15523] Unable to call EJB on remote server after remote server has been restarted, if a call was attempted (and failed) while remote server was inaccessible Created: 11/Jan/11  Updated: 01/Nov/11  Resolved: 01/Nov/11

Status: Closed
Project: glassfish
Component/s: orb
Affects Version/s: v3.0.1
Fix Version/s: 3.1_b42

Type: Bug Priority: Major
Reporter: Vetle Leinonen-Roeim Assignee: Ken Cavanaugh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Glassfish 3.0.1, Windows 7, JDK 1.6.0u22


Attachments: Zip Archive testcase-GLASSFISH-15523.zip    
Tags: 3_1-exclude

 Description   

If an application deployed on a Glassfish server attempts to access an EJB on a remote server, we get the following error:

Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079689 No; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
at no.evote.service._MunicipalityService_Remote_DynamicStub.getMunicipalitiesWithStatus(no/evote/service/_MunicipalityService_Remote_DynamicStub.java)
... 114 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3431)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3452)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:256)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:269)
at com.sun.corba.ee.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:125)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:188)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:186)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:181)
... 117 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection refused: connect
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:340)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:239)
... 122 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:189)
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:328)
... 123 more

When the remote server is started again, subsequent attempts to access EJBs on the remote server gives the following error:

Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
at no.evote.service._ElectionEventService_Remote_DynamicStub.findById(no/evote/service/_ElectionEventService_Remote_DynamicStub.java)
... 120 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3603)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3621)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1726)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1216)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:492)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3687)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3706)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1938)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1651)
... 3 more
Caused by: java.io.IOException: End-of-stream
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1927)
... 4 more

I would expect the call to the remote EJB to work when the server was restarted.

Note that this only happens when an attempt to use the remote EJB is done while the remote server is inaccessible. If the remote server is restarted without attempting to access the EJBs, everything works as normal.



 Comments   
Comment by Ken Cavanaugh [ 12/Jan/11 ]

Remote EJB functionality is well tested in many different internal tests.
What are you doing that is different here?
Is the initial test (with the connection refused) done in a mode where the
server is inaccessible? That would be the expected error in that case.
Is the subsequent test done after making the server accessible?

Comment by Ken Cavanaugh [ 12/Jan/11 ]

Please provide complete details on how to reproduce this problem, including a test case.
Marking the issue incomplete until a test case is available.

Comment by Vetle Leinonen-Roeim [ 14/Jan/11 ]

I have attached a test case for this.

The attachment consists of four Maven projects:

  • admin, the parent project
  • admin-backend, a WAR project containing the EJB we're trying to access
  • admin-common, containing the interface for the EJB
  • admin-frontend, containing a single servlet accessing the EJB

I have also included the WAR files. This has been ripped out from a larger project, and I hope I have been able to remove any strange dependencies. Let me know if there are any problems.

To reproduce the error, I do the following;

1) deploy admin-backend on domain1 on a GF 3.0.1 instance
2) deploy admin-frontend on domain2 on the same GF instance
3) go to http://localhost:18080/status (domain2 is configured to listen to 18080), and see that both injected and looked up EJB show status OK
4) stop domain1 (the domain running the EJB)
5) go to http://localhost:18080/status and wait for timeout, which is expected
6) start domain1
7) go to http://localhost:18080/status and see that both EJBs are still unable to access the remote EJB

I would expect at the very least the new lookup to work.

Hope this helps.

Comment by Ken Cavanaugh [ 14/Jan/11 ]

Re-opening now that a test case is availab. Will investigate for 3.2 (3.1 RC1 build is 3 days aways,
so it's too late for 3.1).

Comment by Vetle Leinonen-Roeim [ 17/Jan/11 ]

A note on the test case: it's configured to connect remotely to the EJB server/domain, and I forgot to remove a hard coded host name from sun-web.xml in the admin-backend project.

Either change the host name in sun-web.xml, or override "admin-backend" in your hosts file.

Comment by Vetle Leinonen-Roeim [ 19/Jan/11 ]

I've looked a bit more into this.

If I remove any injection, add ejb-ref and ejb-local-ref for the EJB to web.xml, and only look up the EJB in the servlet's doGet method, it is able to recover after starting/stopping the backend server.

However, if I look it up in a @PostConstruct method or in the servlet's constructor, it is unable to recover - even if I look it up again in the doGet method.

Hope this helps.

Comment by Vetle Leinonen-Roeim [ 15/Feb/11 ]

Good news! I tried the test case in Glassfish 3.1 build 42, and I can't reproduce it, so feel free to close this one.

Comment by Ken Cavanaugh [ 24/May/11 ]

Apparently fixed in a later build. Closing at the request of the submitter.
.

Comment by tcourant [ 16/Jun/11 ]

Is it possible to have the patched code ?

Thank you.

Comment by Harshad Vilekar [ 31/Oct/11 ]

Reopening to update the fix version.

Comment by Harshad Vilekar [ 31/Oct/11 ]

Updated the fix version - since the bug was verified in 3.1_b42.

Comment by Harshad Vilekar [ 01/Nov/11 ]

Was already closed at the request of the submitter.

Generated at Wed Sep 02 01:16:16 UTC 2015 using JIRA 6.2.3#6260-sha1:63ef1d6dac3f4f4d7db4c1effd405ba38ccdc558.