Issue Details (XML | Word | Printable)

Key: GLASSFISH-15523
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Ken Cavanaugh
Reporter: Vetle Leinonen-Roeim
Votes: 0
Watchers: 1
Operations

If you were logged in you would be able to see more operations.
glassfish

Unable to call EJB on remote server after remote server has been restarted, if a call was attempted (and failed) while remote server was inaccessible

Created: 11/Jan/11 07:44 AM   Updated: 01/Nov/11 01:26 AM   Resolved: 01/Nov/11 01:26 AM
Component/s: orb
Affects Version/s: v3.0.1
Fix Version/s: 3.1_b42

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive testcase-GLASSFISH-15523.zip (32 kB) 14/Jan/11 02:41 AM - Vetle Leinonen-Roeim

Environment:

Glassfish 3.0.1, Windows 7, JDK 1.6.0u22


Tags: 3_1-exclude
Participants: Harshad Vilekar, Ken Cavanaugh, tcourant and Vetle Leinonen-Roeim


 Description  « Hide

If an application deployed on a Glassfish server attempts to access an EJB on a remote server, we get the following error:

Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079689 No; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
at no.evote.service._MunicipalityService_Remote_DynamicStub.getMunicipalitiesWithStatus(no/evote/service/_MunicipalityService_Remote_DynamicStub.java)
... 114 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 201 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3431)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:3452)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:256)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:269)
at com.sun.corba.ee.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:125)
at com.sun.corba.ee.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:188)
at com.sun.corba.ee.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:186)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:181)
... 117 more
Caused by: java.lang.RuntimeException: java.net.ConnectException: Connection refused: connect
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:340)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.(SocketOrChannelConnectionImpl.java:239)
... 122 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.(Socket.java:375)
at java.net.Socket.(Socket.java:189)
at org.glassfish.enterprise.iiop.impl.IIOPSSLSocketFactory.createSocket(IIOPSSLSocketFactory.java:328)
... 123 more

When the remote server is started again, subsequent attempts to access EJBs on the remote server gives the following error:

Caused by: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is:
org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.javax.rmi.CORBA.Util.mapSystemException(Util.java:270)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.privateInvoke(StubInvocationHandlerImpl.java:200)
at com.sun.corba.ee.impl.presentation.rmi.StubInvocationHandlerImpl.invoke(StubInvocationHandlerImpl.java:147)
at com.sun.corba.ee.impl.presentation.rmi.codegen.CodegenStubBase.invoke(CodegenStubBase.java:225)
at no.evote.service._ElectionEventService_Remote_DynamicStub.findById(no/evote/service/_ElectionEventService_Remote_DynamicStub.java)
... 120 more
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3603)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.connectionAbort(ORBUtilSystemException.java:3621)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1726)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doWork(SocketOrChannelConnectionImpl.java:1216)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:492)
at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:528)
Caused by: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 211 completed: No
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3687)
at com.sun.corba.ee.impl.logging.ORBUtilSystemException.ioexceptionWhenReadingConnection(ORBUtilSystemException.java:3706)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1938)
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.doOptimizedReadStrategy(SocketOrChannelConnectionImpl.java:1651)
... 3 more
Caused by: java.io.IOException: End-of-stream
at com.sun.corba.ee.impl.transport.SocketOrChannelConnectionImpl.nonBlockingRead(SocketOrChannelConnectionImpl.java:1927)
... 4 more

I would expect the call to the remote EJB to work when the server was restarted.

Note that this only happens when an attempt to use the remote EJB is done while the remote server is inaccessible. If the remote server is restarted without attempting to access the EJBs, everything works as normal.



Ken Cavanaugh added a comment - 12/Jan/11 10:13 AM

Remote EJB functionality is well tested in many different internal tests.
What are you doing that is different here?
Is the initial test (with the connection refused) done in a mode where the
server is inaccessible? That would be the expected error in that case.
Is the subsequent test done after making the server accessible?


Ken Cavanaugh added a comment - 12/Jan/11 10:14 AM

Please provide complete details on how to reproduce this problem, including a test case.
Marking the issue incomplete until a test case is available.


Vetle Leinonen-Roeim added a comment - 14/Jan/11 02:41 AM

I have attached a test case for this.

The attachment consists of four Maven projects:

  • admin, the parent project
  • admin-backend, a WAR project containing the EJB we're trying to access
  • admin-common, containing the interface for the EJB
  • admin-frontend, containing a single servlet accessing the EJB

I have also included the WAR files. This has been ripped out from a larger project, and I hope I have been able to remove any strange dependencies. Let me know if there are any problems.

To reproduce the error, I do the following;

1) deploy admin-backend on domain1 on a GF 3.0.1 instance
2) deploy admin-frontend on domain2 on the same GF instance
3) go to http://localhost:18080/status (domain2 is configured to listen to 18080), and see that both injected and looked up EJB show status OK
4) stop domain1 (the domain running the EJB)
5) go to http://localhost:18080/status and wait for timeout, which is expected
6) start domain1
7) go to http://localhost:18080/status and see that both EJBs are still unable to access the remote EJB

I would expect at the very least the new lookup to work.

Hope this helps.


Ken Cavanaugh added a comment - 14/Jan/11 09:48 AM

Re-opening now that a test case is availab. Will investigate for 3.2 (3.1 RC1 build is 3 days aways,
so it's too late for 3.1).


Vetle Leinonen-Roeim added a comment - 17/Jan/11 02:08 AM

A note on the test case: it's configured to connect remotely to the EJB server/domain, and I forgot to remove a hard coded host name from sun-web.xml in the admin-backend project.

Either change the host name in sun-web.xml, or override "admin-backend" in your hosts file.


Vetle Leinonen-Roeim added a comment - 19/Jan/11 04:18 AM

I've looked a bit more into this.

If I remove any injection, add ejb-ref and ejb-local-ref for the EJB to web.xml, and only look up the EJB in the servlet's doGet method, it is able to recover after starting/stopping the backend server.

However, if I look it up in a @PostConstruct method or in the servlet's constructor, it is unable to recover - even if I look it up again in the doGet method.

Hope this helps.


Vetle Leinonen-Roeim added a comment - 15/Feb/11 01:10 PM

Good news! I tried the test case in Glassfish 3.1 build 42, and I can't reproduce it, so feel free to close this one.


Ken Cavanaugh added a comment - 24/May/11 04:18 PM

Apparently fixed in a later build. Closing at the request of the submitter.
.


tcourant added a comment - 16/Jun/11 05:28 AM

Is it possible to have the patched code ?

Thank you.


Harshad Vilekar added a comment - 31/Oct/11 12:39 AM

Reopening to update the fix version.


Harshad Vilekar added a comment - 31/Oct/11 12:42 AM

Updated the fix version - since the bug was verified in 3.1_b42.


Harshad Vilekar added a comment - 01/Nov/11 01:26 AM

Was already closed at the request of the submitter.