glassfish
  1. glassfish
  2. GLASSFISH-18402

EJB lookup problem with webstart; multithreaded use of same client-side proxy?

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2_b18, 3.1.2_b19, 3.1.2_b20, 3.1.2_b21, 3.1.2_b22, 3.1.2_b23
    • Fix Version/s: None
    • Component/s: orb
    • Labels:
      None

      Description

      Our enterprise application contains an EJB module and the application client. The remote jndi lookup in the application client works fine, if the client stub runs with the appclient shell script. But when the application is executed via webstart, the beans won't be located properly due to the following error:

      Caused by: javax.naming.NamingException: ejb ref resolution error for remote business interfacede.beans.einkauf.MyServiceBeanRemote [Root exception is java.lang.IllegalArgumentException: java.lang.ClassCastException@532c81b7]
      	at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:434)
      	at com.sun.ejb.containers.RemoteBusinessObjectFactory.getObjectInstance(RemoteBusinessObjectFactory.java:75)
      	at javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
      	at com.sun.enterprise.naming.impl.SerialContext.getObjectInstance(SerialContext.java:556)
      	at com.sun.enterprise.naming.impl.SerialContext.lookup(SerialContext.java:514)
      	... 15 more
      Caused by: java.lang.IllegalArgumentException: java.lang.ClassCastException@532c81b7
      	at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:422)
      	... 19 more
      

      Maybe a cache issue, but we have no clue what goes wrong here.

        Activity

        Hide
        Tim Quinn added a comment -

        Can you attach your application to this issue so we can try to reproduce the problem from here?

        Also, did you try clearing the Java Web Start cache on your client system and relaunching?

        (Yes, this requires Java Web Start to download all the GlassFish system JARs again which can take a while as I'm sure you've experienced.)

        Show
        Tim Quinn added a comment - Can you attach your application to this issue so we can try to reproduce the problem from here? Also, did you try clearing the Java Web Start cache on your client system and relaunching? (Yes, this requires Java Web Start to download all the GlassFish system JARs again which can take a while as I'm sure you've experienced.)
        Hide
        andydr added a comment -

        The application is very complex with about 60 EJBs and requires a JDBC datastore. I can try to build a corresponding example.

        And yes, I cleared the Web Start cache.

        Show
        andydr added a comment - The application is very complex with about 60 EJBs and requires a JDBC datastore. I can try to build a corresponding example. And yes, I cleared the Web Start cache.
        Hide
        Tim Quinn added a comment -

        I am getting some help from the EJB team to direct where we need to look.

        Also, what Java version are you running on the client?

        Show
        Tim Quinn added a comment - I am getting some help from the EJB team to direct where we need to look. Also, what Java version are you running on the client?
        Hide
        andydr added a comment -

        The client runs with 1.6.0_29.

        Just some details on the ejb lookup. The @Remote annotated interface is
        retrieved with InitialContext#lookup(<interfaceName>.class.getName()) and not injected with @EJB.

        I found out, that the problematic remote interfaces are looked up multiple times during startup (in the main-method).
        When I introduce lazy initialization for these stateless session beans and reuse it, the problem is gone.

        During runtime multiple lookups for other stateless session beans are no problem.

        Show
        andydr added a comment - The client runs with 1.6.0_29. Just some details on the ejb lookup. The @Remote annotated interface is retrieved with InitialContext#lookup(<interfaceName>.class.getName()) and not injected with @EJB. I found out, that the problematic remote interfaces are looked up multiple times during startup (in the main-method). When I introduce lazy initialization for these stateless session beans and reuse it, the problem is gone. During runtime multiple lookups for other stateless session beans are no problem.
        Hide
        marina vatkina added a comment -

        How does your impl class look like? Does it have a business method called 'create'?

        Show
        marina vatkina added a comment - How does your impl class look like? Does it have a business method called 'create'?
        Hide
        andydr added a comment -

        The SLSB looks like this

         
        @Stateless
        public class MyServiceBean implements MyServiceBeanRemote {
            @Override
            public MyService create() {
                MyService art = new MyService();
                em.persist(art);
                return art;
            }
        }
        

        So, yes it has a business method 'create'. Is something wrong with this?

        Show
        andydr added a comment - The SLSB looks like this @Stateless public class MyServiceBean implements MyServiceBeanRemote { @Override public MyService create() { MyService art = new MyService(); em.persist(art); return art; } } So, yes it has a business method 'create'. Is something wrong with this?
        Hide
        marina vatkina added a comment -

        Does it make any difference if you rename 'create' to something else?

        Show
        marina vatkina added a comment - Does it make any difference if you rename 'create' to something else?
        Hide
        andydr added a comment - - edited

        Marina, the problem is NOT gone, when I rename the 'create' method.

        One very last question: Quite often I get the following error:

        org.omg.CORBA.COMM_FAILURE: WARNUNG: IOP00410037: Timeout while reading data in buffer manager vmcid: OMG minor code: 37 completed: No

        The configuration TCP Read Timeout is set to 30secs! Is there another ORB configuration parameter available to solve
        this issue?

        Show
        andydr added a comment - - edited Marina, the problem is NOT gone, when I rename the 'create' method. One very last question: Quite often I get the following error: org.omg.CORBA.COMM_FAILURE: WARNUNG: IOP00410037: Timeout while reading data in buffer manager vmcid: OMG minor code: 37 completed: No The configuration TCP Read Timeout is set to 30secs! Is there another ORB configuration parameter available to solve this issue?
        Hide
        andydr added a comment - - edited

        Just a reminder, the problem (ClassCastException during ejb lookup) happens only under Webstart. When the application stub is launched via the appclient script, it does not occur!

        Apologize Marina, after a couple of webstart startups I feel, the timeout problem relates to the ClassCastException. Since sometimes the application starts up without any problems, but sometimes with read timeout exception and ClassCastExc.

        A cross-check with appclient script showed the read timeout exception but no ClassCastExc.

        Show
        andydr added a comment - - edited Just a reminder, the problem (ClassCastException during ejb lookup) happens only under Webstart. When the application stub is launched via the appclient script, it does not occur! Apologize Marina, after a couple of webstart startups I feel, the timeout problem relates to the ClassCastException. Since sometimes the application starts up without any problems, but sometimes with read timeout exception and ClassCastExc. A cross-check with appclient script showed the read timeout exception but no ClassCastExc.
        Hide
        marina vatkina added a comment -

        com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:422) calls a generated "create(String)" method. So my initial guess was that you also have such method in your remote interface, and the match happens correctly in ACC and not in JWS. But if renaming didn't help, it's not the cause of the problem.

        Show
        marina vatkina added a comment - com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:422) calls a generated "create(String)" method. So my initial guess was that you also have such method in your remote interface, and the match happens correctly in ACC and not in JWS. But if renaming didn't help, it's not the cause of the problem.
        Hide
        Tim Quinn added a comment -

        andydr,

        Is it the Java Web Start launch that sometimes works and sometimes does not?

        Or does Java Web Start always fail and appclient launches sometimes work and sometimes not?

        I long time ago I think I have seen cases where ORB failures of certain types can be reported as other types of errors.
        I am not sure that is happening in this case but given the ORB errors you see sometimes perhaps this is a similar case.

        Show
        Tim Quinn added a comment - andydr, Is it the Java Web Start launch that sometimes works and sometimes does not? Or does Java Web Start always fail and appclient launches sometimes work and sometimes not? I long time ago I think I have seen cases where ORB failures of certain types can be reported as other types of errors. I am not sure that is happening in this case but given the ORB errors you see sometimes perhaps this is a similar case.
        Hide
        andydr added a comment - - edited

        Tim, the application launch works in either case. But the ClassCastException during ejb lookup occurs in JWS.

        BTW: The ClassCastExc happens even without the read timeout and is also unrelated to specific beans. I.e. with each application startup the ClassCastExc happens for a different bean.

        java.lang.ClassCastException@4c1e6823]
        	at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:434)
        

        What about JIRA#15775, can this class loader issue be linked to my problem?

        Thanks, Andreas

        Show
        andydr added a comment - - edited Tim, the application launch works in either case. But the ClassCastException during ejb lookup occurs in JWS. BTW: The ClassCastExc happens even without the read timeout and is also unrelated to specific beans. I.e. with each application startup the ClassCastExc happens for a different bean. java.lang.ClassCastException@4c1e6823] at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:434) What about JIRA#15775, can this class loader issue be linked to my problem? Thanks, Andreas
        Hide
        marina vatkina added a comment -

        Does lookup of the EJBs work in JWS?

        Show
        marina vatkina added a comment - Does lookup of the EJBs work in JWS?
        Hide
        Harshad Vilekar added a comment -

        andydr,

        Try increasing com.sun.corba.ee.transport.ORBTCPTimeouts.

        Also, try increasing com.sun.corba.ee.transport.ORBTCPConnectTimeouts. Defaults are: 250:60000:100:5000 (max is 60 seconds).
        "It controls how the ORB behaves on the client side when attempting to connect to an IOR (the wire rep of an EJB reference)."

        Please see this blog for details about these properties:
        https://blogs.oracle.com/ejcorba/entry/more_on_tcp_timeouts_in
        https://blogs.oracle.com/ejcorba/entry/client_side_transport_timeouts_and

        Show
        Harshad Vilekar added a comment - andydr, Try increasing com.sun.corba.ee.transport.ORBTCPTimeouts. Also, try increasing com.sun.corba.ee.transport.ORBTCPConnectTimeouts. Defaults are: 250:60000:100:5000 (max is 60 seconds). "It controls how the ORB behaves on the client side when attempting to connect to an IOR (the wire rep of an EJB reference)." Please see this blog for details about these properties: https://blogs.oracle.com/ejcorba/entry/more_on_tcp_timeouts_in https://blogs.oracle.com/ejcorba/entry/client_side_transport_timeouts_and
        Hide
        Tim Quinn added a comment -

        Andreas,

        When I asked earlier about the Java Web Start launch and the appclient launch, I meant the successful run of the client not just the initial launch.

        Sorry for the confusion.

        You asked whether this could be related to 15775. I am not sure. In that case the last
        exception in the stack trace is quite different and originates from a different place in the code.

        If you want to specify the properties that Harshad mentioned you can pass them on the URL to launch the app client:

        http://host:port/path?prop=A=1&prop=B=2

        would be equivalent to

        java -DA=1 -DB=2 ...

        on a normal java command line.

        Show
        Tim Quinn added a comment - Andreas, When I asked earlier about the Java Web Start launch and the appclient launch, I meant the successful run of the client not just the initial launch. Sorry for the confusion. You asked whether this could be related to 15775. I am not sure. In that case the last exception in the stack trace is quite different and originates from a different place in the code. If you want to specify the properties that Harshad mentioned you can pass them on the URL to launch the app client: http://host:port/path?prop=A=1&prop=B=2 would be equivalent to java -DA=1 -DB=2 ... on a normal java command line.
        Hide
        Tim Quinn added a comment -

        Andreas,

        I meant to ask about this earlier. You mentioned that if you look up the SLSB only once, lazily, (instead of multiple times)
        and reuse the same reference then you do not get the errors.

        Using multiple look-ups, which look-up causes the error? The first one? Second one? etc.

        Also, you said that the look-ups occur in the main method. Can you attach the two versions of the main method to this issue:
        the one that does multiple look-ups and the one that does a single lazy look-up with reuse? Or, of you'd prefer, you could e-mail
        them to me directly (tjquinn at java dot net) and I would of course keep them confidential.

        That might help us reproduce the problem here. Thanks.

        Show
        Tim Quinn added a comment - Andreas, I meant to ask about this earlier. You mentioned that if you look up the SLSB only once, lazily, (instead of multiple times) and reuse the same reference then you do not get the errors. Using multiple look-ups, which look-up causes the error? The first one? Second one? etc. Also, you said that the look-ups occur in the main method. Can you attach the two versions of the main method to this issue: the one that does multiple look-ups and the one that does a single lazy look-up with reuse? Or, of you'd prefer, you could e-mail them to me directly (tjquinn at java dot net) and I would of course keep them confidential. That might help us reproduce the problem here. Thanks.
        Hide
        Tim Quinn added a comment -

        I have a very simple client that I use for various manual tests. I modified it to create a new InitialContext and then
        use it to look up and use a stateless session bean.

        It gave me the buffer timeout warning Andreas reported the first time I ran it using Java Web Start, but that did not
        cause any class cast errors. And the next 7 times I ran it I saw no such warning.

        (I will attach my Java Web Start trace file.) I ran the same client several times using the appclient command and
        never saw the warning, but clearly that problem is intermittent so perhaps I was just lucky - or unlucky.

        By the way, I did not notice any delay at all during the run with the error.

        Show
        Tim Quinn added a comment - I have a very simple client that I use for various manual tests. I modified it to create a new InitialContext and then use it to look up and use a stateless session bean. It gave me the buffer timeout warning Andreas reported the first time I ran it using Java Web Start, but that did not cause any class cast errors. And the next 7 times I ran it I saw no such warning. (I will attach my Java Web Start trace file.) I ran the same client several times using the appclient command and never saw the warning, but clearly that problem is intermittent so perhaps I was just lucky - or unlucky. By the way, I did not notice any delay at all during the run with the error.
        Hide
        Tim Quinn added a comment -

        Attached Java Web Start trace of app client launch with ORB warning...but it completed successfully.

        Show
        Tim Quinn added a comment - Attached Java Web Start trace of app client launch with ORB warning...but it completed successfully.
        Hide
        andydr added a comment - - edited

        After a couple of code changes I assume the problem relates to
        asynchronous ejb access. Our swing gui loads during startup
        the data concurrently. Some EJBs are access from the main thread
        and other EJBs via the awt-event queue.

        I removed all new Thread(..) and SwingUtilitites.* parts and the startup
        works as expected. Is concurrent access of ejbs not supported?

        In a simple concurrent test I could reproduce the buffer timeout issue but
        not the classcast problem:

        for (int i = 0; i < 30; i++) {
           if (i % 2 == 0) {
             new Thread() {
        
               @Override
               public void run() {
                 ... execute ejb
               }
             }.start();
           } else {
             ... execute ejb
           }
        }
        
        Show
        andydr added a comment - - edited After a couple of code changes I assume the problem relates to asynchronous ejb access. Our swing gui loads during startup the data concurrently. Some EJBs are access from the main thread and other EJBs via the awt-event queue. I removed all new Thread(..) and SwingUtilitites.* parts and the startup works as expected. Is concurrent access of ejbs not supported? In a simple concurrent test I could reproduce the buffer timeout issue but not the classcast problem: for ( int i = 0; i < 30; i++) { if (i % 2 == 0) { new Thread () { @Override public void run() { ... execute ejb } }.start(); } else { ... execute ejb } }
        Hide
        Tim Quinn added a comment -

        Thanks for doing the additional investigation and posting your results.

        The EJB itself on the server side is (if I remember the spec correctly) is threadsafe,
        but the layers on the client side (the ORB particularly) could be a different matter entirely.

        Harshad will know more, but from your findings it seems that using the same client-side
        proxy from multiple threads is triggering the problem.

        Show
        Tim Quinn added a comment - Thanks for doing the additional investigation and posting your results. The EJB itself on the server side is (if I remember the spec correctly) is threadsafe, but the layers on the client side (the ORB particularly) could be a different matter entirely. Harshad will know more, but from your findings it seems that using the same client-side proxy from multiple threads is triggering the problem.
        Hide
        Tim Quinn added a comment -

        based on the latest information from Andreas I have added to the title of this issue and am transferring it to the ORB component.

        I am not sure whether this is a bug or whether the use of a single EJB proxy from multiple threads is unsupported, but Harshad is the right person to answer.

        Show
        Tim Quinn added a comment - based on the latest information from Andreas I have added to the title of this issue and am transferring it to the ORB component. I am not sure whether this is a bug or whether the use of a single EJB proxy from multiple threads is unsupported, but Harshad is the right person to answer.

          People

          • Assignee:
            Harshad Vilekar
            Reporter:
            andydr
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated: