glassfish
  1. glassfish
  2. GLASSFISH-15677

Regression. The deployment of generic-ra.rar failed with a timeout message.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: jca
    • Labels:
      None

      Description

      Build 39. Linux, Windows machines.

      Created a cluster with two instances on the same machine.

      Tried to deploy to the cluster generic-ra.rar. After 10 minutes saw a timeot message.

      Then tried to stop and start a cluster. The cluster did not start, but created a timeout message.

      Please see the attached server.log of one instance. To reproduce the issue just execute this hudson job: http://sqe-hudson.us.oracle.com:8080/hudson/view/Elena_Jobs/job/deployment/

      I saw this issue on Windows, Linux, Mac. This is a regression issue, it was not seen before.

      1. server.log
        46 kB
        easarina

        Activity

        Hide
        easarina added a comment -

        I've tried just wait for a long time and don't do anything after I saw:
        sadmin deploy --target c1 archives_nodb/generic-ra.rar
        No response from Domain Admin Server after 600 seconds.
        The command is either taking too long to complete or the server has failed.
        Please see the server log files for command status.
        Command deploy failed.

        Then in the instance server.log I saw such messages:
        ==================================================

        [#|2011-01-25T11:22:13.695-0800|INFO|glassfish3.1|org.hibernate.validator.util.Version|_ThreadID=22;_ThreadName=Thread-1;|Hibernate Validator 4.1.0.Final|#]

        [#|2011-01-25T11:22:13.715-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=22;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#]

        [#|2011-01-25T11:22:18.528-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=22;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#]

        [#|2011-01-25T11:22:18.560-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> constructor...|#]

        [#|2011-01-25T11:22:18.826-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> setTestName called... name = ConfigPropertyForRA|#]

        [#|2011-01-25T11:22:19.079-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> 001. Simple RA start...|#]

        [#|2011-01-25T11:22:19.080-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> 002. Simple RA start...|#]

        [#|2011-01-25T11:22:19.081-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> 003. Simple RA start...|#]

        [#|2011-01-25T11:22:19.082-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> 004. Simple RA start...|#]

        [#|2011-01-25T11:22:19.113-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=134;_ThreadName=Thread-1;|TestWMWork[1000].start running|#]

        [#|2011-01-25T11:22:20.127-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=133;_ThreadName=Thread-1;|TestWMWork[8888].start running|#]

        [#|2011-01-25T11:22:21.130-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=133;_ThreadName=Thread-1;|TestWMWork[8888].stop running|#]

        [#|2011-01-25T11:22:21.201-0800|INFO|glassfish3.1|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=136;_ThreadName=Thread-1;|JTS5014: Recoverable JTS instance, serverId = [100]|#]

        [#|2011-01-25T11:22:21.313-0800|INFO|glassfish3.1|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|_ThreadID=138;_ThreadName=Thread-1;|Recovery of Inbound Transactions started.|#]

        [#|2011-01-25T11:37:12.440-0800|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-1;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-14848(4).|#]

        [#|2011-01-25T11:37:12.453-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| ex = java.lang.InterruptedException, error code: 0|#]

        [#|2011-01-25T11:37:12.454-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;|[SimpleResourceAdapterImpl] ==> 005. Simple RA start...|#]

        [#|2011-01-25T11:37:14.092-0800|INFO|glassfish3.1|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=122;_ThreadName=Thread-1;|generic-ra was successfully deployed in 901,194 milliseconds.|#]

        [#|2011-01-25T11:37:14.111-0800|SEVERE|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=22;_ThreadName=Thread-1;|service exception
        java.lang.RuntimeException: ClientAbortException: java.nio.channels.ClosedChannelException
        at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:254)
        at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168)
        at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117)
        at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234)
        at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:818)
        at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719)
        at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1008)
        at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
        at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
        at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
        at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
        at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
        at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
        at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
        at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
        at java.lang.Thread.run(Thread.java:662)
        Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
        at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:439)
        at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.flush(GrizzlyOutputBuffer.java:405)
        at com.sun.grizzly.tcp.http11.GrizzlyOutputStream.flush(GrizzlyOutputStream.java:140)
        at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:251)
        ... 17 more
        Caused by: java.nio.channels.ClosedChannelException
        at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
        at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108)
        at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76)
        at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:326)
        at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:398)
        at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:376)
        at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1236)
        at com.sun.grizzly.tcp.Response.action(Response.java:268)
        at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:434)
        ... 20 more

        #]

        [#|2011-01-25T11:37:14.118-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=26;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#]

        [#|2011-01-25T11:37:14.719-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=136;_ThreadName=Thread-1;|TestWMWork[9999].start running|#]

        [#|2011-01-25T11:37:15.730-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=136;_ThreadName=Thread-1;|TestWMWork[9999].stop running|#]

        [#|2011-01-25T11:37:15.732-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=134;_ThreadName=Thread-1;|TestWMWork[1000].stop running|#]

        Show
        easarina added a comment - I've tried just wait for a long time and don't do anything after I saw: sadmin deploy --target c1 archives_nodb/generic-ra.rar No response from Domain Admin Server after 600 seconds. The command is either taking too long to complete or the server has failed. Please see the server log files for command status. Command deploy failed. Then in the instance server.log I saw such messages: ================================================== [#|2011-01-25T11:22:13.695-0800|INFO|glassfish3.1|org.hibernate.validator.util.Version|_ThreadID=22;_ThreadName=Thread-1;|Hibernate Validator 4.1.0.Final|#] [#|2011-01-25T11:22:13.715-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=22;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#] [#|2011-01-25T11:22:18.528-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=22;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#] [#|2011-01-25T11:22:18.560-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> constructor...|#] [#|2011-01-25T11:22:18.826-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> setTestName called... name = ConfigPropertyForRA|#] [#|2011-01-25T11:22:19.079-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> 001. Simple RA start...|#] [#|2011-01-25T11:22:19.080-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> 002. Simple RA start...|#] [#|2011-01-25T11:22:19.081-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> 003. Simple RA start...|#] [#|2011-01-25T11:22:19.082-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> 004. Simple RA start...|#] [#|2011-01-25T11:22:19.113-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=134;_ThreadName=Thread-1;|TestWMWork [1000] .start running|#] [#|2011-01-25T11:22:20.127-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=133;_ThreadName=Thread-1;|TestWMWork [8888] .start running|#] [#|2011-01-25T11:22:21.130-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=133;_ThreadName=Thread-1;|TestWMWork [8888] .stop running|#] [#|2011-01-25T11:22:21.201-0800|INFO|glassfish3.1|javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions|_ThreadID=136;_ThreadName=Thread-1;|JTS5014: Recoverable JTS instance, serverId = [100] |#] [#|2011-01-25T11:22:21.313-0800|INFO|glassfish3.1|javax.enterprise.resource.resourceadapter.com.sun.enterprise.connectors.inbound|_ThreadID=138;_ThreadName=Thread-1;|Recovery of Inbound Transactions started.|#] [#|2011-01-25T11:37:12.440-0800|WARNING|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=12;_ThreadName=Thread-1;|GRIZZLY0023: Interrupting idle Thread: http-thread-pool-14848(4).|#] [#|2011-01-25T11:37:12.453-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| ex = java.lang.InterruptedException, error code: 0|#] [#|2011-01-25T11:37:12.454-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=122;_ThreadName=Thread-1;| [SimpleResourceAdapterImpl] ==> 005. Simple RA start...|#] [#|2011-01-25T11:37:14.092-0800|INFO|glassfish3.1|javax.enterprise.system.tools.admin.org.glassfish.deployment.admin|_ThreadID=122;_ThreadName=Thread-1;|generic-ra was successfully deployed in 901,194 milliseconds.|#] [#|2011-01-25T11:37:14.111-0800|SEVERE|glassfish3.1|com.sun.grizzly.config.GrizzlyServiceListener|_ThreadID=22;_ThreadName=Thread-1;|service exception java.lang.RuntimeException: ClientAbortException: java.nio.channels.ClosedChannelException at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:254) at com.sun.grizzly.tcp.http11.GrizzlyAdapter.service(GrizzlyAdapter.java:168) at com.sun.enterprise.v3.server.HK2Dispatcher.dispath(HK2Dispatcher.java:117) at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:234) at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:818) at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:719) at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1008) at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225) at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104) at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90) at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79) at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54) at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59) at com.sun.grizzly.ContextTask.run(ContextTask.java:71) at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532) at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513) at java.lang.Thread.run(Thread.java:662) Caused by: ClientAbortException: java.nio.channels.ClosedChannelException at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:439) at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.flush(GrizzlyOutputBuffer.java:405) at com.sun.grizzly.tcp.http11.GrizzlyOutputStream.flush(GrizzlyOutputStream.java:140) at com.sun.enterprise.v3.admin.AdminAdapter.service(AdminAdapter.java:251) ... 17 more Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:108) at com.sun.grizzly.util.OutputWriter.flushChannel(OutputWriter.java:76) at com.sun.grizzly.http.SocketChannelOutputBuffer.flushChannel(SocketChannelOutputBuffer.java:326) at com.sun.grizzly.http.SocketChannelOutputBuffer.flushBuffer(SocketChannelOutputBuffer.java:398) at com.sun.grizzly.http.SocketChannelOutputBuffer.flush(SocketChannelOutputBuffer.java:376) at com.sun.grizzly.http.ProcessorTask.action(ProcessorTask.java:1236) at com.sun.grizzly.tcp.Response.action(Response.java:268) at com.sun.grizzly.tcp.http11.GrizzlyOutputBuffer.doFlush(GrizzlyOutputBuffer.java:434) ... 20 more #] [#|2011-01-25T11:37:14.118-0800|INFO|glassfish3.1|org.hibernate.validator.engine.resolver.DefaultTraversableResolver|_ThreadID=26;_ThreadName=Thread-1;|Instantiated an instance of org.hibernate.validator.engine.resolver.JPATraversableResolver.|#] [#|2011-01-25T11:37:14.719-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=136;_ThreadName=Thread-1;|TestWMWork [9999] .start running|#] [#|2011-01-25T11:37:15.730-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=136;_ThreadName=Thread-1;|TestWMWork [9999] .stop running|#] [#|2011-01-25T11:37:15.732-0800|INFO|glassfish3.1|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=134;_ThreadName=Thread-1;|TestWMWork [1000] .stop running|#]
        Hide
        Hong Zhang added a comment -

        Yes, this stack trace was in the server.log you attached earlier too. We had some discussions on this and it seems due to transaction timeout was expired, but not sure why the transaction was expired yet.

        I had spent some time trying to further narrow down the steps and the following seems the simplest set of steps to reproduce the problem:

        asadmin start-domain
        asadmin create-cluster c1
        asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=18080:HTTP_SSL_LISTENER_PORT=18181:IIOP_SSL_LISTENER_PORT=13800:IIOP_LISTENER_PORT=13700:JMX_SYSTEM_CONNECTOR_PORT=17676:IIOP_SSL_MUTUALAUTH_PORT=13801:JMS_PROVIDER_PORT=18686:ASADMIN_LISTENER_PORT=14848 in1
        asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=28080:HTTP_SSL_LISTENER_PORT=28181:IIOP_SSL_LISTENER_PORT=23800:IIOP_LISTENER_PORT=23700:JMX_SYSTEM_CONNECTOR_PORT=27676:IIOP_SSL_MUTUALAUTH_PORT=23801:JMS_PROVIDER_PORT=28686:ASADMIN_LISTENER_PORT=24848 in2
        asadmin create-resource-ref --target c1 jdbc/__default
        asadmin start-local-instance --node localhost-domain1 in1
        asadmin start-local-instance --node localhost-domain1 in2
        asadmin deploy --target c1 generic-ra.rar

        A few observations:
        1. If we remove the create-resource-ref step, the deployment went through successfully.
        2. It does not matter if the create-resource-ref happens before or after the cluster instances are started.
        3. I tried to deploy an enterprise application with ejb and servlet and the deployment went through successfully.

        So somehow the create-resource-ref step did not work well with the particular rar application, I will assign to Jagadish for further evaluation.

        Show
        Hong Zhang added a comment - Yes, this stack trace was in the server.log you attached earlier too. We had some discussions on this and it seems due to transaction timeout was expired, but not sure why the transaction was expired yet. I had spent some time trying to further narrow down the steps and the following seems the simplest set of steps to reproduce the problem: asadmin start-domain asadmin create-cluster c1 asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=18080:HTTP_SSL_LISTENER_PORT=18181:IIOP_SSL_LISTENER_PORT=13800:IIOP_LISTENER_PORT=13700:JMX_SYSTEM_CONNECTOR_PORT=17676:IIOP_SSL_MUTUALAUTH_PORT=13801:JMS_PROVIDER_PORT=18686:ASADMIN_LISTENER_PORT=14848 in1 asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=28080:HTTP_SSL_LISTENER_PORT=28181:IIOP_SSL_LISTENER_PORT=23800:IIOP_LISTENER_PORT=23700:JMX_SYSTEM_CONNECTOR_PORT=27676:IIOP_SSL_MUTUALAUTH_PORT=23801:JMS_PROVIDER_PORT=28686:ASADMIN_LISTENER_PORT=24848 in2 asadmin create-resource-ref --target c1 jdbc/__default asadmin start-local-instance --node localhost-domain1 in1 asadmin start-local-instance --node localhost-domain1 in2 asadmin deploy --target c1 generic-ra.rar A few observations: 1. If we remove the create-resource-ref step, the deployment went through successfully. 2. It does not matter if the create-resource-ref happens before or after the cluster instances are started. 3. I tried to deploy an enterprise application with ejb and servlet and the deployment went through successfully. So somehow the create-resource-ref step did not work well with the particular rar application, I will assign to Jagadish for further evaluation.
        Hide
        Jagadish added a comment -

        The use-case works fine in b32-dec-8-2010 nightly and starts failing
        from b32-dec-10-2010 nightly.

        I could narrow down to the svn revision "43607".

        It looks like the bug fix for 14809 is causing (or exposing) this issue.
        http://java.net/jira/browse/GLASSFISH-14809

        Once I reverted the changes for the above fix, the use-case reported in
        issue 15677 works fine.
        http://java.net/jira/browse/GLASSFISH-15677

        Show
        Jagadish added a comment - The use-case works fine in b32-dec-8-2010 nightly and starts failing from b32-dec-10-2010 nightly. I could narrow down to the svn revision "43607". It looks like the bug fix for 14809 is causing (or exposing) this issue. http://java.net/jira/browse/GLASSFISH-14809 Once I reverted the changes for the above fix, the use-case reported in issue 15677 works fine. http://java.net/jira/browse/GLASSFISH-15677
        Hide
        Jagadish added a comment -

        1. How bad is its impact? (Severity)

        • Identify why the fix needs to occur now:
          o Is a regression in SQE test-case
          o Fix for GLASSFISH-14809 changed the way in which transaction manager starts recovery process which in turn causes this issue.
          2. How often does it happen? (Frequency)
          o In cluster environment, during deployment of a resource-adapter that specifically tries to do transactional activity during ResourceAdapter.start().
          3. How much effort is required to fix it? (Cost)
          o Fix would be to relax the synchronization of resource-adapter deployment from only one resource-adapter deployment at a time, to synchronization per resource-
          adapter so that multiple RARs can be bootstrapped simultaneously, but avoiding same RAR to be bootstrapped again due to calls from multiple threads.
          4. What is the risk of fixing it? (Risk)
          o Minimal, all tests pass [ QL (Web, Classic), connector-dev, jdbc-dev, connector-standalone-cts, resources-admin-cli, connector-sqe and the test-case
          reported by SQE]
          5. Does a work around for the issue exist? Can the workaround be reasonably employed by the end user?
          o Workaround would be to avoid doing transactional activity during ResourceAdapter.start().
          o Refer the old references where we have advised users not to do transactional activity during ResourceAdapter.start() :
          http://markmail.org/message/ogc7qndhaywfkdrp#query:+page:1+mid:kyyzpcexusbnv7ri+state:results
          6. If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes?
          o This particular use-case can be fixed as the fix is less intrusive and all tests pass.
          o There is another use-case where jms-resources are present in the cluster and the deployed resource-adapter does transactional work
          in ResourceAdapter.start(). We need to release note the issue advising developers/users, not to do transactional work/activity in ResourceAdapter.start()
          Re-opened the issue http://java.net/jira/browse/GLASSFISH-3916 raised during the original forum discussion stated above for release noting the same.
        Show
        Jagadish added a comment - 1. How bad is its impact? (Severity) Identify why the fix needs to occur now: o Is a regression in SQE test-case o Fix for GLASSFISH-14809 changed the way in which transaction manager starts recovery process which in turn causes this issue. 2. How often does it happen? (Frequency) o In cluster environment, during deployment of a resource-adapter that specifically tries to do transactional activity during ResourceAdapter.start(). 3. How much effort is required to fix it? (Cost) o Fix would be to relax the synchronization of resource-adapter deployment from only one resource-adapter deployment at a time, to synchronization per resource- adapter so that multiple RARs can be bootstrapped simultaneously, but avoiding same RAR to be bootstrapped again due to calls from multiple threads. 4. What is the risk of fixing it? (Risk) o Minimal, all tests pass [ QL (Web, Classic), connector-dev, jdbc-dev, connector-standalone-cts, resources-admin-cli, connector-sqe and the test-case reported by SQE] 5. Does a work around for the issue exist? Can the workaround be reasonably employed by the end user? o Workaround would be to avoid doing transactional activity during ResourceAdapter.start(). o Refer the old references where we have advised users not to do transactional activity during ResourceAdapter.start() : http://markmail.org/message/ogc7qndhaywfkdrp#query:+page:1+mid:kyyzpcexusbnv7ri+state:results 6. If the issue is not fixed should the issue and its workaround (if applicable) be described in the Release Notes? o This particular use-case can be fixed as the fix is less intrusive and all tests pass. o There is another use-case where jms-resources are present in the cluster and the deployed resource-adapter does transactional work in ResourceAdapter.start(). We need to release note the issue advising developers/users, not to do transactional work/activity in ResourceAdapter.start() Re-opened the issue http://java.net/jira/browse/GLASSFISH-3916 raised during the original forum discussion stated above for release noting the same.
        Hide
        Jagadish added a comment -

        svn log -v -r 44745

        Modified Paths:
        ---------------
        trunk/v3/connectors/connectors-runtime/src/main/java/com/sun/enterprise/connectors/service/ResourceAdapterAdminServiceImpl.java
        trunk/v3/connectors/connectors-runtime/src/main/java/com/sun/enterprise/connectors/ConnectorRegistry.java

        Fix will be available in (first) RC build.

        Tested the following scenario and is working fine without deadlock/timeout issue.

        Install GlassFish
        $AS_HOME/bin/asadmin start-domain
        $AS_HOME/bin/asadmin create-cluster c1
        $AS_HOME/bin/asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=18080:HTTP_SSL_LIST
        ENER_PORT=18181:IIOP_SSL_LISTENER_PORT=13800:IIOP_LISTENER_PORT=13700:JMX_SYSTEM_CONNECTOR_PORT=17676:IIOP_SSL_MU
        TUALAUTH_PORT=13801:JMS_PROVIDER_PORT=18686:ASADMIN_LISTENER_PORT=14848 in1

        $AS_HOME/bin/asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=28080:HTTP_SSL_LIS
        TENER_PORT=28181:IIOP_SSL_LISTENER_PORT=23800:IIOP_LISTENER_PORT=23700:JMX_SYSTEM_CONNECTOR_PORT=27676:IIOP_SSL_M
        UTUALAUTH_PORT=23801:JMS_PROVIDER_PORT=28686:ASADMIN_LISTENER_PORT=24848 in2
        $AS_HOME/bin/asadmin start-local-instance in1
        $AS_HOME/bin/asadmin start-local-instance in2
        $AS_HOME/bin/asadmin deploy --target c1 generic-ra.rar
        $AS_HOME/bin/asadmin stop-cluster c1
        $AS_HOME/bin/asadmin start-cluster c1

        NOTE : This test resource-adapter tries to start a transactional work in ResourceAdapter.start() which is not a recommended practise.

        Show
        Jagadish added a comment - svn log -v -r 44745 Modified Paths: --------------- trunk/v3/connectors/connectors-runtime/src/main/java/com/sun/enterprise/connectors/service/ResourceAdapterAdminServiceImpl.java trunk/v3/connectors/connectors-runtime/src/main/java/com/sun/enterprise/connectors/ConnectorRegistry.java Fix will be available in (first) RC build. Tested the following scenario and is working fine without deadlock/timeout issue. Install GlassFish $AS_HOME/bin/asadmin start-domain $AS_HOME/bin/asadmin create-cluster c1 $AS_HOME/bin/asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=18080:HTTP_SSL_LIST ENER_PORT=18181:IIOP_SSL_LISTENER_PORT=13800:IIOP_LISTENER_PORT=13700:JMX_SYSTEM_CONNECTOR_PORT=17676:IIOP_SSL_MU TUALAUTH_PORT=13801:JMS_PROVIDER_PORT=18686:ASADMIN_LISTENER_PORT=14848 in1 $AS_HOME/bin/asadmin create-local-instance --cluster c1 --systemproperties HTTP_LISTENER_PORT=28080:HTTP_SSL_LIS TENER_PORT=28181:IIOP_SSL_LISTENER_PORT=23800:IIOP_LISTENER_PORT=23700:JMX_SYSTEM_CONNECTOR_PORT=27676:IIOP_SSL_M UTUALAUTH_PORT=23801:JMS_PROVIDER_PORT=28686:ASADMIN_LISTENER_PORT=24848 in2 $AS_HOME/bin/asadmin start-local-instance in1 $AS_HOME/bin/asadmin start-local-instance in2 $AS_HOME/bin/asadmin deploy --target c1 generic-ra.rar $AS_HOME/bin/asadmin stop-cluster c1 $AS_HOME/bin/asadmin start-cluster c1 NOTE : This test resource-adapter tries to start a transactional work in ResourceAdapter.start() which is not a recommended practise.

          People

          • Assignee:
            Jagadish
            Reporter:
            easarina
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: