glassfish
  1. glassfish
  2. GLASSFISH-18763

EJB bundle hangs on stopping when the bundle is updated

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2
    • Fix Version/s: None
    • Component/s: OSGi-JavaEE
    • Labels:
      None

      Description

      Bundles that contain EJBs deadlock on "stopping" when the bundle is restarted for an update. Glassfish has to be killed (shutdown doesn't work) and the bundle cache cleaned to resolve the issue.
      Probably not related, but the update is triggered by DeploymentAdmin (Apache ACE). Other bundles (non-ejb) work without problem.

      The attached bundle is an example of a bundle that always hangs when updated.

        Activity

        Hide
        Sanjeeb Sahoo added a comment -

        I am using osgi-javaee-base 1.0.2 which has some deadlock fix. I have tried deploying and updating your ejb bundle several times and I could not reproduce. Knowing timing issues, I am not surprised.

        I am afraid I am not very inclined to change the locking model unless I know what's exactly going on. Lack of any framework API to lock a bundle does not make things easier. As per the requirements of WAB and other enterprise applications spec, when an enterprise bundle is stopped, the extender must undeploy synchronously upon receiving the Bundle.STOPPING event, which means there is some amount locking that needs to happen in a synchronous listener. On the other hand, the spec allows for bundle to be deployed asynchronously. So, we had seen some deadlocks when bundles were started and stopped in quick succession. Those deadlocks were (successfully) broken by introduction of a timeout and cancellation facility in our undeployer thread. We did that in osgi-javaee-base:1.0.2 when we fixed GLASSFISH-18159. The original thread dump you have noticed should not occur after our fix. You have also mentioned that after upgrading to osgi-javaee-base 1.0.2, the behavior slightly changed. Could you tell me how the behavior changed? What's the new thread dump? Part of the reason for me being reluctant to make drastic changes is that we have other people using it and they seem to be fine. So, I would like to get to the root of the current problem before really doing something of that sort.

        Show
        Sanjeeb Sahoo added a comment - I am using osgi-javaee-base 1.0.2 which has some deadlock fix. I have tried deploying and updating your ejb bundle several times and I could not reproduce. Knowing timing issues, I am not surprised. I am afraid I am not very inclined to change the locking model unless I know what's exactly going on. Lack of any framework API to lock a bundle does not make things easier. As per the requirements of WAB and other enterprise applications spec, when an enterprise bundle is stopped, the extender must undeploy synchronously upon receiving the Bundle.STOPPING event, which means there is some amount locking that needs to happen in a synchronous listener. On the other hand, the spec allows for bundle to be deployed asynchronously. So, we had seen some deadlocks when bundles were started and stopped in quick succession. Those deadlocks were (successfully) broken by introduction of a timeout and cancellation facility in our undeployer thread. We did that in osgi-javaee-base:1.0.2 when we fixed GLASSFISH-18159 . The original thread dump you have noticed should not occur after our fix. You have also mentioned that after upgrading to osgi-javaee-base 1.0.2, the behavior slightly changed. Could you tell me how the behavior changed? What's the new thread dump? Part of the reason for me being reluctant to make drastic changes is that we have other people using it and they seem to be fine. So, I would like to get to the root of the current problem before really doing something of that sort.
        Hide
        Sanjeeb Sahoo added a comment -

        Pl. accept my apology for not investigating yet. I am busy in another higher priority task and hope to get back to this early next week. Thanks for your patience.

        Show
        Sanjeeb Sahoo added a comment - Pl. accept my apology for not investigating yet. I am busy in another higher priority task and hope to get back to this early next week. Thanks for your patience.
        Hide
        pbakker added a comment -

        I have attached the agenda.api bundle, you should now be able to deploy them. Thanks for looking into it!

        Show
        pbakker added a comment - I have attached the agenda.api bundle, you should now be able to deploy them. Thanks for looking into it!
        Hide
        Sanjeeb Sahoo added a comment -

        Thanks for this instructions. I will try them out. Although I have many osgi/ejb bundles with me, I would rather use what you used to reproduce than using my own. I had mentioned in my comment on 25 May that the set of bundles you have attached to this issue does not include some required bundles and I had asked for the missing bundles to be attached. Could you do the same? Thanks again.

        Show
        Sanjeeb Sahoo added a comment - Thanks for this instructions. I will try them out. Although I have many osgi/ejb bundles with me, I would rather use what you used to reproduce than using my own. I had mentioned in my comment on 25 May that the set of bundles you have attached to this issue does not include some required bundles and I had asked for the missing bundles to be attached. Could you do the same? Thanks again.
        Hide
        pbakker added a comment -

        I have spent a few hours creating a test case as simple as possible. The problem is I keep on seeing different results, and it often doesn't brake (but sometimes it does). Although not the simplest way, but the most effective way to test this is by using Apache ACE to deploy bundles.

        1) Download and install ACE (ace.apache.org).
        2) Add two system properties to glass fish (e.g. in the domain.xml)
        <system-property name="discovery" value="http://localhost:8080"></system-property>
        <system-property name="identification" value="glassfish"></system-property>
        The discovery url is the url where ACE is running.
        3) Upload an EJB bundle to ACE and deploy it to GF
        4) Upload a new version of the EJB bundle (just change the filename and version in the manifest)
        5) "Save" the new version in the ACE UI so it will be pushed to GF
        6) The EJB bundle is now deadlocked in "stopping" in most cases, if not, just update again.

        Sorry the test case isn't that easy to execute, but as marrs said it is often hard to spot concurrency issues from automated tests because timings are very different.
        You can use the ACE management agent without ACE too by using a file url in the discovery property, but this test fails a lot less often (on my machine). It may be hard to reproduce the issue that way. The file url should specify a directory where you "install" bundles. Start with one EJB bundle and let it deploy, then add a second bundle to the directory with a higher version in the manifest.
        <system-property name="discovery" value="file:///Users/paul/Desktop/glassfish/"></system-property>

        Show
        pbakker added a comment - I have spent a few hours creating a test case as simple as possible. The problem is I keep on seeing different results, and it often doesn't brake (but sometimes it does). Although not the simplest way, but the most effective way to test this is by using Apache ACE to deploy bundles. 1) Download and install ACE (ace.apache.org). 2) Add two system properties to glass fish (e.g. in the domain.xml) <system-property name="discovery" value="http://localhost:8080"></system-property> <system-property name="identification" value="glassfish"></system-property> The discovery url is the url where ACE is running. 3) Upload an EJB bundle to ACE and deploy it to GF 4) Upload a new version of the EJB bundle (just change the filename and version in the manifest) 5) "Save" the new version in the ACE UI so it will be pushed to GF 6) The EJB bundle is now deadlocked in "stopping" in most cases, if not, just update again. Sorry the test case isn't that easy to execute, but as marrs said it is often hard to spot concurrency issues from automated tests because timings are very different. You can use the ACE management agent without ACE too by using a file url in the discovery property, but this test fails a lot less often (on my machine). It may be hard to reproduce the issue that way. The file url should specify a directory where you "install" bundles. Start with one EJB bundle and let it deploy, then add a second bundle to the directory with a higher version in the manifest. <system-property name="discovery" value="file:///Users/paul/Desktop/glassfish/"></system-property>

          People

          • Assignee:
            Sanjeeb Sahoo
            Reporter:
            pbakker
          • Votes:
            3 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated: