glassfish
  1. glassfish
  2. GLASSFISH-20992

Clustered deployment sometimes fails with error "Command disable failed on server instance xxx: remote failure: Application not registered Command deploy failed."

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 4.0
    • Fix Version/s: None
    • Component/s: deployment
    • Labels:
      None
    • Environment:

      Linux, JDK 1.7.0_45

      Description

      Sometimes when redeploying an application, the deploy fails with the following error:

      Failure: Command disable failed on server instance xxxxxx: remote failure: Application not registered Command deploy failed.

      Repeated attempts always fail. After the error appears once it seems like nothing can be deployed.

      Sometimes the error is:

      Error occurred during deployment: Keys cannot be duplicate. Old value of this key property, nullwill be retained. 
      

      Here is the command:

      asadmin deploy --force=true --deploymentorder 300 --target yyyyy 
      

      Sometimes by trial and error I can get the deploy to work. Here is what I've tried:

      1. Stop the domain then remove the "generated" and "osgi-cache" and re-start the domain
      2. Stop the cluster and the domain then remove the "generated" and "osgi-cache" from the domain and the nodes and re-start the domain and cluster
      3. Undeploy the problem app, then redeploy it. However, sometimes the app is listed as having been un-deployed already and this fails
      4. Undeploy everything that can be undeployed. Manually delete the applications that could not be undeployed via asadmin from the applications folder and manually edit the domain.xml to remove references to those apps.

        Activity

        Hide
        Hong Zhang added a comment -

        Can you provide the test application and the set of steps for us to reproduce from our side?

        Show
        Hong Zhang added a comment - Can you provide the test application and the set of steps for us to reproduce from our side?
        Hide
        electricsam added a comment -

        It's not always the same app.

        Let me describe our set-up. We have a complex web service. It is comprised of about 60 ears. We have a nightly script which will deploy / re-deploy any apps that have changed (varies between 5-60).

        This error occurs probably 3-4 times a month and requires manual intervention and possibly an outage to resolve.

        We are running a cluster of 2 nodes. The DAS and each of the nodes are on their own separate boxes.

        Other than all being ears, there is no commonality between the apps. I have not been able to find one scenario that will consistently reproduce the issue.

        I will continue to look for patterns, but so far none are obvious.

        Show
        electricsam added a comment - It's not always the same app. Let me describe our set-up. We have a complex web service. It is comprised of about 60 ears. We have a nightly script which will deploy / re-deploy any apps that have changed (varies between 5-60). This error occurs probably 3-4 times a month and requires manual intervention and possibly an outage to resolve. We are running a cluster of 2 nodes. The DAS and each of the nodes are on their own separate boxes. Other than all being ears, there is no commonality between the apps. I have not been able to find one scenario that will consistently reproduce the issue. I will continue to look for patterns, but so far none are obvious.

          People

          • Assignee:
            Hong Zhang
            Reporter:
            electricsam
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated: