glassfish
  1. glassfish
  2. GLASSFISH-17386

On Windows AdminTask Hangs After Succesfully Executing Some Commands

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 3.1.1
    • Fix Version/s: None
    • Component/s: other
    • Labels:
      None
    • Environment:

      Windows 7

      Description

      Remote execution asadmin tasks hang after successfully executing some commands like
      create-instance
      or
      create-jms-resource --target cluster-1 --restype javax.jms.TopicConnectionFactory ...

      The ant script can be interrupted with Ctrl-C and repeated.
      The ant script will resume with the next task as the previous resource had been created successfully even though the previous task did not pass control back to ant.

      This problems do not occur on Linux.

      Testcase (attached): gftestcase.zip
      ant setup #does setup a cluster with a connection factory and a topic both with the cluster as the target.
      ant clean #does teardown the cluster and the connection factory and topic

        Issue Links

          Activity

          Hide
          Tom Mueller added a comment - - edited

          This problem might be related to issue GLASSFISH-12777 which explains a difference between Windows and other systems in the way that it deals with input/output streams and process hierarchies.

          Marking this as ide-integration because we don't have a subcategory for the Ant tasks.

          Show
          Tom Mueller added a comment - - edited This problem might be related to issue GLASSFISH-12777 which explains a difference between Windows and other systems in the way that it deals with input/output streams and process hierarchies. Marking this as ide-integration because we don't have a subcategory for the Ant tasks.
          Hide
          Bhavanishankar added a comment -

          Recatorizing.

          Show
          Bhavanishankar added a comment - Recatorizing.
          Hide
          Manfred Riem added a comment -

          We are using the Glassfish optional ANT tasks (version 3.1.1) and when we use
          the AdminTask it never returns and thus our ANT script is unable to continue.

          Note it happens on more than one call of the AdminTask. The snippet below
          is one of the circumstances it fails. If you need more information, please let
          me know

          <taskdef
          name="glassfish-admin"
          classname="org.glassfish.ant.tasks.AdminTask"
          classpath="$

          {dependency.base.dir}

          /jars/ant-tasks.jar" />
          <glassfish-admin
          installDir="$

          {container.home}

          "
          command="--user $

          {glassfish.admin.user}

          --passwordfile $

          {glassfish.password.file}

          --host
          $

          {glassfish.admin.host}

          --port $

          {glassfish.custom.admin.port}

          create-cluster
          --systemproperties ASADMIN_LISTENER_PORT=$

          {glassfish.cluster.admin.port}

          $

          {glassfish.cluster}

          "/>

          Show
          Manfred Riem added a comment - We are using the Glassfish optional ANT tasks (version 3.1.1) and when we use the AdminTask it never returns and thus our ANT script is unable to continue. Note it happens on more than one call of the AdminTask. The snippet below is one of the circumstances it fails. If you need more information, please let me know <taskdef name="glassfish-admin" classname="org.glassfish.ant.tasks.AdminTask" classpath="$ {dependency.base.dir} /jars/ant-tasks.jar" /> <glassfish-admin installDir="$ {container.home} " command="--user $ {glassfish.admin.user} --passwordfile $ {glassfish.password.file} --host $ {glassfish.admin.host} --port $ {glassfish.custom.admin.port} create-cluster --systemproperties ASADMIN_LISTENER_PORT=$ {glassfish.cluster.admin.port} $ {glassfish.cluster} "/>
          Hide
          Manfred Riem added a comment -

          In the Mojarra ANT build file we use the above mentioned call to create a cluster added to the mojarra domain we use for integration testing. Once this call comes around it calls out and then it hangs.

          Since we are trying to use these tasks to refactor the build process so we don't have anything specific to Windows in it I do not have a SVN url can look at for the entire build script. A small ANT file like below should allow you to reproduce the issue.

          <target name="create-cluster">
          snippet from above
          </target>

          Show
          Manfred Riem added a comment - In the Mojarra ANT build file we use the above mentioned call to create a cluster added to the mojarra domain we use for integration testing. Once this call comes around it calls out and then it hangs. Since we are trying to use these tasks to refactor the build process so we don't have anything specific to Windows in it I do not have a SVN url can look at for the entire build script. A small ANT file like below should allow you to reproduce the issue. <target name="create-cluster"> snippet from above </target>
          Hide
          Byron Nevins added a comment -

          This is THE CLASSIC DEADLOCK in operation.

          I can easily reproduce this if I have

          AS_DEBUG=true

          in my environment.
          If I have, instead,

          AS_TERSE=true

          then everything works perfectly.

          ============

          The thread dump in the attached file from Manfred reveals the bug in AdminTask.java

          See my comment in <common-utils> ProcessStreamDrainer.java:

          /**

          • If you don't drain a process' stdout and stderr it will cause a deadlock after a few hundred bytes of output.
          • At that point the Process is blocked because its stdout and/or stderr buffer is full and it is waiting for the Java caller
          • to drain it. Meanwhile the Java program is blocked waiting on the external process.
          • This class makes this common, but messy and tricky, procedure easier.
          • It creates 2 threads that drain output on stdout and stderr of the external process.
          • <p> Sample Code:
            *
            ===================

          That's exactly what happens here. Here is the code from AdminTask. Notice how it does not create any threads:

          File asadmin = getAsAdmin(f);
          Process pr = Runtime.getRuntime().exec(asadmin.getAbsolutePath() + " " + commandExec);

          BufferedReader error = new BufferedReader(new InputStreamReader(pr.getErrorStream()));
          String errorLine=null;
          while((errorLine=error.readLine()) != null)

          { log(errorLine); }

          BufferedReader input = new BufferedReader(new InputStreamReader(pr.getInputStream()));
          String inputLine=null;
          while((inputLine=input.readLine()) != null)

          { log(inputLine); }

          ============================

          Show
          Byron Nevins added a comment - This is THE CLASSIC DEADLOCK in operation. I can easily reproduce this if I have AS_DEBUG=true in my environment. If I have, instead, AS_TERSE=true then everything works perfectly. ============ The thread dump in the attached file from Manfred reveals the bug in AdminTask.java See my comment in <common-utils> ProcessStreamDrainer.java: /** If you don't drain a process' stdout and stderr it will cause a deadlock after a few hundred bytes of output. At that point the Process is blocked because its stdout and/or stderr buffer is full and it is waiting for the Java caller to drain it. Meanwhile the Java program is blocked waiting on the external process. This class makes this common, but messy and tricky, procedure easier. It creates 2 threads that drain output on stdout and stderr of the external process. <p> Sample Code: * =================== That's exactly what happens here. Here is the code from AdminTask. Notice how it does not create any threads: File asadmin = getAsAdmin(f); Process pr = Runtime.getRuntime().exec(asadmin.getAbsolutePath() + " " + commandExec); BufferedReader error = new BufferedReader(new InputStreamReader(pr.getErrorStream())); String errorLine=null; while((errorLine=error.readLine()) != null) { log(errorLine); } BufferedReader input = new BufferedReader(new InputStreamReader(pr.getInputStream())); String inputLine=null; while((inputLine=input.readLine()) != null) { log(inputLine); } ============================
          Hide
          Byron Nevins added a comment -

          Why do you see this in Windows but not Linux? Because Windows' buffer is much smaller.

          You would see it in Linux if you simply forced create-instance to, say, dump out a few MB of text to stdout.

          Show
          Byron Nevins added a comment - Why do you see this in Windows but not Linux? Because Windows' buffer is much smaller. You would see it in Linux if you simply forced create-instance to, say, dump out a few MB of text to stdout.
          Hide
          Byron Nevins added a comment -

          Moral:

          Thou shalt not EVER create a process and not attach threads to drain stdout and stderr!!

          Use my class in common-util if possible. If not possible then copy & paste it and use it. It's very easy to use.

          com.sun.enterprise.universal.process.ProcessManager

          Show
          Byron Nevins added a comment - Moral: Thou shalt not EVER create a process and not attach threads to drain stdout and stderr!! Use my class in common-util if possible. If not possible then copy & paste it and use it. It's very easy to use. com.sun.enterprise.universal.process.ProcessManager
          Hide
          scatari added a comment -

          We should strongly consider this fix in custom ANT task for 3.1.2. Thank you Byron for the detailed evaluation.

          Show
          scatari added a comment - We should strongly consider this fix in custom ANT task for 3.1.2. Thank you Byron for the detailed evaluation.
          Hide
          sirajg added a comment -

          Duplicate of : GLASSFISH-12764

          Show
          sirajg added a comment - Duplicate of : GLASSFISH-12764

            People

            • Assignee:
              sirajg
              Reporter:
              rgirsten
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: