glassfish
  1. glassfish
  2. GLASSFISH-19498

Java Web Start generated files intermittently being deleted, causing Web Start applications to fail

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.2.2
    • Fix Version/s: 4.0_b71
    • Component/s: standalone_client
    • Labels:
      None
    • Environment:

      Java7u9 64Bit, Linux Fedora 17

      Description

      Somewhat rarely, our WebStart application's <glassfish_home>/glassfish/domains/domain1/generated/xml directory is getting deleted. We can not seem to determine the cause, or reproduce consistently. But it does happen fairly often. We have about 10 QA machines, and this happens to 1 or 2 a week it seems.

      It seems to happen most of the time after a machine reboot. We have 3 WebStart applications, and all 3 of their generated/xml directories, and all the contents completely disappear. But our web applications generated/xml directories remain (Although they are empty, but they were always empty).

      There was another report of a similar issue, but it was closed as can not reproduce. GLASSFISH-17341

      Any thoughts on how we can help debug this? I'm sure it will be tricky to reproduce?

      Here is the last message before server shutdown (related to the Web Start app)
      [#|2013-01-07T08:55:06.276-0500|INFO|glassfish3.1.2|javax.enterprise.system.container.appclient.org.glassfish.appclient.server.core|_Threa
      dID=9658;_ThreadName=Thread-2;|ACDEPL104: Java Web Start services stopped for the app client POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-termina
      l.jar|#]

      And here is an error after the reboot:

      [#|2013-01-07T09:08:25.824-0500|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1
      ;_ThreadName=Thread-2;|Exception while invoking class org.glassfish.appclient.server.core.AppClientServerApplication start method
      java.lang.RuntimeException: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo$1.run(JavaWebStartInfo.java:259)
      at org.glassfish.appclient.server.core.jws.JavaWebStartState.transition(JavaWebStartState.java:85)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.start(JavaWebStartInfo.java:253)
      at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:150)
      at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:139)
      at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
      at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
      at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
      at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:461)
      at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:375)
      at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:219)
      at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
      at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
      at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
      at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
      at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
      at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
      at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
      at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
      at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
      at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
      at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
      Caused by: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar
      at org.glassfish.appclient.server.core.jws.servedcontent.AutoSignedContent.<init>(AutoSignedContent.java:66)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createSignedStaticContent(JavaWebStartInfo.java:605)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createAndAddSignedStaticContent(JavaWebStartInfo.java:596)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createAndAddSignedStaticContentFromGeneratedFile(JavaWebStartInfo.java:576)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.initClientStaticContent(JavaWebStartInfo.java:486)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.addClientContentToHTTPAdapters(JavaWebStartInfo.java:456)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.startJWSServices(JavaWebStartInfo.java:326)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.access$000(JavaWebStartInfo.java:98)
      at org.glassfish.appclient.server.core.jws.JavaWebStartInfo$1.run(JavaWebStartInfo.java:257)
      ... 29 more

      #]

        Activity

        Hide
        Hong Zhang added a comment -

        Tim and I had some discussions on this. One theory Tim has was if the application failed to load during server restart for some reason (for example due to this NPE or other cause), the deployment failure roll back logic would try to clean up the generated directories and resulted in the JWS file being cleaned up.

        So I have updated this part of the deployment logic to only clean up for initial deployment failure and not for subsequent loading (like during server restart) so the JWS files will not be removed if the application failed to load during server start.

        We however are not too sure why the application would fail to load, that particular NPE as you found out most times seems harmless, maybe it's due to some other cause. But in any case the NPE issue is tracked through other issues already, so we will not create another new issue for that. The fix I have checked in will at least address the problem that the generated files got deleted when the application failed to load during restart when they should not be.

        Show
        Hong Zhang added a comment - Tim and I had some discussions on this. One theory Tim has was if the application failed to load during server restart for some reason (for example due to this NPE or other cause), the deployment failure roll back logic would try to clean up the generated directories and resulted in the JWS file being cleaned up. So I have updated this part of the deployment logic to only clean up for initial deployment failure and not for subsequent loading (like during server restart) so the JWS files will not be removed if the application failed to load during server start. We however are not too sure why the application would fail to load, that particular NPE as you found out most times seems harmless, maybe it's due to some other cause. But in any case the NPE issue is tracked through other issues already, so we will not create another new issue for that. The fix I have checked in will at least address the problem that the generated files got deleted when the application failed to load during restart when they should not be.
        Hide
        skrall added a comment -

        Is there a way we can get a patched 3.1.2.2 jar file? This is fairly painful for us, we keep having server's intermittently have this issue, which causes our application to not start.

        Show
        skrall added a comment - Is there a way we can get a patched 3.1.2.2 jar file? This is fairly painful for us, we keep having server's intermittently have this issue, which causes our application to not start.
        Hide
        Hong Zhang added a comment -

        Let me see if I can make a 3.1.2 patch jar and send to you...

        Show
        Hong Zhang added a comment - Let me see if I can make a 3.1.2 patch jar and send to you...
        Hide
        skrall added a comment -

        Thanks, 3.1.2.2, you just have 3.1.2 in your comment, just want to avoid confusion.

        And we will be able to test and let you know if it totally fixes the issue or not.

        Show
        skrall added a comment - Thanks, 3.1.2.2, you just have 3.1.2 in your comment, just want to avoid confusion. And we will be able to test and let you know if it totally fixes the issue or not.
        Hide
        Hong Zhang added a comment -

        Thanks for the clarification. I have built a patch jar locally for 3.1.2.2 now and I will send it to you via email (as the attachments are disabled in the JIRA issue now).

        Show
        Hong Zhang added a comment - Thanks for the clarification. I have built a patch jar locally for 3.1.2.2 now and I will send it to you via email (as the attachments are disabled in the JIRA issue now).

          People

          • Assignee:
            Hong Zhang
            Reporter:
            skrall
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: