xadisk
  1. xadisk
  2. XADISK-144

out-of-file-handles on transaction involing large number of copied files

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.2
    • Fix Version/s: None
    • Component/s: filesystem
    • Labels:
      None
    • Environment:

      Description

      An XADisk transaction involving a very large number of files extracted from an archive fails fatally due to an OS-level out-of-file-handle situation.

      The issue is critical in so far as we want to write the files extracted from the archive under transaction control and the transaction fails fatally:

      XASystemNoMoreAvailableException: The XADisk instance has encoutered a critial issue and is no more available. Such a condition is very rare. If you think you have setup everything right for XADisk to work, please consider discussing in XADisk forums, or raising a bug with details

      We took care to close all files immediately after writing the data and we also ensured that only a limited number of files are processed concurrently.

      At first it looked as if the files might be cached in NativeSession.getCachedXAFileOutputStream.
      On further inspection of the XA-Disk code it became clear that closing the XAFileOutputStream doesn't actually close the underlying file handle/channel. The close happens delayed in VirtualViewFile.forceAndFreePhysicalChannel() upon commit/rollback. This is also the case for heavyWrite which we use in this case as the files are completely written in one chunk.

      This means that we can do nothing about this because the number of files that are to be extracted is potentially arbitrarily large - in in particular significantly larger than any acceptable ulimit.

      I am wondering why the file handles cannot be closed earlier.
      I have seen that the data is flushed upon XAFileOutputStream.close().
      It should be possible to close them upon close at least in the heavy write case. Alternatively it might be possible to close the file and later reopen at the same position - should that unexpetcedly be neccessary.


      Additional information:

      The file handle limits is at the default of 1024:

      bash-4.1$ ulimit -a
      core file size (blocks, -c) 0
      data seg size (kbytes, -d) unlimited
      scheduling priority (-e) 0
      file size (blocks, -f) unlimited
      pending signals (-i) 29738
      max locked memory (kbytes, -l) 64
      max memory size (kbytes, -m) unlimited
      open files (-n) 1024
      pipe size (512 bytes, -p) 8
      POSIX message queues (bytes, -q) 819200
      real-time priority (-r) 0
      stack size (kbytes, -s) 10240
      cpu time (seconds, -t) unlimited
      max user processes (-u) 1024
      virtual memory (kbytes, -v) unlimited
      file locks (-x) unlimited

      Reported Exception:

      [...]
      Caused by: org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The XADisk instance has encoutered a critial issue and is no more available. Such a condition is very rare. If you think you have setup everything right for XADisk to work, please consider discussing in XADisk forums, or raising a bug with details
      at org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:489)
      at org.xadisk.filesystem.virtual.NativeXAFileOutputStream.<init>(NativeXAFileOutputStream.java:56)
      at org.xadisk.filesystem.NativeSession.getCachedXAFileOutputStream(NativeSession.java:1266)
      at org.xadisk.filesystem.NativeSession.createXAFileOutputStream(NativeSession.java:187)
      at org.xadisk.filesystem.NativeSession.createXAFileOutputStream(NativeSession.java:43)
      at de.konzentrik.lib.io.xa.XaDecoupledFileSystem$XaDecoupledSession.createXAFileOutputStream(XaDecoupledFileSystem.java:216)
      at de.konzentrik.lib.io.XaDiskStore.getOutputStream(XaDiskStore.java:263)
      at de.konzentrik.app.preprocess.process.step.ImageCopierStep.process(ImageCopierStep.java:65)
      at de.konzentrik.app.preprocess.process.step.ImageCopierStep.process(ImageCopierStep.java:25)
      [...]
      at de.konzentrik.app.preprocess.process.ProcessStepRunner$StepRunner.run(ProcessStepRunner.java:132)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      ... 1 more
      Caused by: java.io.IOException: Zu viele offene Dateien [german for "too many open files"]
      at java.io.UnixFileSystem.createFileExclusively(Native Method)
      at java.io.File.createNewFile(File.java:947)
      at org.xadisk.filesystem.utilities.FileIOUtility.createFile(FileIOUtility.java:91)
      at org.xadisk.filesystem.DurableDiskSession.createFile(DurableDiskSession.java:158)
      at org.xadisk.filesystem.virtual.VirtualViewFile.safeSetupForPhysicalFileExistence(VirtualViewFile.java:142)
      at org.xadisk.filesystem.virtual.VirtualViewFile.setUpForHeavyWriteOptimization(VirtualViewFile.java:154)
      at org.xadisk.filesystem.virtual.NativeXAFileOutputStream.<init>(NativeXAFileOutputStream.java:54)
      ... 16 more

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Nitin Verma
            Reporter:
            gunnar_zarncke
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: