Issue Details (XML | Word | Printable)

Key: XADISK-144
Type: Bug Bug
Status: Open Open
Priority: Major Major
Assignee: Nitin Verma
Reporter: gunnar_zarncke
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
xadisk

out-of-file-handles on transaction involing large number of copied files

Created: 11/Sep/13 11:09 AM   Updated: 11/Sep/13 11:09 AM
Component/s: filesystem
Affects Version/s: 1.2.2
Fix Version/s: None

Time Tracking:
Not Specified

Environment:

CentOS, Linux testhost 2.6.32-358.2.1.el6.x86_64 #1 SMP Wed Mar 13 00:26:49 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux, java version "1.7.0_25"
OpenJDK Runtime Environment (rhel-2.3.10.4.el6_4-x86_64)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)


Tags:
Participants: gunnar_zarncke and Nitin Verma


 Description  « Hide

An XADisk transaction involving a very large number of files extracted from an archive fails fatally due to an OS-level out-of-file-handle situation.

The issue is critical in so far as we want to write the files extracted from the archive under transaction control and the transaction fails fatally:

XASystemNoMoreAvailableException: The XADisk instance has encoutered a critial issue and is no more available. Such a condition is very rare. If you think you have setup everything right for XADisk to work, please consider discussing in XADisk forums, or raising a bug with details

We took care to close all files immediately after writing the data and we also ensured that only a limited number of files are processed concurrently.

At first it looked as if the files might be cached in NativeSession.getCachedXAFileOutputStream.
On further inspection of the XA-Disk code it became clear that closing the XAFileOutputStream doesn't actually close the underlying file handle/channel. The close happens delayed in VirtualViewFile.forceAndFreePhysicalChannel() upon commit/rollback. This is also the case for heavyWrite which we use in this case as the files are completely written in one chunk.

This means that we can do nothing about this because the number of files that are to be extracted is potentially arbitrarily large - in in particular significantly larger than any acceptable ulimit.

I am wondering why the file handles cannot be closed earlier.
I have seen that the data is flushed upon XAFileOutputStream.close().
It should be possible to close them upon close at least in the heavy write case. Alternatively it might be possible to close the file and later reopen at the same position - should that unexpetcedly be neccessary.


Additional information:

The file handle limits is at the default of 1024:

bash-4.1$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 29738
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Reported Exception:

[...]
Caused by: org.xadisk.filesystem.exceptions.XASystemNoMoreAvailableException: The XADisk instance has encoutered a critial issue and is no more available. Such a condition is very rare. If you think you have setup everything right for XADisk to work, please consider discussing in XADisk forums, or raising a bug with details
at org.xadisk.filesystem.NativeXAFileSystem.notifySystemFailure(NativeXAFileSystem.java:489)
at org.xadisk.filesystem.virtual.NativeXAFileOutputStream.<init>(NativeXAFileOutputStream.java:56)
at org.xadisk.filesystem.NativeSession.getCachedXAFileOutputStream(NativeSession.java:1266)
at org.xadisk.filesystem.NativeSession.createXAFileOutputStream(NativeSession.java:187)
at org.xadisk.filesystem.NativeSession.createXAFileOutputStream(NativeSession.java:43)
at de.konzentrik.lib.io.xa.XaDecoupledFileSystem$XaDecoupledSession.createXAFileOutputStream(XaDecoupledFileSystem.java:216)
at de.konzentrik.lib.io.XaDiskStore.getOutputStream(XaDiskStore.java:263)
at de.konzentrik.app.preprocess.process.step.ImageCopierStep.process(ImageCopierStep.java:65)
at de.konzentrik.app.preprocess.process.step.ImageCopierStep.process(ImageCopierStep.java:25)
[...]
at de.konzentrik.app.preprocess.process.ProcessStepRunner$StepRunner.run(ProcessStepRunner.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more
Caused by: java.io.IOException: Zu viele offene Dateien [german for "too many open files"]
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.createNewFile(File.java:947)
at org.xadisk.filesystem.utilities.FileIOUtility.createFile(FileIOUtility.java:91)
at org.xadisk.filesystem.DurableDiskSession.createFile(DurableDiskSession.java:158)
at org.xadisk.filesystem.virtual.VirtualViewFile.safeSetupForPhysicalFileExistence(VirtualViewFile.java:142)
at org.xadisk.filesystem.virtual.VirtualViewFile.setUpForHeavyWriteOptimization(VirtualViewFile.java:154)
at org.xadisk.filesystem.virtual.NativeXAFileOutputStream.<init>(NativeXAFileOutputStream.java:54)
... 16 more



There are no comments yet on this issue.