pdf-renderer
  1. pdf-renderer
  2. PDF_RENDERER-74

Allow passing in a PDF that isn't in a ByteBuffer

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: current
    • Fix Version/s: milestone 1
    • Component/s: www
    • Labels:
      None
    • Environment:

      Operating System: All
      Platform: All

    • Issuezilla Id:
      74

      Description

      Our application takes up a relatively large amount of memory (most of the heap
      space we allocate at viewing time is for the purpose of caches.) Our experience
      of using memory mapping is that it often, in low memory conditions, causes major
      problems. It shouldn't crash the application but I have seen it do so from time
      to time.

      So we would like some way to pass in something that isn't a ByteBuffer.

      What would be truly useful is some kind of generic API which has "read this from
      this offset", which can then be implemented to be backed by a ByteBuffer, a
      RandomAccessFile, or "something else" – the "something else" being our internal
      binary reading interface which was designed to overcome this precise problem.

        Activity

        Hide
        tomoke added a comment -

        The use of the MappedByteBuffer is supposed to alleviate the heap use, by using
        memory outside of the java heap. There are other problems with actually
        disconnecting the file from being mapped, but that is a different issue.

        Testing I have done with very large files, through mapping, appears to support
        this.

        It would be my intent to add additional methods to construct the PDFFile, that
        ensures that the file is a temp copy, so that we no longer tie down the
        original, and that the mapping is disconnected, safely, when the PDFFile usage
        is closed (for issue 75).

        Would that resolve your problems?

        Show
        tomoke added a comment - The use of the MappedByteBuffer is supposed to alleviate the heap use, by using memory outside of the java heap. There are other problems with actually disconnecting the file from being mapped, but that is a different issue. Testing I have done with very large files, through mapping, appears to support this. It would be my intent to add additional methods to construct the PDFFile, that ensures that the file is a temp copy, so that we no longer tie down the original, and that the mapping is disconnected, safely, when the PDFFile usage is closed (for issue 75). Would that resolve your problems?
        Hide
        trejkaz added a comment -

        The main problem with MappedByteBuffer is that it can exhaust the memory outside the heap. We
        tend to set the heap to up to 1300M or so, leaving 700M for everything else. PermGen space and other
        JVM stuff eats a bit more of what's left, and other things want to use mapped byte buffers too, so we do
        run out of memory. This is of course on Windows, where the OS eats up the first half of the available
        virtual memory space.

        So this feature request was mainly to give us a way to dodge that by using our usual trick of a sliding
        buffer over a RandomAccessFile or possibly FileChannel.

        As you say there are also issues on knowing when you can close the buffer, I think those could be
        resolved by PDFFile having a close() or similar, which I raised a separate request for.

        Show
        trejkaz added a comment - The main problem with MappedByteBuffer is that it can exhaust the memory outside the heap. We tend to set the heap to up to 1300M or so, leaving 700M for everything else. PermGen space and other JVM stuff eats a bit more of what's left, and other things want to use mapped byte buffers too, so we do run out of memory. This is of course on Windows, where the OS eats up the first half of the available virtual memory space. So this feature request was mainly to give us a way to dodge that by using our usual trick of a sliding buffer over a RandomAccessFile or possibly FileChannel. As you say there are also issues on knowing when you can close the buffer, I think those could be resolved by PDFFile having a close() or similar, which I raised a separate request for.
        Hide
        tomoke added a comment -

        So you are into the industrial level of things. Looks like what we want here is
        to keep the code the same, permitting a ByteBuffer, but to do our own
        ByteBuffer based on a random access file, such that the buffer is only a cache
        of parts of the file, and that these parts keep on moving. It only looks like
        the buffer holds it all at once.

        Show
        tomoke added a comment - So you are into the industrial level of things. Looks like what we want here is to keep the code the same, permitting a ByteBuffer, but to do our own ByteBuffer based on a random access file, such that the buffer is only a cache of parts of the file, and that these parts keep on moving. It only looks like the buffer holds it all at once.
        Hide
        trejkaz added a comment -

        Yes. That is exactly what I would have implemented myself from the outside,
        except that ByteBuffer itself can't be subclassed. One of its subclasses might
        provide a way, however.

        Show
        trejkaz added a comment - Yes. That is exactly what I would have implemented myself from the outside, except that ByteBuffer itself can't be subclassed. One of its subclasses might provide a way, however.

          People

          • Assignee:
            pdf-renderer-issues
            Reporter:
            trejkaz
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: