Skip to main content

Synchronization causing performance problems

  • From: Billy Newman <newmanw10@...>
  • To: users@...
  • Subject: Synchronization causing performance problems
  • Date: Sun, 5 Feb 2012 10:32:36 -0700

Simone,

Thought this might be a more appropriate place to continue our conversation from:

http://www.java.net/forum/topic/javadesktop/java-desktop-technologies/jai/imagereader-thread-safety

1. I have an application that scans a disk for imagery, pulls out the metadata from the images and stores it.  Some of the disks this application scans are huge, including 20 million images.  To pull out the metadata I am using the NITFImageReader as follows:

[code]
// loop over millions of image files
// this could happen in many different threads to process faster

ImageReader reader = new NITFImageReaderSpi().createReaderInstance();
reader.setInput(nitfImageFile);
// Use 0 index to get image metadata for the one and only tile in this file
// This is where blocking will occur
CoreCommonImageMetadata metadata = (CoreCommonImageMetadata) reader.getImageMetadata(0);
// Now pull out width/height, bounding box, and lat/lon per pixel and store

[code]

My problem with this application is that it gets the metadata using

GDALImageReader.getImageMetadata(int index) -> calls GDALImageReader.getDatasetMetadata(int index), which uses a synchronous metadata map and synchronous block to protect the metadata map.  When I profile my multi threaded application a ton of time is spent by each thread blocking to pull out the metadata.  Since I only have one tile/image per file I create a new NITFImageReader each time making the synchronous code unnecessary.  If I pull out the synchronization and recompile my multi-threaded application has run up to five times faster (depending on throughput of disk).  This app can take days to run, speeding it up by 5 times is a huge gain.

2.  I have an application the tiles individual images together to create a mosaic.  This application is a server that is serving out the tiles.  The application server is handling many requests at a time to create mosaics.  Lets say I am using the NITFImageReader again for these images.

[code]
// in a loop to process all images that will be part of the mosaic (could be hundreds of images per mosaic)
// this is called per request, so will have many threads running at once (ie 20 requests, 20 threads)
ImageReader reader = new NITFImageReaderSpi().createReaderInstance();
ParameterBlockJAI pb = new ParameterBlockJAI("ImageRead");
pb.setParameter("Input", imageFile);
pb.setParameter("Reader", reader);
[code]

My problem here is that when reading using GDALImageReader the GDALUtilitites.acquireDataSet(...) is called, which is synchronous.  So when I have multiple threads all trying to open/read different image files my application spends a lot of time blocking.  Again by modifying the code to pull out the synchronous method call the application run a lot faster.


So in case #1 and #2 I am creating a new GDALImageReader each time I need to read an image.  I do not cache these readers because each image has only one tile and most of the time I am reading a different source image.

I do not really want to modify the imageio-ext code to gain more performance if I don't have to.  If there is already something built into imageio-ext that I do not know about please let me know.  If not maybe you have some ideas of how we can modify imageio-ext o support this.

Thanks for all your help thus far.

Billy


Synchronization causing performance problems

Billy Newman 02/05/2012

Re: Synchronization causing performance problems

Simone Giannecchini 02/06/2012

Re: Synchronization causing performance problems

Simone Giannecchini 02/07/2012

Re: Synchronization causing performance problems

Billy Newman 02/10/2012

Re: Synchronization causing performance problems

Simone Giannecchini 02/10/2012

Re: Synchronization causing performance problems

Billy Newman 02/11/2012
 
 
Close
loading
Please Confirm
Close