jersey
  1. jersey
  2. JERSEY-2337

GrizzlyConnectorProvider throws Out Of Memory exceptions when POSTing large message bodies, despite using chunked transfer encoding

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.1
    • Fix Version/s: 2.7
    • Component/s: connectors
    • Labels:
      None
    • Environment:

      The web service is running on a Linux box, the client test code was on a Max OS X

      Description

      When sending large message bodies in a POST request, the GrizzlyConnectorProvider throws an Out Of Memory exception despite 'chunked' transfer encoding being turned on. It appears that entity buffering is taking place on the client side.

      Exception message:

       
      Jan 14, 2014 1:27:31 PM org.glassfish.grizzly.filterchain.DefaultFilterChain execute
      WARNING: Exception during FilterChain execution
      java.lang.OutOfMemoryError: Java heap space
      	at org.glassfish.grizzly.memory.HeapMemoryManager.createTrimAwareBuffer(HeapMemoryManager.java:289)
      	at org.glassfish.grizzly.memory.HeapMemoryManager.allocateHeapBuffer(HeapMemoryManager.java:201)
      	at org.glassfish.grizzly.memory.HeapMemoryManager.reallocateHeapBuffer(HeapMemoryManager.java:255)
      	at org.glassfish.grizzly.memory.HeapMemoryManager.reallocate(HeapMemoryManager.java:100)
      	at org.glassfish.grizzly.memory.HeapMemoryManager.reallocate(HeapMemoryManager.java:60)
      	at org.glassfish.grizzly.utils.BufferOutputStream.ensureCapacity(BufferOutputStream.java:216)
      	at org.glassfish.grizzly.utils.BufferOutputStream.write(BufferOutputStream.java:181)
      	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
      	at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
      	at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
      	at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
      	at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
      	at java.io.BufferedWriter.write(BufferedWriter.java:230)
      	at java.io.Writer.write(Writer.java:157)
      	at webservice.client.JerseyClientTest$ExampleMessageBodyWriter.writeTo(JerseyClientTest.java:190)
      	at webservice.client.JerseyClientTest$ExampleMessageBodyWriter.writeTo(JerseyClientTest.java:1)
      	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:243)
      	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:230)
      	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:149)
      	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1139)
      	at org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:495)
      	at org.glassfish.jersey.grizzly.connector.GrizzlyConnector$5.writeEntity(GrizzlyConnector.java:376)
      	at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$EntityWriterBodyHandler.doHandle(GrizzlyAsyncHttpProvider.java:2002)
      	at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider.sendRequest(GrizzlyAsyncHttpProvider.java:571)
      	at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.sendAsGrizzlyRequest(GrizzlyAsyncHttpProvider.java:921)
      	at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientFilter.handleWrite(GrizzlyAsyncHttpProvider.java:803)
      	at org.glassfish.grizzly.filterchain.ExecutorResolver$8.execute(ExecutorResolver.java:111)
      	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:288)
      	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:206)
      	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:136)
      	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:114)
      

      Code:

      /*
      
      Note: 'ner.train.gz' is a file containing about 270,000 lines of plain text, for a total size of about 117MB. I tried to iterate over this file 100 times, therefore trying to
      POST about 11 GB of data to the web service.
      
      The web service and the client both run fine for small amounts of data (< 250 MB), however, once the data gets huge, the client crashes before sending anything to the server (not even request headers) because it cannot buffer the entity anymore in memory.
      
      Why is it trying to buffer, though? Chunked encoding has been explicitly specified...
      
      */
      
      //A simple JUnit test case to demonstrate the issue.
      @Test
      	public void hugeTest() throws IOException, InterruptedException, ExecutionException {
      
      		final ClientConfig clientConfig = new ClientConfig();
      		clientConfig.property(ClientProperties.REQUEST_ENTITY_PROCESSING, RequestEntityProcessing.CHUNKED);
      		clientConfig.property(ClientProperties.OUTBOUND_CONTENT_LENGTH_BUFFER, Integer.valueOf(-1));
      		clientConfig.register(ExampleMessageBodyWriter.class);
      
      		clientConfig.connectorProvider(new GrizzlyConnectorProvider()); 
      
      		Client client = ClientBuilder.newClient(clientConfig);
      
      		WebTarget target = client.target("http://localhost:8080").path("/vw-webservice/predict/main");
      
      		LOGGER.debug("About to submit request...");
      
      		Future<Response> future = target.request("*/*").async().post(Entity.entity(new ExampleClass(), MediaType.TEXT_PLAIN));
      
      		LOGGER.debug("Req submitted...");
      
      		Response response = future.get();
      
      		final ChunkedInput<String> chunkedInput = response.readEntity(new GenericType<ChunkedInput<String>>() {
      		});
      		chunkedInput.setParser(ChunkedInput.createParser("\n"));
      		String chunk;
      		LOGGER.debug("starting to read responses...");
      
      		while ((chunk = chunkedInput.read()) != null) {
      			//System.out.println("Next chunk received: " + chunk);
      		}
      
      	}
      
      	class ExampleClass {
      	};
      
      	@Produces({ MediaType.TEXT_PLAIN })
      	@Provider
      	static class ExampleMessageBodyWriter implements MessageBodyWriter<ExampleClass> {
      
      		private static Logger LOGGER = LoggerFactory.getLogger(ExampleMessageBodyWriter.class);
      
      		@Override
      		public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
      			return true;
      		}
      
      		@Override
      		public long getSize(ExampleClass t, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
      			return -1;
      		}
      
      		@Override
      		public void writeTo(ExampleClass t, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException, WebApplicationException {
      
      			LOGGER.debug("Writing to a stream of type: {}", entityStream.getClass());
      
      			try {
      
      				BufferedWriter exampleWriter = new BufferedWriter(new OutputStreamWriter(entityStream));
      
      				for (int numTimes = 0; numTimes < 100; numTimes++) {
      					//the examples
      					GZIPInputStream gzipInputStream = new GZIPInputStream(this.getClass().getClassLoader().getResourceAsStream("ner.train.gz"));
      
      					BufferedReader testReader = new BufferedReader(new InputStreamReader(gzipInputStream, Charsets.UTF_8));
      
      					String example = null;
      
      					long numExamplesSubmitted = 0;
      
      					while ((example = testReader.readLine()) != null) {
      						exampleWriter.write(example);
      						exampleWriter.newLine();
      
      						numExamplesSubmitted++;
      
      					}
      
      					testReader.close();
      
      					exampleWriter.flush();
      				}
      
      				exampleWriter.close();
      
      			}
      			catch (Exception e) {
      				LOGGER.error("Error in message body writer! {}", e.getMessage(), e);
      			}
      		}
      
      	}
      

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Jakub Podlesak
            Reporter:
            verun.karim
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 30 minutes
              30m
              Remaining:
              Remaining Estimate - 0 minutes
              0m
              Logged:
              Time Spent - 1 minute Time Not Required
              1m