Outputstream start writing when inputstream starts reading

Question

I want to send a big object (lets say 4G) over http.

We have a custom serializer that writes the object to an OutputStream. At the moment we write the object to disk and use that file for the inputstream which is used for the request.

Something like these lines:

private static Response sendObject(Object bigObject) throws IOException {
  File tempFile = File.createTempFile("x", "y");
  OutputStream out = new FileOutputStream(tempFile);
  CustomSerializer.serialize(bigObject, out);
  out.close();

  WebTarget resource = service.path("data");

  FormDataMultiPart multiPartEntity = new FormDataMultiPart();

  InputStream inputStream = new FileInputStream(tempFile);
  StreamDataBodyPart streamBodyPart = new StreamDataBodyPart(
        "data",
        inputStream,
        "data",
        MediaType.APPLICATION_OCTET_STREAM_TYPE);

  MultiPart multiPart = multiPartEntity.bodyPart(streamBodyPart);
  return resource.request(MediaType.APPLICATION_JSON_TYPE)
        .post(Entity.entity(multiPart, multiPart.getMediaType()));
 }

We save some memory because we don't serialize to a byte array in memory. Thats nice. But could I save the memory without writing to disk.

Could you write directly to the input stream without rewriting the CustomSerializer?

Could you write directly to the input stream while it reads into the request?

-

It is a little hard to explain, But I think I am after something like this pseudo code:

private static Response sendObject(Object bigObject) throws IOException {
  WebTarget resource = service.path("data");

  FormDataMultiPart multiPartEntity = new FormDataMultiPart();

  // A type of stream I don't know if exist
  OutputStream outIn = new OutputInputStream() {
     public void openInputStream() {
        CustomSerializer.serialize(bigObject, this);
     }
  };
  StreamDataBodyPart streamBodyPart = new StreamDataBodyPart(
        "data",
        outIn.getInputStream(),
        "data",
        MediaType.APPLICATION_OCTET_STREAM_TYPE);

  MultiPart multiPart = multiPartEntity.bodyPart(streamBodyPart);
  return resource.request(MediaType.APPLICATION_JSON_TYPE)
        .post(Entity.entity(multiPart, multiPart.getMediaType()));
} 

Answer 1

You could use a StreamingOutput and use your CustomSerializer to write to the provided OutputStream

StreamingOutput entity = new StreamingOutput() {
    @Override
    public void write(OutputStream out)
            throws IOException, WebApplicationException {

        CustomSerializer.serialize(bigObject, out);
    }
};

The write() method will be called by Jersey, giving you a chance to write directly to response entity stream.

Then just use a FormDataBodyPart

BodyPart bigPart = new FormDataBodyPart(
        "data", entity, MediaType.APPLICATION_OCTET_STREAM_TYPE);
MultiPart multiPart = new FormDataMultiPart().bodyPart(bigPart);