How to set InputStream content Length

Question

I am uploading files to Amazon S3 bucket. The files are being uploaded but i get the following Warning.

WARNING: No content length specified for stream data. Stream contents will be buffered in memory and could result in out of memory errors.

So I added the following line to my code

metaData.setContentLength(IOUtils.toByteArray(input).length);

but then i got the following message. I don't even know if it is a warning or what.

Data read has a different length than the expected: dataLength=0; expectedLength=111992; includeSkipped=false; in.getClass()=class sun.net.httpserver.FixedLengthInputStream; markedSupported=false; marked=0; resetSinceLastMarked=false; markCount=0; resetCount=0

How can i set contentLength to the metaData of InputSteam? Any help would be greatly appreciated.

Answer 1

When you read the data with IOUtils.toByteArray, this consumes the InputStream. When the AWS API tries to read it, it's zero length.

Read the contents into a byte array and provide an InputStream wrapping that array to the API:

byte[] bytes = IOUtils.toByteArray(input);
metaData.setContentLength(bytes.length);
ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream(bytes);
PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, byteArrayInputStream, metadata);
client.putObject(putObjectRequest);

You should consider using the multipart upload API to avoid loading the whole InputStream into memory. For example:

byte[] bytes = new byte[BUFFER_SIZE];
String uploadId = client.initiateMultipartUpload(new InitiateMultipartUploadRequest(bucket, key)).getUploadId();

int bytesRead = 0;
int partNumber = 1;
List<UploadPartResult> results = new ArrayList<>();
bytesRead = input.read(bytes);
while (bytesRead >= 0) {
    UploadPartRequest part = new UploadPartRequest()
        .withBucketName(bucket)
        .withKey(key)
        .withUploadId(uploadId)
        .withPartNumber(partNumber)
        .withInputStream(new ByteArrayInputStream(bytes, 0, bytesRead))
        .withPartSize(bytesRead);
    results.add(client.uploadPart(part));
    bytesRead = input.read(bytes);
    partNumber++;
}
CompleteMultipartUploadRequest completeRequest = new CompleteMultipartUploadRequest()
    .withBucketName(bucket)
    .withKey(key)
    .withUploadId(uploadId)
    .withPartETags(results);
client.completeMultipartUpload(completeRequest);