Search code examples
javamicronaut

Downloading large result

Is it possible to use the declarative client to download a large result by e.g. using an InputStream? I tried a client signature like 

HttpResponse<InputStream> getQueryResult(String jobId, String resultId);

But it tries to download the whole body, which then leads to 

io.micronaut.http.client.exceptions.ContentLengthExceededException: The received length exceeds the maximum content length

Thank you in advance.

Solution

  • What happens here is that your client requests a fully received (aggregated) HttpResponse, wrapping a byte array which is then converted into an InputStream. In order to get the response bytes without aggregation, you need to request one of the reactive types, such as a org.reactivestreams.Publisher (or a suitable subclass thereof) of ByteBuffers. Then you need to process those.

    Example:

    Flowable<ByteBuffer<?>> getQueryResult(String jobId, String resultId);

    You can then run map, forEach, blockingForEach, etc. on that io.reactivex.Flowable - BUT REMEMBER TO FREE THE BUFFERS, or you'll generate a lot of garbage, and get nasty log messages. Example (in Groovy):

    Flowable<ByteBuffer<?>> responseFlowable = myClient.getQueryResult("job1", "foo")
int sum = 0
responseFlowable.blockingForEach { ByteBuffer byteBuffer ->
    sum += byteBuffer.toByteArray().count('!')
    ((ReferenceCounted)byteBuffer).release() // Let Netty do its thing!
}

    (Obviously, blocking is bad for high throughput, but it's just an example)

    I hope this helps.