I am building a HTTP proxy with netty, which supports HTTP pipelining. Therefore I receive multiple HttpRequest
Objects on a single Channel and got the matching HttpResponse
Objects. The order of the HttpResponse
writes is the same than I got the HttpRequest
. If a HttpResponse
was written, the next one will be written when the HttpProxyHandler
receives a writeComplete
event.
The Pipeline should be convenient:
final ChannelPipeline pipeline = Channels.pipeline();
pipeline.addLast("decoder", new HttpRequestDecoder());
pipeline.addLast("encoder", new HttpResponseEncoder());
pipeline.addLast("writer", new HttpResponseWriteDelayHandler());
pipeline.addLast("deflater", new HttpContentCompressor(9));
pipeline.addLast("handler", new HttpProxyHandler());
Regarding this question only the order of the write calls should be important, but to be sure I build another Handler (HttpResponseWriteDelayHandler
) which suppresses the writeComplete
event until the whole response was written.
To test this I enabled network.http.proxy.pipelining
in Firefox and visited a page with many images and connections (a news page). The problem is, that the browser does not receive some responses in spite of the logs of the proxy consider them as sent successfully.
I have some findings:
304 - Not Modified
responses were sent (refreshing the page considering browser cache)bootstrap.setOption("sendBufferSize", 1048576);
or above does not helpwriteComplete
event in HttpResponseWriteDelayHandler
solves the problem, but is a very bad solution.I found the solution and want to share it, if anyone else has a similar problem:
The content of the HttpResponse
is too big. To analyze the content the whole HTML document was in the buffer. This must be splitted in Chunks again to send it properly. If the HttpResponse
is not chunked I wrote a simple solution to do it. One needs to put a ChunkedWriteHandler
next to the logic handler and write this class instead of the response itself:
public class ChunkedHttpResponse implements ChunkedInput {
private final static int CHUNK_SIZE = 8196;
private final HttpResponse response;
private final Queue<HttpChunk> chunks;
private boolean isResponseWritten;
public ChunkedHttpResponse(final HttpResponse response) {
if (response.isChunked())
throw new IllegalArgumentException("response must not be chunked");
this.chunks = new LinkedList<HttpChunk>();
this.response = response;
this.isResponseWritten = false;
if (response.getContent().readableBytes() > CHUNK_SIZE) {
while (CHUNK_SIZE < response.getContent().readableBytes()) {
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(CHUNK_SIZE)));
}
chunks.add(new DefaultHttpChunk(response.getContent().readSlice(response.getContent().readableBytes())));
chunks.add(HttpChunk.LAST_CHUNK);
response.setContent(ChannelBuffers.EMPTY_BUFFER);
response.setChunked(true);
response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING, HttpHeaders.Values.CHUNKED);
}
}
@Override
public boolean hasNextChunk() throws Exception {
return !isResponseWritten || !chunks.isEmpty();
}
@Override
public Object nextChunk() throws Exception {
if (!isResponseWritten) {
isResponseWritten = true;
return response;
} else {
HttpChunk chunk = chunks.poll();
return chunk;
}
}
@Override
public boolean isEndOfInput() throws Exception {
return isResponseWritten && chunks.isEmpty();
}
@Override
public void close() {}
}
Then one can call just channel.write(new ChunkedHttpResponse(response)
and the chunking is done automatically if needed.