while getting the xml from server, i am facing an issue, where the the response xml provided by HttpClient GET method is partial. Whereas the xml provide using browser is full as expected.
The code for the same is
InputStream response = null;
HttpClient client = new HttpClient();
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false));
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
response = method.getResponseBodyAsStream();
following is the output of httpclient debug logs
16:07:24(GetMethod.java:89) DEBUG org.apache.commons.httpclient.methods.GetMethod - enter GetMethod(String)
16:07:26(DefaultHttpParams.java:150) DEBUG org.apache.commons.httpclient.params.DefaultHttpParams - Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@578cef18
16:07:29(HttpClient.java:322) DEBUG org.apache.commons.httpclient.HttpClient - enter HttpClient.executeMethod(HttpMethod)
16:07:29(HttpClient.java:373) DEBUG org.apache.commons.httpclient.HttpClient - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
16:07:29(HttpMethodDirector.java:378) DEBUG org.apache.commons.httpclient.HttpMethodDirector - Attempt number 1 to process request
16:07:29(HttpConnection.java:684) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.open()
16:07:29(HttpConnection.java:691) DEBUG org.apache.commons.httpclient.HttpConnection - Open connection to mysite:80
16:07:29(HttpMethodBase.java:976) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.execute(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1916) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequest(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:2071) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequestLine(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1379) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.generateRequestLine(HttpConnection, String, String, String, String)
16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "GET /mypath HTTP/1.1[\r][\n]"
16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
16:07:29(HttpMethodBase.java:2034) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection)
16:07:29(HttpMethodBase.java:1300) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addRequestHeaders(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1325) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1211) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1235) DEBUG org.apache.commons.httpclient.HttpMethodBase - Adding Host request header
16:07:29(HttpMethodBase.java:1163) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection)
16:07:29(HttpState.java:172) DEBUG org.apache.commons.httpclient.HttpState - enter HttpState.getCookies()
16:07:29(CookieSpecBase.java:548) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.match(String, int, String, boolean, Cookie[])
16:07:29(HttpMethodBase.java:1262) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection)
16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "User-Agent: Jakarta Commons-HttpClient/3.0.1[\r][\n]"
16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "Host: mysite[\r][\n]"
16:07:29(HttpConnection.java:1031) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.print(String)
16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
16:07:29(HttpConnection.java:1000) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.writeLine()
16:07:29(HttpConnection.java:941) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[])
16:07:29(HttpConnection.java:962) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.write(byte[], int, int)
16:07:29(Wire.java:69) DEBUG httpclient.wire.header - >> "[\r][\n]"
16:07:29(HttpConnection.java:825) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.flushRequestOutputStream()
16:07:29(HttpMethodBase.java:1585) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponse(HttpState, HttpConnection)
16:07:29(HttpMethodBase.java:1823) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readStatusLine(HttpState, HttpConnection)
16:07:29(HttpConnection.java:1112) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.readLine()
16:07:29(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:07:29(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:07:29(Wire.java:69) DEBUG httpclient.wire.header - << "HTTP/1.0 200 OK[\r][\n]"
16:09:31(HttpMethodBase.java:1785) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseHeaders(HttpState,HttpConnection)
16:09:31(HttpConnection.java:856) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.getResponseInputStream()
16:09:31(HttpParser.java:159) DEBUG org.apache.commons.httpclient.HttpParser - enter HeaderParser.parseHeaders(InputStream, String)
16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:09:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:09:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:09:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(HttpParser.java:104) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readLine(InputStream, String)
16:11:31(HttpParser.java:73) DEBUG org.apache.commons.httpclient.HttpParser - enter HttpParser.readRawLine()
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-DGKeepAlive: on[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Date: Thu, 10:37:29 GMT[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Server: Apache[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Set-Cookie: JSESSIONID=1DA9A4C37BD6883B837BA8725E3CCC5D; Path=/mypath[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Vary: Accept-Encoding[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Content-Type: text/xml;charset=ISO-8859-1[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-Cache: MISS from ajay[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "X-Cache-Lookup: MISS from ajay:3128[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Via: 1.0 ajay:3128 (squid/2.6.STABLE21)[\r][\n]"
16:11:31(Wire.java:69) DEBUG httpclient.wire.header - << "Proxy-Connection: close[\r][\n]"
16:11:31(HttpMethodBase.java:1462) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection)
16:11:31(CookieSpecBase.java:242) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.parse(String, port, path, boolean, String)
16:11:31(CookieSpecBase.java:112) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.parse(String, port, path, boolean, Header)
16:11:31(HeaderElement.java:193) DEBUG org.apache.commons.httpclient.HeaderElement - enter HeaderElement.parseElements(char[])
16:11:31(Cookie.java:108) DEBUG org.apache.commons.httpclient.Cookie - enter Cookie(String, String, String, String, Date, boolean)
16:11:31(RFC2109Spec.java:130) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter RFC2109Spec.validate(String, int, String, boolean, Cookie)
16:11:31(CookieSpecBase.java:368) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter CookieSpecBase.validate(String, port, path, boolean, Cookie)
16:11:31(HttpState.java:124) DEBUG org.apache.commons.httpclient.HttpState - enter HttpState.addCookie(Cookie)
16:11:31(RFC2109Spec.java:243) DEBUG org.apache.commons.httpclient.cookie.CookieSpec - enter RFC2109Spec.formatCookie(Cookie)
16:11:31(HttpMethodBase.java:1506) DEBUG org.apache.commons.httpclient.HttpMethodBase - Cookie accepted: "$Version=0; JSESSIONID=1DA9A4C37BD6883B837BA8725E3CCC5D; $Path=/mypath"
16:11:31(HttpMethodBase.java:1636) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseBody(HttpState, HttpConnection)
16:11:31(HttpMethodBase.java:1671) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.readResponseBody(HttpConnection)
16:11:31(HttpConnection.java:856) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.getResponseInputStream()
16:11:31(HttpMethodBase.java:2140) DEBUG org.apache.commons.httpclient.HttpMethodBase - enter HttpMethodBase.canResponseHaveBody(int)
16:11:31(Wire.java:83) DEBUG httpclient.wire.content - <<"#######INCOMPLETE XML#########"
16:12:12(HttpMethodBase.java:921) DEBUG org.apache.commons.httpclient.HttpMethodBase - Resorting to protocol version default close connection policy
16:12:12(HttpMethodBase.java:929) DEBUG org.apache.commons.httpclient.HttpMethodBase - Should close connection, using HTTP/1.0
16:12:12(HttpConnection.java:1147) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.close()
16:12:12(HttpConnection.java:1214) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.closeSockedAndStreams()
16:12:12(HttpConnection.java:1173) DEBUG org.apache.commons.httpclient.HttpConnection - enter HttpConnection.releaseConnection()
16:12:12(HttpConnection.java:1177) DEBUG org.apache.commons.httpclient.HttpConnection - Releasing connection back to connection manager.
When the same url is fetched through browser, it works. Headers of request are following Browser Response Headersview source
Content-Encoding gzip
Content-Type text/xml;charset=ISO-8859-1
Date Thu, 20 Jun 2013 11:24:54 GMT
Proxy-Connection close
Server Apache
Vary Accept-Encoding
Via 1.0 ajay:3128 (squid/2.6.STABLE21)
X-Cache MISS from ajay
X-Cache-Lookup MISS from ajay:3128
Browser Request Headersview source
Accept text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection keep-alive
Cookie JSESSIONID=6265E9445BB5CB5EF2594A9BEC19B119
Host mysite
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:20.0) Gecko/20100101 Firefox/20.0
Can anyone explain where is the issue, in the httpclient calling code or some other at server side and how to correct that? Thanks in advance.
The exception is thrown while parsing the response
DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(inputStreamOfResponse);
exception is
[Fatal Error] :2:1: XML document structures must start and end within the same entity.
org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 1; XML document structures must start and end within the same entity.
finally solved the issue, setting the following header worked for me.
method.setRequestHeader("Accept-Encoding", "gzip, deflate");
and
postResponse = IOUtils.toString(new GZIPInputStream(method.getResponseBodyAsStream()));