If I upload 100K file to certain url of my service, wget takes ~20 seconds to complete:
wget --quiet --post-file data.txt --output-document - --header "Content-Type: text/csv" http://localhost:8080/ingest
But if I do it like this in java, strangely this happens immediately:
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("Content-Type", "text/csv;charset=UTF-8");
con.setDoOutput(true);
OutputStream outputStream = con.getOutputStream();
outputStream.write(str.getBytes("UTF-8"));
outputStream.flush();
outputStream.close();
System.out.println("code=" + con.getResponseCode());
BufferedReader in = new BufferedReader(
new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
so my guess is that actually this code is not waiting for data to be submitted, but is doing this in background. How can I force it to block until actual data transfer is finished?
That code should be waiting for the response to complete. The con.getResponseCode() call will not (cannot!) return until the server has at least responded with the HTTP reply header containing the response code.
It may be that the server is sending the HTTP reply header before it has finished reading the data that the client has posted. That would be a mistake. (If the server sends the response too soon, it can't set the response code correctly!)
It is also possible that the server response is not a 2xx response, and there are server error messages / diagnostics on the error stream rather than the input stream. (Read the javadocs on getInputStream versus getErrorStream.)
So the most likely reason that is not blocking for ~20 seconds is because the request has failed ... and this is not being reported properly, due to server or client-side implementation issues.
UPDATE - It turns out that the real issues was that "curl" was behaving strangely on some platforms, probably due to network config issues.