Even with using HttpCompletionOption.ResponseHeadersRead like this:
var httpClient = new HttpClient();
httpClient.BaseAddress = new Uri("https://myserver.com");
var request = new HttpRequestMessage(HttpMethod.Get, "huge_file.bin");
var response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
As soon as SendAsync is called, HttpClient issues a GET request. This can be seen with Fiddler.
Under the hood, where does the underlying HttpClient actually store the the request response? Is it in memory? If it's in memory, what if the response is really huge, couldn't that be problem? In my tests, it does not seem to be in memory. Is it maybe in non-user memory, like in the kernel memory? I'm not familiar how does the network things work internally when downloading data.
Note that I'm not asking about consuming the response content by calling for example response.Content.ReadAsStringAsync(). I'm asking how does HttpClient reads and stores the response internally, before I start to consume it by calling response.Content.ReadAsStringAsync() for example.
It doesn't store it anywhere if you use ResponseHeadersRead. The response hasn't been fully sent by the server until you call ReadAs....
As far as seeing the request in Fiddler: yes the request is issued, but is it actually showing all the bytes being received in the response?
In any case, Fiddler is a transparent proxy, which is probably going to fully cache requests and responses in order to parse them. Try using Wireshark and watch the actual network packets, and you will see that the response stream is not fully read until you call ReadAsStream and actually read the bytes off the stream.
That's just how TCP works (HTTP runs on TCP sockets): it waits for an ACK after a certain amount of data. It might be more than just the headers though, as TCP commonly pushes through more than one packet before waiting for an ACK.
If you don't read the data after a long enough amount of time, the server will just break the connection.