Wget says "No headers" receiving my HTTP response

Question

I'm running a server on a Raspberry Pi Pico W. I'd like the HTTP response to be consumable by Wget, cURL, Python Requests and browsers but I'm having trouble getting my HTTP response understood.

Chrome parses response headers, Firefox does not, cURL and Python Requests think there are no HTTP headers and close the connection, and Wget thinks headers are part of response body:

WiFiServer server(80); // server on port 80
void loop(){
  WiFiClient wificlient = server.available();   // Listen for incoming clients

  // If a new client connects,
  if (wificlient) {
    HttpClient client(wificlient, WiFi.localIP());
    net_time = millis();
    net_prev_time = net_time;

    while ((client.connected() || client.available()) && (!client.endOfBodyReached()) &&
           (net_time - net_prev_time <= net_timeout))
    {
      net_time = millis();
      if (client.available()) {
        char c = client.read(); // read a byte
        Serial.write(c);
        httpbody += c;

        if (!client.available()) { // client is done, respond
          if (httpbody.indexOf("GET /api/last") == 0){
            Serial.println("Serving api request...");
            client.println("HTTP/1.1 200 OK");
            client.println("Content-type: text/plain; charset=utf-8");
            client.println("Content-length: " + String(last_str.length()));
            client.println("Content-Disposition: inline");
            client.println("Connection: close");
            client.println("");
            client.println(last_str);
            client.println("");
            break;
         }
       }
    }
...
}

client is an instance of HttpClient from ArduinoHttpClient library (via the arduino-pico core). Requesting with Wget it says:

No headers, assuming HTTP/0.9

$ wget -O - http://10.0.0.27/api/last
--2023-02-24 22:09:34--  http://10.0.0.27/api/last
Connecting to 10.0.0.27:80... connected.
HTTP request sent, awaiting response... 200 No headers, assuming HTTP/0.9
Length: unspecified
Saving to: ‘STDOUT’

-
HTTP/1.1 200 OK
Content-type:text/plain; charset=utf-8
Content-length:136
Content-Disposition: inline
Connection: close

3006, MHZTemp (C): 17, Temp (C): 22.40, Humidity (%): 30.00, Feels like (C): 21.48, CO2 (ppm): 653, Dust (pcs/283ml): 599.09, GPS Fix: 0
-

Using cURL I get:

Received HTTP/0.9 when not allowed

How should I format HTTP response correctly?

Answer 1

Substituting WifiClient instead of HttpClient solves the issue, and both wget and curl no longer complain of missing headers or HTTP/0.9. Looking at the traffic in wireshark, it appears that when WifiClient is responding it sends a TCP [PSH, ACK] packet with the HTTP status code whereas and then after wget or curl respond with a TCP [ACK] WifiClient responds with the rest of the data. HttpClient was not sending [PSH, ACK] (probably because it isn't meant to) and curl closes the connection in response to more data.

In the following is the wireshark traffic with different configurations. 10.0.0.27 is my Pico server and 10.0.0.193 is my desktop machine (the requestor).

Requesting with curl and responding with WifiClient (expected behaviour):

Requesting with curl and responding with HttpClient (connection gets closed):

So I think the PSH packet is the answer. WifiClient knows to send it while HttpClient isn't meant for this use and doesn't send it.