NodeMcu using WiFiClientSecure readStringUntil() taking 15 seconds to complete

Question

I have the following code:

if (net.connect(host, port)) {
  String req = "GET /curTemp?temp=" + String(temperatureFahrenheit) + " HTTP/1.1\r\n" + 
               "Host: " + host + "\r\n" +
               "Authorization: Basic xxxxxxxxxxxxxxxxx\r\n" +
               "Connection: close\r\n\r\n";
  net.print(req);

  // Get headers
  while (net.connected()) {
    String line = net.readStringUntil('\r');
    if (line == "\r") {
      break;
    }
  }

  // Get temperature
  StaticJsonBuffer<50> jsonBuffer;
  JsonObject& root = jsonBuffer.parseObject(net.readStringUntil('\r'));
  const char* temp = root["temp"];
  Serial.print("Temp:");
  Serial.println(temp);
} else {      
  Serial.println("connection failed");
}

This is taking 15 seconds to complete and I'm not sure why. I can use the same request in a web browser and it comes back immediately. It specifically is the net.readStringUntil that seems to be taking the time.

UPDATE: I found a workaround by setting setTimeout(1000) but I'm not understanding why this is necessary. Shouldn't the request close the connection and readStringUntil() terminate when complete? Maybe I don't understand WiFiClientSecure?

UPDATE2: I found the problem. See answer below.

Answer 1

Ok, I'm putting this in to clarify. Every example shows '\r' for readStringUntil() which can be confusing. So what happens is if you use '\r' then the next read will begin with '\n' so if you are looking for the end of the headers, it will actually be '\n\r\n' because of the carry over from the previously read line. So what happens is your data is out of sync with crlfs. I found that using '\n' for the terminator works better because the standard format for headers ends with '\r\n'. Then you can test for end of headers by looking for just '\r' (the \n terminator gets eaten by the readStringUntil()). You can then start another loop to get your data. Then everything works fine with no need of setTimeout.

Example:

if (net.connect(host, port)) {
  String req = "GET /curTemp?temp=" + String(temperatureFahrenheit) + " HTTP/1.1\r\n" + 
               "Host: " + host + "\r\n" +
               "Authorization: Basic dXNlcjpkYWYxMjM0\r\n" +
               "Connection: close\r\n\r\n";
  net.print(req);

  delay(1000);

  // Get headers
  while (net.available()) {

    // Note \n for terminator
    String line = net.readStringUntil('\n');

    // Only \r because \n is eaten by readStringUntil()
    if (line == "\r") {
      break;
    }
  }

  // Now look for data (in my case only one line of JSON)

  // No \r\n on this one.
  String line = net.readStringUntil('}');

  // Put back the character eaten by readStringUntil()
  line = line + "}";

  Serial.println(line);