TIdHTTP character encoding of POST error response

I'm using Delphi 7 with Indy 10.6.2.5459 to make a POST request to a server. It's all working fine, except for the response encoding when an EIdHTTPProtocolException occurs.

When I don't get EIdHTTPProtocolException raised, I can decode the response like this to properly get special characters:

responseBody := '';
responseContent := TStringStream.Create('');
try
  try
    IdHTTP.Post(GetUrlMetodo(ASngpcCloudRequest.Tipo), requestBody, responseContent);
    responseBody := UTF8Decode(responseContent.DataString);
  except
    on E: EIdHTTPProtocolException do
      responseBody := UTF8Decode(E.ErrorMessage);
  end;
finally
  FreeAndNil(responseContent);
end;

But, when an EIdHTTPProtocolException is raised, the E.ErrorMessage property has ? instead of the expected special characters, even when using UTF8Decode().

So, how can I properly decode the E.ErrorMessage?

Solution

Since you are using Delphi 7, the native string type is AnsiString, which is important to note because it means Indy has to do more work when decoding strings.

When TIdHTTP parses an HTTP response body into a string, it is first decoded into Unicode using the response charset, as reported by the server. If the server sends a response in UTF-8 encoding, for instance, it is required to specify that in the charset attribute of the Content-Type header.

In pre-Unicode versions of Delphi/FreePascal, that Unicode data is then converted to ANSI in order to fit in an AnsiString. In those compiler versions, TIdHTTP methods which return a string have an optional ADestEncoding parameter to let you specify which AnsiString encoding you want the Unicode data converted to. If not specified, Indy's default encoding is used, which is US-ASCII by default (see the global GIdDefaultTextEncoding variable in the IdGlobal unit).

You really should let Indy handle this decoding for you, since there is no guarantee that any given response with be UTF-8 encoded. You can, however, specify that you want Indy's output to always be UTF-8 encoded (pre-Unicode versions only), eg:

try
  responseBody := UTF8Decode(
    IdHTTP.Post(GetUrlMetodo(ASngpcCloudRequest.Tipo), requestBody,
      IndyTextEncoding_UTF8)
  );
except
  on E: EIdHTTPProtocolException do
    responseBody := E.ErrorMessage;
end;

If you ever upgrade to a Unicode version of Delphi, you can then simply remove the extra UTF-8 step:

try
  responseBody := IdHTTP.Post(GetUrlMetodo(ASngpcCloudRequest.Tipo), requestBody);
except
  on E: EIdHTTPProtocolException do
    responseBody := E.ErrorMessage;
end;

In the example you provided in your question, you are bypassing TIdHTTP's automated decoding logic, receiving the raw response body as-is into a TStream instead of a string. In which case, you are then responsible for making sure you check the response's charset to know how to decode the raw data properly. It may not always be UTF-8. Indy has ReadStringFromStream() and ReadStringAsCharset() functions which allow you to specify an encoding/charset when reading a string from a TStream.

Now, to answer your question, why can't you decode the EIdHTTPProtocolException.ErrorMessage correctly? Well, because it has already been decoded by TIdHTTP for you.

HOWEVER, here is the rub - when decoding an error response to put into EIdHTTPProtocolException, the ADestEncoding parameter is NOT currently accessible from the code that is raising the exception, so Indy's default encoding gets used instead, and that is US-ASCII by default. So that is why you are seeing "special" characters getting converted to ? (again, this only affects pre-Unicode versions of Delphi/FreePascal).

You have a couple of options to work around this issue:

set the global IdGlobal.GIdDefaultTextEncoding variable to encUTF8 before calling Post(). This way, if EIdHTTPProtocolException is raised, its ErrorMessage will be UTF-8 encoded. Note that this does affect Indy globally, and has much more influence in pre-Unicode versions of Delphi than in Unicode versions, so be careful with it.
```
GIdDefaultTextEncoding := encUTF8;        
...
try
  ...
  responseBody := ...;
except
  on E: EIdHTTPProtocolException do
    responseBody := UTF8Decode(E.ErrorMessage);
end;
```
since you are saving both success and failure responses to the same responseBody variable, you may as well just disable the use of EIdHTTPProtocolException altogether, and remove your try/except block. You can do this by enabling the hoNoProtocolErrorException and hoWantProtocolErrorContent flags in the TIdHTTP.HTTPOptions property before calling Post(). You can check the TIdHTTP.ResponseCode property to differentiate between success and failure responses:
```
IdHTTP.HTTPOptions := IdHTTP.HTTPOptions + [hoNoProtocolErrorException, hoWantProtocolErrorContent];
responseBody := UTF8Decode(
  IdHTTP.Post(GetUrlMetodo(ASngpcCloudRequest.Tipo), requestBody,
    IndyTextEncoding_UTF8)
);
```