No mapping for the Unicode character exists in the target multi-byte code page : how to convert MemoryStream to String

Question

My goal is to be able to convert TMemoryStream to string. I have this code to get the data into the TMemoryStream:

var
  idHttp : TIdHTTPEx;
  url : string;
  slTemp : TStringList;
  memoryStream : TMemoryStream;
begin
  try

    idHttp := TIdHTTPEx.Create(nil);
    slTemp := TStringList.Create;
    memoryStream := TMemoryStream.Create;
    try
      url := GetURL;
      SetParams(slTemp);

  idHttp.Request.Accept := 'application/json, text/javascript, */*; q=0.01';
  idHttp.Request.AcceptEncoding := 'gzip, deflate, br';
  idHttp.Request.AcceptLanguage := 'en-US,en;q=0.9';
  idHttp.Request.CacheControl := 'no-cache';
  idHttp.Request.Connection := 'keep-alive';
  idHttp.Request.ContentLength := 16;
  idHttp.Request.ContentType := 'application/x-www-form-urlencoded; charset=UTF-8';

      idHttp.Post(url, slTemp, memoryStream);
      Result := MemoryStreamToString(memoryStream);
    finally
      memoryStream.Free;
      slTemp.Free;
      idHttp.Free;
    end;

  except on E : Exception do
    begin
      Result := 'e:' + E.Message;
    end;
  end;
end;

And this is my code to convert it to a string:

function MemoryStreamToString(MemoryStream : TMemoryStream): string;
var
  StringStream: TStringStream;
begin
  Result:='';

    StringStream:= TStringStream.Create('', TEncoding.UTF8);
  try
    MemoryStream.Position := 0;
    StringStream.CopyFrom(MemoryStream, MemoryStream.Size);

    Result:= StringStream.DataString;
    Result := Result;
  finally
    FreeAndNil(StringStream);
  end;

end;

My function works fine in most of the conversion but not this. I checked these links: link1, link2 but they are different than my situation. I tried link3 too but still fail.

Any idea how to solve the problem?

Answer 1

You don't need to decode the raw data manually. Just let TIdHTTP do it for you. The Post() method has an overload that returns a decoded string:

Result := idHttp.Post(url, slTemp);

Also, you need to get rid of this line completely:

idHttp.Request.AcceptEncoding := 'gzip, deflate, br';

Otherwise TIdHTTP will not be able to decode the response correctly if the server decides to send a compressed response. You are manually giving the server permission to do so, but you are not setting up the TIdHTTP.Compressor property so TIdHTTP can handle decompression. Do not set the AcceptEncoding manually unless you are willing and able to manually detect and decode a response that has been actually been encoded in one of the formats you specify. Otherwise, just let TIdHTTP manage the AcceptEncoding property internally based on its actual capabilites.