delphi http-status-code-403 indy10 idhttp

Error 403 using TIdHTTP for a GET request in Delphi RIO - Same url works in Postman

I am having a problem.

I try various possible solutions, but something is still wrong in my code, and I can't figure out what is it.

Function:

function HTTPRestApiGet(var HTTP: TIdHTTP; necesitaSSL: Boolean; var SSL: TIdSSLIOHandlerSocketOpenSSL; Usuario, Clave, URLBase, Recurso, CamposSolicitados: string; TiempoEspera: Integer = 5000;
  LogActivado: Boolean = True): THTTPRestApiResponse;
var
  requerimiento, respuesta: string;
  LogFile                 : TIdLogFile;
begin
  LogFile := TIdLogFile.Create(nil);
  //
  HTTP.HandleRedirects := True;
  HTTP.RedirectMaximum := 10;
  HTTP.ReadTimeout := TiempoEspera;
  HTTP.MaxAuthRetries := 0;
  HTTP.HTTPOptions := [hoInProcessAuth];
  HTTP.Request.BasicAuthentication := True;
  HTTP.Request.Username := Usuario;
  HTTP.Request.Password := Clave;
  HTTP.Request.Accept := 'http';
  HTTP.Request.ContentType := 'application/json';
  //
  HTTP.IOHandler := nil;
  if necesitaSSL then
    begin
      HTTP.IOHandler := SSL;
      SSL.SSLOptions.Mode := sslmClient;
      SSL.SSLOptions.Method := sslvSSLv23;
    end;
  // Configurar logging
  LogFile.Filename := 'http_log.txt';
  LogFile.Active := True;
  HTTP.Intercept := LogFile;
  //
  try
    requerimiento := URLBase + IfThen(Recurso <> EmptyStr, '/' + Recurso + IfThen(CamposSolicitados <> EmptyStr, '?_fields=' + CamposSolicitados, ''), EmptyStr);
    try
      respuesta := HTTP.Get(requerimiento);
    except
    end;
  finally
    HTTPRestApiLogRespuesta('GET', requerimiento, respuesta, HTTP.URL.uri, HTTP.ResponseText, HTTP.ResponseCode, LogActivado);
    with Result do
      begin
        requerimiento_exitoso := (HTTP.ResponseCode >= 200) and (HTTP.ResponseCode < 300);
        respuesta_codigo := HTTP.ResponseCode;
        respuesta_codigotexto := HTTP.ResponseText;
        respuesta_texto := respuesta;
        respuesta_id := -1;
      end;
    //
    LogFile.Free;
  end;
end;

Call:

HTTP := TIdHTTP.Create(nil);
SSL := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
respuesta := HTTPRestApiGet(HTTP, True, SSL, 'brgroup', '1234', 'https://apis.datos.gob.ar/series/api/series/?ids=148.3_INIVELNAL_DICI_M_26&start_date=' + fecha_desde + '&end_date=' + fecha_hasta,
  EmptyStr, EmptyStr, 5000, False);
SSL.Free;
HTTP.Free;

Response (log file):

HTTP/1.1 403 Forbidden

Stat Connected.
Sent 03/08/2024 14:31:22: GET /series/api/series/?ids=148.3_INIVELNAL_DICI_M_26&start_date=2024-04-01&end_date=2024-07-01 HTTP/1.1<EOL>Content-Type: application/json<EOL>Host: apis.datos.gob.ar<EOL>Accept: http<EOL>User-Agent: Mozilla/3.0 (compatible; Indy Library)<EOL>Authorization: Basic Og==<EOL><EOL>
Recv 03/08/2024 14:31:22: HTTP/1.1 403 Forbidden<EOL>Date: Sat, 03 Aug 2024 17:31:18 GMT<EOL>Content-Type: text/plain; charset=UTF-8<EOL>Content-Length: 16<EOL>Connection: keep-alive<EOL>X-Frame-Options: SAMEORIGIN<EOL>Referrer-Policy: same-origin<EOL>Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0<EOL>Expires: Thu, 01 Jan 1970 00:00:01 GMT<EOL>Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=R4%2FV8fU73LVHwpLIjJ6e%2Bq91jVM76aHSX9KolkAQ2jd6gUdAGWjUl5yOZd728jdQb%2BoIx%2BzuQyFnqhINrrA76CWjQv%2Fw17ciaSSFRAmKOMSEBszwUe3s6K0en7dhPVXEC3wH5A%3D%3D"}],"group":"cf-nel","max_age":604800}<EOL>NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}<EOL>Server: cloudflare<EOL>CF-RAY: 8ad80a629ecaa78f-EZE<EOL>alt-svc: h3=":443"; ma=86400<EOL><EOL>error code: 1010

With Postman, same request, works fine:

curl --location 'https://apis.datos.gob.ar/series/api/series/?ids=148.3_INIVELNAL_DICI_M_26&start_date=2024-04-01&end_date=2024-07-01' \
--header 'Accept: http' \
--header 'Content-Type: application/json' \
--header 'Authorization: Basic Og=='

Can anyone help me with this?

I am trying various TIdHTTP options but it doesn't work. The function is working fine for other purposes.

Solution

You did not show the raw request data that curl is sending, but the only difference of consequence that I see between your TIdHTTP code and your curl command would be the User-Agent request header.

Many servers are sensitive to the requesting agent, where they send different data to different agents. It is not uncommon for such servers to reject/not recognize Indy's default User-Agent value.

When I tested your code, it indeed failed with Indy's default User-Agent value of Mozilla/3.0 (compatible; Indy Library), and when I instead set the TIdHTTP.Request.UserAgent property (or the global GIdDefaultUserAgent variable) to a value that mimicked a real web browser (in my case, Firefox), the request worked as expected and a JSON response was returned:

HTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0';

Given curl's popularity, it would not surprise me if agent-aware servers do recognize curl. FYI, curl's default User-Agent value is curl/[version] (eg: curl/8.9.1, which also worked in my TIdHTTP test).

When you are comparing curl requests to other HTTP libraries, you should always strive to make the requests be as close to identical as possible to eliminate any possible differences that can confuse servers. And when there are behavior differences between similar requests, it is typically the User-Agent at fault.

On a side note:

There is no reason to send a Content-Type header in a GET request, since there is no data being sent in the HTTP request body. On the other hand, if you want to tell the server that you accept only JSON in the response, then send an Accept: application/json request header instead.
'http' is not a valid media type for an Accept request header.
Setting the TIdHTTP.SSLOptions.Method property to sslvSSLv23 will enable SSL v2.0-v3.0 in addition to TLS v1.0-1.2. Don't do that! Use this instead:
```
HTTP.SSLOptions.SSLVersions := [sslvTLSv1,sslvTLSv1_1,sslvTLSv1_2];
```
You are not sending any authentication credentials. In your request's Authorization header in both cases:

Authorization: Basic Og==

Og== is the base64 encoding for a single ':' character, which separates the username and password. This would imply that your Usuario and Clave variables are blank strings, but that is not actually the case in the TIdHTTP code you have shown, so the Authorization header that TIdHTTP generates should have looked like the following instead, given Usuario=brgroup and Clave=1234:
```
Authorization: Basic YnJncm91cDoxMjM0
```
But, given that the server in question does not actually require authentication to begin with, there is no reason to send an Authorization header at all. I omitted it in my TIdHTTP test and was still able to get a JSON response.