Search code examples
httppython-requestsrequestpostman

http request difference between postman request and the curl cmd or python code it generates


I have a postman request that gets me the correct response from server, but when I use the curl command or python script it generates from the code snippet section, it fails. But they should be equivalent? I hope to understand how the server sees the request differently from the error messages I got, no access to the server itself though.

The curl command postman gives is

curl --location --request POST 'http://10.50.50.50:8081/v1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "min_tokens": 128
}'

the error I get from server is Access Denied, so I think it is reaching the server but somehow different then the postman request and getting denied? what other part of postman request that is not being captured by the curl command?

I also tried the python http client lib code from postman

conn = http.client.HTTPSConnection(url, port,  context = 
ssl._create_unverified_context())
payload = json.dumps({
  "min_tokens": 128
})
headers = {
  'Content-Type': 'application/json'
}
conn.request("POST", "/v1", payload, headers)
res = conn.getresponse()

this time I got some ssl error

self._sslobj.do_handshake() ssl.SSLError: [SSL] record layer failure (_ssl.c:1000)

I also tried the same for python request library

  payload = json.dumps({
    "min_tokens": 128
  })
  headers = {
    'Content-Type': 'application/json'
  }

  try:
      response = requests.post(url, data=payload, headers=headers, verify=False, proxies={})
      response.raise_for_status()  # Raise an exception for bad status codes
      print(response.text)
  except requests.exceptions.RequestException as e:
      print("Error:", e)

and get a 403

Error: 403 Client Error: Forbidden for url: http://10.50.50.50:8081/v1

given that the request working in postman with just one header and body, what else is different in curl or python code? I am behind a company proxy and have a custom CA installed, postman by default uses them the same way curl or python lib should? I have tried unverified ssl and with/without proxy but same response from server


Solution

  • CURL request

    Your request is directed to http://10.50.50.50:8081/v1 using the HTTP protocol (Hypertext Transfer Protocol). That means, no transport layer encryption. Also note, that there is no authentication data included in the request (which explains your 'Access Denied' error.

    curl --location --request POST 'http://10.50.50.50:8081/v1' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "min_tokens": 128
    }'
    

    If you want additional output, you can use the -v flag. This will make CURL emit additional output. Which is pretty handy for debugging.

    Example output

    $ curl http://google.de// -v
    * Host google.de:80 was resolved.
    * IPv6: 2a00:1450:4001:831::2003
    * IPv4: 142.250.184.195
    *   Trying [2a00:1450:4001:831::2003]:80...
    * Connected to google.de (2a00:1450:4001:831::2003) port 80
    > GET // HTTP/1.1
    > Host: google.de
    > User-Agent: curl/8.9.1
    > Accept: */*
    >
    * Request completely sent off
    < HTTP/1.1 301 Moved Permanently
    < Location: http://www.google.de/
    < Content-Type: text/html; charset=UTF-8
    < Content-Security-Policy-Report-Only: object-src 
    ...
    

    Python request

    Your python request uses the HTTPS protocol (Hypertext Transfer Protocol Secure), which uses TLS (Transport Layer Security) for transport encryption (default port is 443). For these security features, both the server and the client have to exchange the encryption details before they can send any payloads.

    conn = http.client.HTTPSConnection(url, port,  context = ssl._create_unverified_context())
    payload = json.dumps({
      "min_tokens": 128
    })
    headers = {
      'Content-Type': 'application/json'
    }
    conn.request("POST", "/v1", payload, headers)
    res = conn.getresponse()
    

    this time I got some ssl error

    self._sslobj.do_handshake() ssl.SSLError: [SSL] record layer failure (_ssl.c:1000)
    

    Without knowing the library, I would say, this problem is caused by the server, which refuses to establish a connection. In other words, the server terminates the connection immediatly without sending any response (that is your handshake error).

    You would have to use the protocol selector https://... instead of http://..., since you explicitly try to establish a HTTPS connection in your code.

    Alternate client

    In your third example, you use another approach, which is similar to your CURL command. This code does not force a secure connection! If you use the same URL here, there must be a response from the server.

      payload = json.dumps({
        "min_tokens": 128
      })
      headers = {
        'Content-Type': 'application/json'
      }
    
      try:
          response = requests.post(url, data=payload, headers=headers, verify=False, proxies={})
          response.raise_for_status()  # Raise an exception for bad status codes
          print(response.text)
      except requests.exceptions.RequestException as e:
          print("Error:", e)
    

    As expected, the server sends a response, verifying that the connection was established.

    and get a 403

    Error: 403 Client Error: Forbidden for url: http://10.50.50.50:8081/v1
    

    The HTTP status 403 is defined as "The server understood the request but refuses to authorize it."; further "If authentication credentials were provided in the request, the server considers them insufficient to grant access.".

    Since you do not provide any authorization headers in your request, this could in turn indicate, that the resource/endpoint your request is targeted at, is not public or a proxy services fobids the connection.

    Try to find out what kind of authentication data postman includes and add it to your CURL command or python code. For servers, that answer with the HTTP status 401, there is an authentication challenge sent with the response. More information can be found in RFC 2617, which defines simple HTTP authentication methods.