Calling curl from Python - stderr is different from when running on the commend line. Why?

When running curl on the command line against a domain that doesn't exist, I get an error message as expected

$ curl https://doesnoexist.test/
curl: (6) Could not resolve host: doesnoexist.test

But if I do the same thing from Python, printing the standard error

import subprocess

proc = subprocess.Popen(['curl', 'https://doesnoexist.test/'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
outs, errs = proc.communicate()
print(errs)

then I get the the start of what seems to be a download progress indicator, and then the same error message right after

$ python curl.py
b'  % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current\n                                 Dload  Upload   Total   Spent 
   Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:--
 --:--:-- --:--:--     0curl: (6) Could not resolve host: doesnoexist.test\n'

Why? How can I only get the error message in Python?

(Ideally answers won't just be for curl, but more general where similar things happen when running other processes)

Solution

Essentially curl seems to test whether stdout is connected to a tty (ie. whether it is interactive, essentially it does the equivalent of sys.stdout.isatty()) and if it isn't add a progress meter by default (which makes sense, since then the output will not mingle with stderr on the terminal).

You can test this directly in the shell by comparing what you get with redirection to output

$ curl https://test.doestaa > output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: test.doestaa

and without

$ curl https://test.doestaa
curl: (6) Could not resolve host: test.doestaa

In python this means you could now just leave stdout alone:

from subprocess import run, PIPE
proc = run(["curl", "https//test.doestaa/"], stderr=PIPE)
print(proc.stderr)

to get the same effect, but without the possibility to get to its output.

Or you would have to set up a pty and connect it there like with ptyprocess, but you should indeed just use curl's --no-progress-meter flag or consider using a python http client like the requests or HTTPX libraries, instead.