UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 8024: character maps to <undefined> when re-directing output to disk

On Windows 10, I'm using httpie to retrieve data. It works fine when displaying on the screen, but as soon as I pipe the output to disk, I get http: error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 8024: character maps to <undefined>

Is there a way to change the encoding when writing to disk?

Here is the complete debug output:


>> http --debug --print=Bb --pretty=all --session=C:\Users\PETERK~1\AppData\Local\Temp/access.json http://localhost:8080/nexus/extract/path  filePath=c:/TestFiles/100Mb.pdf > foo
HTTPie 2.3.0
Requests 2.25.1
Pygments 2.7.3
Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]
c:\python\python39\python.exe
Windows 10

<Environment {'colors': 256,
 'config': {'default_options': []},
 'config_dir': WindowsPath('C:/Users/PeterKronenberg/AppData/Roaming/httpie'),
 'devnull': <property object at 0x000001112C1815E0>,
 'is_windows': True,
 'log_error': <function Environment.log_error at 0x000001112C1AC1F0>,
 'program_name': 'http',
 'stderr': <colorama.ansitowin32.StreamWrapper object at 0x000001112C176F10>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <colorama.ansitowin32.StreamWrapper object at 0x000001112C176BE0>,
 'stdout_encoding': 'cp1252',
 'stdout_isatty': False}>

>>> requests.request(**{'auth': None,
 'data': '{"filePath": "c:/TestFiles/100Mb.pdf"}',
 'headers': {'User-Agent': b'HTTPie/2.3.0', 'Accept': b'application/json, */*;q=0.5', 'Content-Type': b'application/json', 'authorization': b'eyJhbGciOiJSUzI1NiIsImtpZCI6InB1YmxpYzplMjM3ODE1Ny04MmY0LTQ2YjItYWI5NS1hMDRhMzg4ZGY5ZTQiLCJ0eXAiOiJKV1QifQ.eyJhdWQiOltdLCJjbGllbnRfaWQiOiIxMTExMTExMS0xMTExLTExMTEtMTExMS0xMTExMTExMTExMTUiLCJleHAiOjE2MDk0NDAxNzYsImV4dCI6e30sImlhdCI6MTYwOTQzNjU3NiwiaXNzIjoiaHR0cHM6Ly9vaWRjLnRvcmNoLmFpLyIsImp0aSI6IjZkN2QwOTFkLTQyMTctNGRmMS04ZmY3LTk2MDdiMzkzZjBlZCIsIm5iZiI6MTYwOTQzNjU3Niwic2NwIjpbXSwic3ViIjoiMTExMTExMTEtMTExMS0xMTExLTExMTEtMTExMTExMTExMTE1In0.QeCNhpoE-1UyLcHer2_JjhiMx3qKV1qIC3LX0rlRUKSIK5vlMbJ2KmVP1mNR647SJhAxN9zdqf9vDaG6Fk0ZRLBFSBbcq0WuyTlaJsPVVggpe9ys_mcqO8uPMTl4-tO0H5kXVzRi_TFcmfCH8W3X8KAe3K7kjfs8O207EAenPr8n-KB12qUQgpC4ELXRbC_9SwuZY_ZhOic9POUYMCvrmJkMB8Th4kOqix5uI45lwowVQI9kTgAB4kQF8VelYJF7xxxn81U24T3ab9lRtp6ShY1mUkhDwfhrz2YdhYQ4uhQhKNZc3Hxz8OPSV_0Q-ykdZVgi03su51E_SE0odnryiUrR4nheu1cxmGmKNq8gb2CII9NktgrZEvvXlYJ5cV5mI2AAVlgkp4A0FBCd_wt39mXadIWPjv3YfilkzwUZCoeKA8dpp1mOKebFjq7Iw4LnxhObuKw1BnhYjHzdtQC5mhwFtc5iCeYChkQG_b8vybap12S4HzQ7HHFBKjdJZ2Il5mbuF01BAUBhZ1TO2GQ_t3O0JVrp8rmob75Yiza5FiBZaeAASSp2p0mb-_CmTVnQq3N-f099yL7byNYKl0toLIfgDLuib6bv_rhushdvEyWOD1c-nJnqMNFekEZ4NXTvBskL9PJCWcW-r8l1Fn567s7SAFan-G37YeQYBfmwADI'},
 'method': 'post',
 'params': <generator object MultiValueOrderedDict.items at 0x000001112CA5FB30>,
 'url': 'http://localhost:8080/nexus/extract/path'})


http: error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1617: character maps to <undefined>

Traceback (most recent call last):
  File "c:\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Python\Python39\Scripts\http.exe\__main__.py", line 7, in <module>
  File "c:\python\python39\lib\site-packages\httpie\__main__.py", line 11, in main
    exit_status = main()
  File "c:\python\python39\lib\site-packages\httpie\core.py", line 79, in main
    exit_status = program(
  File "c:\python\python39\lib\site-packages\httpie\core.py", line 222, in program
    write_message(
  File "c:\python\python39\lib\site-packages\httpie\output\writer.py", line 38, in write_message
    write_stream_with_colors_win_py3(**write_stream_kwargs)
  File "c:\python\python39\lib\site-packages\httpie\output\writer.py", line 82, in write_stream_with_colors_win_py3
    outfile.write(chunk.decode(encoding))
  File "c:\python\python39\lib\encodings\cp1252.py", line 15, in decode
    return codecs.charmap_decode(input,errors,decoding_table)
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1617: character maps to <undefined>

Solution

HTTPie is taking encoding from the system's stdout configuration. You have it configured for Windows 1252 and trying to write down data out of its capabilities (presumably utf8).

While HTTPie doesn't provide an option to separately configure the writer's output encoding, you can set output's default through PYTHONIOENCODING environment variable. Try this

PYTHONIOENCODING=utf-8 http --debug --print=Bb --pretty=all --session=C:\Users\PETERK~1\AppData\Local\Temp/access.json http://localhost:8080/nexus/extract/path  filePath=c:/TestFiles/100Mb.pdf > foo

I am actually not too familiar with Windows—if this wouldn't work, set run this before running HTTPie:

setx PYTHONIOENCODING "utf-8"

Note this will override encoding for all other Python scripts for this command line session.