Post list field In urllib3

Question

I'm creating a simple AWS lambda function to post some data to an API but the API expects a list however I can't get it to work. I've got the following code that I'm testing with

import urllib3
import json

http = urllib3.PoolManager()
file_path = 'file.png'
with open(file_path, 'rb') as fp:
  file_data = fp.read()

payload = {
  'from': ['a', 'b', 'c'],
  'file': ('file.png', file_data),
}

r = http.request(
  'POST',
  'http://httpbin.org/post',
  fields=payload
)

json.loads(r.data.decode('utf-8'))

But this returns an error about expecting the field to be a certain type and basically not being allowed to be a list which is fine, but how can I make this work.

I'm connecting to a Rails based server, and in order for a form-encoded form submission to be a list, I need to send multiple params with a name containing [] on the end, but of course if I try creating a dict with multiple keys the same, it'll just ignore them all so I'm not really sure how to handle this.

I'm pretty new to python so sorry if this is a stupid question

Answer 1

You have two problems here, one is posting multiple values of the same key and the other is posting form data with file data in one request using urllib3.

You can use tuples to resolve the first problem and use urllib3.fields.RequestField to resolve the second problem like this:

"""
Following https://urllib3.readthedocs.io/en/stable/reference/urllib3.fields.html
"""
import urllib3
from urllib3.fields import RequestField

http = urllib3.PoolManager()
FILE_PATH = 'file.png'
with open(FILE_PATH, 'rb') as file_handle:
    file_data = file_handle.read()
fields = []
request_field = RequestField(name='file', data=file_data, filename=FILE_PATH)
content_disposition = f'form-data; size={len(file_data)}'
request_field.make_multipart(content_disposition=content_disposition, content_type='image/png')
fields.append(request_field)
fields.append(('from[]', 'a'))
fields.append(('from[]', 'b'))
fields.append(('from[]', 'c'))
response = http.request(
    'POST',
    'http://httpbin.org/post',
    fields=fields
)
print(response.data.decode('utf-8'))

Note multiple values for the same key are added as multiple tuples, not a list.

The output (with the image as a single pixel png) is:

{
  "args": {}, 
  "data": "", 
  "files": {
    "file": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKAAAAA1BMVEUAAACnej3aAAAAAXRSTlMAQObYZgAAAApJREFUCNdjYAAAAAIAAeIhvDMAAAAASUVORK5CYII="
  }, 
  "form": {
    "from[]": [
      "a", 
      "b", 
      "c"
    ]
  }, 
  "headers": {
    "Accept-Encoding": "identity", 
    "Content-Length": "537", 
    "Content-Type": "multipart/form-data; boundary=e6441032102e8f3fc564107bcd9ea02f", 
    "Host": "httpbin.org", 
    "User-Agent": "python-urllib3/1.26.9", 
    "X-Amzn-Trace-Id": "Root=1-64447758-4cbca2b82c1917ff5a8cebc7"
  }, 
  "json": null, 
  "origin": "xxx.xxx.xxx.xxx", 
  "url": "http://httpbin.org/post"
}

I took the liberty of renaming some of your variables following Pylint Predefined Naming Styles.