asynchronous HTTP POST requests in Python

Question

I have a code which posts HTTP request sequentially for all the tuples in a list and append the response to another list.

import requests

url = 'https://www....'
input_list = [(1,2),(6,4),(7,8)]
output_list = []

for i in input_list:
   resp = (requests.post('url',data = i[1]).text,i[0])
   output_list.append(resp)

print(output_list)

Can someone please help me with the directions to make the HTTP requests in parallel ?

Answer 1

Since requests library doesn't support asyncio natively, I'd use multiprocessing.pool.ThreadPool, assuming the most of the time is spent waiting for IO. Otherwise it might be beneficial to use multiprocessing.Pool

from multiprocessing.pool import ThreadPool
import requests

url = 'https://www....'
input_list = [(1,2),(6,4),(7,8)]

def get_url(i):
  return (requests.post('url',data = i[1]).text,i[0])

with ThreadPool(10) as pool: #ten requests to run in paralel
  output_list = list(pool.map(get_url, input_list))