Using requests and concurrent.futures to send several API post requests asynchronously

Question

I've been trying to use concurrent.futures in addition to requests in order to send several DIFFERENT direct messages from several DIFFERENT users. The purpose of the app I am designing is to send these direct messages as fast as possible and sending each request individually was taking too long.

The code below is something that I've tried working on but I have clearly found out that futures will not read requests stored in an array.

Any suggestions on how to go about doing this would be greatly appreciated.

from concurrent import futures
import requests
from requests_oauthlib import OAuth1
import json
from datetime import datetime

startTime = datetime.now()

URLS = ['https://api.twitter.com/1.1/direct_messages/new.json'] * 1

def get_oauth():
    oauth = OAuth1("xxxxxx",
                client_secret="zzzxxxx",
                resource_owner_key="xxxxxxxxxxxxxxxxxx",
                resource_owner_secret="xxxxxxxxxxxxxxxxxxxx")
    return oauth

oauth = get_oauth()

req = []


def load_url(url, timeout):
    req.append(requests.post(url, data={'screen_name':'vancephuoc','text':'hello pasdfasasdfdasdfasdffpls 1 2 3 4 5'}, auth=oauth, stream=True, timeout=timeout))
    req.append(requests.post(url, data={'screen_name':'vancephuoc','text':'hello this is tweetnumber2 1 2 3 4 5 7'}, auth=oauth, stream=True, timeout=timeout))



with futures.ThreadPoolExecutor(max_workers=100) as executor:
    future_to_url = dict((executor.submit(req, url, 60 ), url)
                         for url in URLS)
    for future in futures.as_completed(future_to_url):
        url = future_to_url[future]
        print ("DM SENT IN")
        print (datetime.now()-startTime)
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))

Answer 1

It may be worth to take look at some existing libraries that try to simplify using concurrency with requests.

From: http://docs.python-requests.org/en/latest/user/advanced/#blocking-or-non-blocking

[..] there are lots of projects out there that combine Requests with one of Python’s asynchronicity frameworks. Two excellent examples are grequests and requests-futures.