Multiple http get and post

Question

The task is:

1)send an http get to url based on a parameter
2)Modify the response based on the same parameter
3)send an http post to url based on the same parameter

I am currently doing this through requests library, but it takes a lot of time to do this one by one and it can be upto 20,000.

I tried multiprocessing but for some reason it hangs after sending 5000-10000 get and post.

I read about grequest but it says there Order of these responses does not map to the order of the requests you send out..I need the order as i have to modify each reponse based on the get i sent.

What is the best option here?I have read about threading,tornado too but as i have messed up my first approach with multiprocessing i want to be sure before getting onto it again

Answer 1

Here is a solution that allows you to use grequest's imap (which is theoretically faster than grequest's map function) and know an index to map a response to a request. Credit to a question asked on the project's GitHub issues.

from functools import partial

def callback(index, response, **kwargs):
    response.image_index = index

rs = [
    grequests.get(
        url,
        callback=partial(callback, index)
    )
    for index, url in enumerate(urls)
]

You should be able to tailor this to suit your needs.

EDIT: I used this successfully with hooks.

grequests.get(
        url,hooks={'response': partial(process_response, index)})