How to do concurrent jsonrpc using gevent and requests in volttron

I am trying to do concurrent JSONRPC calls to remote agents in Volttron. And am using Volttron 5.1.0 (with gevent 1.1.2 & requests 2.11.1).

The code works as expected. However, from the log files, I notice that the requests are not run concurrently. I am not sure what I am missing.

I tried suggestions mentioned in how enable requests async mode? (i.e,, handling async part in the request, monkey patch, etc.,). But none work-out, either the solution is outdated or required modules not available in volttron env (I am a bit worried about version requirements).

Any suggestions or inputs would be of great help.

The relevant part of the agent code is as follows:

Agent Code

import gevent
import requests

url_roots = ['http://192.168.1.51:8080/', 'http://192.168.1.52:8080/']
jobs = [gevent.spawn(do_rpc, self._agent_id, url_root, 'pricepoint'
                                , get_json_params()
                                ) for url_root in url_roots
                                ]
gevent.joinall(jobs, timeout=11)


def do_rpc(id, url_root, method, params=None):
    result = False
    json_package = {
        'jsonrpc': '2.0',
        'id': id,
        'method':method,
    }
    json_package['params'] = params
    response = requests.post(url_root, data=json.dumps(json_package), timeout=10)
    if response.ok:
        if 'result' in response.json().keys():
            success = response.json()['result']
            if success:
                result = True
    return result

Log

2020-02-19 21:12:15,913 (xyzagent-0.4 28079) xyz.ispace_msg_utils DEBUG: validate_bustopic_msg()
2020-02-19 21:12:15,918 (xyzagent-0.4 28079) xyz.agent DEBUG: New price point (pp) msg on the local-bus, topic: building/pricepoint ...
2020-02-19 21:12:15,919 (xyzagent-0.4 28079) xyz.agent DEBUG: ***** New bid price point from local: 0.20 price_id: 2218566
2020-02-19 21:12:15,931 (xyzagent-0.4 28079) xyz.agent DEBUG: post_ds_new_pp()...
2020-02-19 21:12:15,932 (xyzagent-0.4 28079) xyz.agent DEBUG: us pp messages count: 1...
2020-02-19 21:12:15,933 (xyzagent-0.4 28079) xyz.agent DEBUG: processing pp msg 1/1, price id: 2218566
2020-02-19 21:12:15,938 (xyzagent-0.4 28079) xyz.agent DEBUG: new ttl: 28.
2020-02-19 21:12:15,942 (xyzagent-0.4 28079) xyz.agent DEBUG: _ds_rpc_1_to_m()...
2020-02-19 21:12:15,953 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.51
2020-02-19 21:12:16,079 () volttron.platform.web DEBUG: {'jsonrpc': '2.0', 'id': '2503402', 'result': True}
2020-02-19 21:12:16,080 () volttron.platform.web DEBUG: res is a dictionary.
2020-02-19 21:12:16,238 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-19 21:12:16,245 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.52
2020-02-19 21:12:16,526 (xyzagent-0.4 28079) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-19 21:12:16,529 (xyzagent-0.4 28079) xyz.agent DEBUG: post pp to ds (ZoneController-51), result: success!!!
2020-02-19 21:12:16,529 (xyzagent-0.4 28079) xyz.agent DEBUG: post pp to ds (ZoneController-52), result: success!!!
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: _ds_rpc_1_to_m()...done
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: msg successfully posted to all ds, removing it from the queue
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: reset the retry counter for success ds msg
2020-02-19 21:12:16,530 (xyzagent-0.4 28079) xyz.agent DEBUG: post_ds_new_pp()...done

Solution

The below changes did the trick. Please note that just monkey.patch_all() breaks volttron. Need to set flag thread=False.

Agent Code

import gevent
from gevent import monkey
monkey.patch_all(thread=False, select=False)
import requests

Logs

2020-02-20 14:36:29,981 (xyzagent-0.4 8657) xyz.agent DEBUG: _ds_rpc_1_to_m()...
2020-02-20 14:36:29,987 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.51
2020-02-20 14:36:29,992 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool INFO: Starting new HTTP connection (1): 192.168.1.52
2020-02-20 14:36:30,260 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-20 14:36:30,333 (xyzagent-0.4 8657) requests.packages.urllib3.connectionpool DEBUG: "POST /bridge HTTP/1.1" 200 53
2020-02-20 14:36:30,341 (xyzagent-0.4 8657) xyz.agent DEBUG: post pp to ds (ZoneController-51), result: success!!!
2020-02-20 14:36:30,342 (xyzagent-0.4 8657) xyz.agent DEBUG: post pp to ds (ZoneController-52), result: success!!!
2020-02-20 14:36:30,344 (xyzagent-0.4 8657) xyz.agent DEBUG: _ds_rpc_1_to_m()...done