How to combine Flask-RESTPlus with Requests: HTTP for Humans?

Question

I'm using Flask-RESTPlus to make an endpoint and using Requests: HTTP for Humans to scraping product details from one of e-commerce site.

I have got the product details perfectly using Requests, but when I combine it with FLASK-RESTPlus, I'm facing some problems.

Here is the snippet of the code:

@api.route('/product')
class ProductDetails(Resource):
    query_parser = api.parser()
    query_parser.add_argument('marketplace', required=True, location='args')

    def post(self, query_parser=query_parser):
        args = query_parser.parse_args()
        url = args['marketplace']

        try:
            response = requests.post(
                url=args,
                json={
                     'url': url
                }, timeout=60
           )
            }
        except Exception as e:
           return {
                'status': 'error',
                'message': str(e)
            }

When I try to access the endpoint

http://localhost:5000/api/v1/product?marketplace=http://xx.xx.xx.xx/v1/markeplace_name/url

am always got this error:

{
    "status": "error",
    "message": "No connection adapters were found for '{'marketplace': 'http://xx.xx.xx.xx/v1/market_place_name/url'}'"
}

The things that make me confuse are why I can get the product details before.

So, what's wrong with my code?, any example or source to learn would be great.

Answer 1

The issue is that you are passing the args dict to requests.post as the url parameter. Requests validates that the url you supply to .post() is valid and a url that starts with {'marketplace': ...} is obviously an invalid url.

This part of the code:

response = requests.post(
    url=args,
    json={
         'url': url
    }, timeout=60
)

And args = query_parser.parse_args().

As you asked for source to help you learn, this is the code where requests checks the adapter at the start of the url which you can find in the sourcecode here:

def get_adapter(self, url):
    """
    Returns the appropriate connection adapter for the given URL.
    :rtype: requests.adapters.BaseAdapter
    """
    for (prefix, adapter) in self.adapters.items():

        if url.lower().startswith(prefix.lower()):
            return adapter

    # Nothing matches :-/
    raise InvalidSchema("No connection adapters were found for '%s'" % url)

The self.adapters.items() that is used to check the url comes from here:

# Default connection adapters.
self.adapters = OrderedDict()
self.mount('https://', HTTPAdapter())
self.mount('http://', HTTPAdapter())

And the mount() method essentially maps the expected url prefix to one of requests' connection adapter types in the self.adapters dict.