Search code examples
pythonurlliburlencode

How can I escape certain characters while using python's urllib.urlencode()?

I have a dictionary that I want to urlencode as query parameters. The server that I am hitting expects the the query to look like this: http://www.example.com?A=B,C

But when I try to use urllib.urlencode to build the URL, I find that the comma gets turned into %2C:

>>> import urllib
>>> urllib.urlencode({"A":"B,C"})
'A=B%2CC'

Is there any way I can escape the comma so that urlencode treats it like a normal character?

If not, how can I work around this problem?

Solution

  • You can do this by adding the query params as a string before hitting the endpoint.

    I have used requests for making a request.

    For example: 

    GET Request

import requests

url = "https://www.example.com/?"
query = "A=B,C"

url_final = url + query

url  = requests.get(url_final)

print(url.url)
# https://www.example.com/?A=B,C

    The comma (along with some other characters) is defined in RFC 3986 as a reserved character. This means the comma has defined meaning at various parts in a URL, and if it is not being used in that context it needs to be percent-encoded.

    That said, the query parameter doesn't give the comma any special syntax, so in query parameters, we probably shouldn't be encoding it. That said, it's not entirely Requests' fault: the parameters are encoded using urllib.urlencode(), which is what is percent-encoding the query parameters.

    This isn't easy to fix though, because some web services use , and some use %2C, and neither is wrong. You might just have to handle this encoding yourself.