Python requests urlencode not working?

Question

EDIT: I figured out the problem. The # in #user_sex is not being converting to %23 by python requests. Is there a way to force python requests to convert # to %23 or will I just have to hand code that part?

I'm trying to make a facebook fql multiquery. When I use the fql_url below

fql_url = (
    'https://graph.facebook.com/fql?q='
    '{"user_sex":"SELECT sex FROM user WHERE uid=me()",'
    '"friends":"SELECT uid, name FROM user WHERE uid IN '
    '(SELECT uid2 FROM friend WHERE uid1 = me()) '
    'AND not (sex in (SELECT sex FROM #user_sex)) '
    ' ORDER BY name"}'
    '&access_token='+access_token
)

and run requests.get(fql_url), the json returned is

{u'error': {
    u'code': 601,
    u'message': u"(#601) Parser error: unexpected '{' at position 0.",
    u'type': u'OAuthException'}
}

However when I hand code the fql_url as this

fql_url = (
    'https://graph.facebook.com/fql?q=%7B%22'
    'user_sex%22:%22SELECT%20sex%20FROM%20user%20WHERE%20uid=me()%22,%22'
    'friends%22:%22SELECT%20uid,%20name%20FROM%20user%20WHERE%20uid%20IN%20'
    '(SELECT%20uid2%20FROM%20friend%20WHERE%20uid1%20=%20me())%20'
    'AND%20not%20(sex%20in%20(select%20sex%20from%20%23user_sex))%20%20'
    'ORDER%20BY%20name%22%7D&'
    'access_token='+access_token
)

everything works (the json has the desired data).

I've compared both the first fql_url and the hand coded fql_url and both should result in the same url being used to get the json. Is the requests urlencode not working or am I doing something wrong here?

Answer 1

The problem is, that # is indeed a valid character in an URL. It denotes the fragment part. As the fragment is always resolved by the useragent it is never sent to the server. You can try this:

>>> import urllib3
>>> urllib3.util.parse_url(fql_url)
Url(scheme='https', auth=None, host='graph.facebook.com', port=None, path='/fql',
    query='q={"user_sex":"SELECT sex FROM user WHERE uid=me()","friends":"SELECT uid, name FROM user WHERE uid IN (SELECT uid2 FROM friend WHERE uid1 = me()) AND not (sex in (SELECT sex FROM ',
    fragment='user_sex))  ORDER BY name"}')

As you can see, the last part of your URL ended up being parsed as the fragment.

Edit:

The most convenient way would probably be to let requests do all the encoding.

import requests
s = requests.Session()
s.params = {'access_token': 'foobarbaz'} # so you don't have to specify it every time
query = ('{"user_sex":"SELECT sex FROM user WHERE uid=me()",'
         '"friends":"SELECT uid, name FROM user WHERE uid IN '
         '(SELECT uid2 FROM friend WHERE uid1 = me()) '
         'AND not (sex in (SELECT sex FROM #user_sex)) '
         ' ORDER BY name"}')
s.get('https://graph.facebook.com/fql', params={'q': query})