Search code examples
pythonscrapyscrapy-shell

python convert chinese characters in url

I have a url like

href="../job/jobarea.asp?C_jobtype=經營管理主管&peoplenumber=151",

this is shown in inspect element. But when opened in new tab it is showing as

../job/jobarea.asp?C_jobtype=%B8g%C0%E7%BA%DE%B2z%A5D%BA%DE&peoplenumber=151

How do I know which type of encoding is used by the browser to convert it. When I try to do scrapy it is showing some other format and it is stopping as 500 internal server error. Could you please explain me??

Solution

  • It's Tradtional Chinese, so try cp950

    #-*-coding:utf8 -*-

import urllib
s = '經營管理主管'.decode('utf-8').encode('cp950')
print urllib.quote(s)

q ='%B8g%C0%E7%BA%DE%B2z%A5D%BA%DE'
print urllib.unquote(q).decode('cp950').encode('utf-8')

    Result

    %B8g%C0%E7%BA%DE%B2z%A5D%BA%DE
經營管理主管