Search code examples
pythonparsingurlscreen-scraping

Parsing a url in Python

I want to scrape some info about couple of companies from a website, and below you can see two companies url's of the same info requsted :

url for Company "A1" :

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMIoDLVIGrTQ0C6mAxqiFAejr2_dXNSYG--OHEWCv08nf5uW4C80kFsdFCqzlxmDSIoEcqt3Rc_WDHUevX3z0ByQWWO6qhEpgL9u0Rr-ujXTbZ1Csh-3HgOpzTKPZMnlH8uSU0e2jJH9Gl3nfnio4_kJwpcKOw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

url for Company "A2":

url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDoIwGAYf6f8oVMKomNQiadBCQ7uYDmiIUhyMvr51c1Fjbrw7chQJ_j6e_G2cg79QT25xUCjNXmYMIikSyK3eFR1bMlR59PbNQ3NAZrVpy1qkAP66RWP469ZMN1UKyX7cegikNvM0kCWXfyxXnFqy6xjZr9F16vqHOp6fJIF9Sw!!/dl2/d1/L3dJMjIyQSEhL3dQRUJGUUJnTlFCaGpRQmhyUUJoSFFCZ1BRQmdBQSEhL1lJNXcvN19OMENWUkk0MjBHMTkxMElLU1E5VTJBMjBCNQ!!/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'

As you can see the url keeps changing before this part:

/?CompanyName=A2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/

and after this part:

http://www.example.com/wps/portal/!ut/p/c1/

and what i want is to ignore any changing part in the url and only pass the company names below:

Companies = ['A1', 'A2', 'A3']

as exapmle , of course it will be with a for loop:

url = 'http://www.example.com/wps/portal/!ut/p/c1/04_SB8K8xLLM9MSSzPy8xBz9CP0os3g_A-ewIE8TIwN3Q0tDA0_v4EDLUCNHIwMvc6B8JJK8QbCpgYGniU9YiLOPu7GBgQFJut0DwkxBuoONggO8jA08jQjo9vPIz03Vj9SPMsepyslUP0Q_0hWoKBKvooLc0IhyQ91AAHb2Eas!/dl2/d1/L0lDUmlTUSEhL3dHa0FKRnNBL1lCUlp3QSEhL2Vu/?typeofsearch=1&company=&CompanyName=' + companies +'A1&tabOrder=1&x=0&y=0'

How can i do it ?

Solution

  • You can do it without using regular expressions,

    url = 'http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=A1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/'
url = url.replace('Name=A1', 'Name=')
compaines = ['a1','a2','a3']
for ele in compaines:
    start = url.find("CompanyName=") + 12
    newUrl1 = url[:start] + ele
    newUrl2 = newUrl1 + url[start:]
    print newUrl2

    Note: I have simplified the URL for testing purpose.

    If you run the above program, you will get the below output,

        http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a1&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a2&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/
http://www.example.com/wps/portal/!ut/p/c1/lYuxDJBMjBCNQ!!/?CompanyName=a3&tabOrder=7&chart_type=chart_oneDay&announcmentNumber=&isAnnual=/