Search code examples
pythonlistloopsscreen-scraping

Python: Use of for loops to iterate through a list of categories and sub-categories. TypeError: must be str, not list


I have a sample list of car makes and models to iterate through and carry out code on.

...
makes = ['Ford', 'Audi']
ford_models = ['C-MAX', 'Focus']
audi_models = ['A3', 'A4']
models = [ford_models, audi_models]

base_url = "https://www.donedeal.ie/cars/"

Program code:

#Go through a make in the list along with all its models 
#and then move onto the second make and all its models and so on..

for make in makes: 
    for model in models:
       for response in range(0,58,29):
            headers = {
                'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
            response = requests.get(base_url + make + "/" + model + "?start=" + str(response), headers=headers)
            soup = BeautifulSoup(response.text, 'html.parser')

            ...

I want the program to first go to Ford as the make and firstly run all the code for C-MAX, then go back into the loop and run all the code for Focus. After it finishes with Focus, I want it to change the make to Audi, and go through the Audi models (first A3 and then A4). I made an attempt but getting an error:

response = requests.get(base_url + make + "/" + model + "?start=" + str(response), headers=headers)
TypeError: must be str, not list

How can I fix the error and achieve this using for loops?


Solution

  • The immediate problem is that model is a list of models, not the name of an individual model. You neglected to do any basic tracing of your operation.

    for make in makes: 
        for model in models:
            print (make, model)
    

    will show you the logic problem.

    You need to key your models to the make. Use a dict:

    make_model {
        'Ford': ['C-MAX', 'Focus'],
        'Audi':  ['A3', 'A4']
    }
    
    for make in make_model:
        for model in make_model[make]:
    

    This will iterate your make/model pairings as you want. I expect you can finish from here.