Search code examples
python-3.xweb-scrapingscreen-scraping

Automate the Construction of Search Terms


I'm doing some scraping, but I'm looking to automate the construction of an extensive list of keywords. One approach that I've devised, which is neither convenient nor inconvenient, would be the following:

def build_search_terms():
    words1 = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']
    words2 = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']
    for word in words2:
        result = words1[0] + word
        words2.pop(0)
        search_for(result)

What I'm trying to do is create a function that spits out aa to az, then ba to bz, then ca to cz, so on and so forth.

Has anybody tackled this issue before?

Are there more efficient ways to do this?


Solution

  • You can get desired output as below:

    def build_search_terms():
        words_list = []
        words = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']
        for i in words:
            for j in words:
                yield i + j
    

    and use it as

    for word in build_search_terms():
        print(word)
    

    or

    def build_search_terms():
        words = ['a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z']
        return (i +j for i in words for j in words)
    

    and use it as

    words = build_search_terms()
    print(next(words))  # 'aa'
    print(next(words))  # 'ab'
    print(next(words))  # 'ac'
    ....