Search code examples
python-3.xpython-requestspython-multiprocessingconcurrent.futures

Python requests returns me status 200, but does not perform the expected action


I'm trying to create an application that can like posts with multiple profiles at the same time. I'm trying to parallelize the logins and for each user logged in to like the posts in parallel as follows.

with ProcessPoolExecutor() as exe:
    bot = Insta()
    results = []
    for credential in credentials: # go through credentials, login in parallel.
        results.append(
            exe.submit(bot.login, credential) # Each login takes 15 sec
        ) # Add status of each process in a list
    for result in as_completed(results): # When complete, I call like
        if result.result() == 200:
            with Pool(4) as p:
                resp = p.map(bot.like, urls)
                print(resp)            

Despite returning status_code 200 to me, when I look at the post, it is not liked.

When I try to do this individually for each login, it returns the same thing to me, but this time, the post is really liked. That is:

bot = Insta()
resp = bot.login(credential)
if resp == 200:
    with Pool(5) as p:
        p.map(bot.like, urls)

Can anyone tell me what the problem is? I would like to know if I'm doing something wrong. My like method looks like this right now:

    def like(self, url_post):
        self._set_id_post(url_post)  # id of post
        resp = self.session.get(url_post)
        self.session.headers = {'user-agent': self.user_agent}
        self.session.headers.update({'Referer': url_post})
        self.session.headers.update({'X-CSRFToken': resp.cookies['csrftoken']}, )

        url = endpoints['like_url'] % self.post_id
        time.sleep(random.gauss(6, 1.5))
        response = self.session.post(url)
        self.session.headers.update({'X-CSRFToken': resp.cookies['csrftoken']})
        if response.status_code == 200:
            return response.status_code
        elif response.status_code == 403:
            return response.status_code

        elif response.status_code == 400:
            return response.status_code

Solution

  • I solved this problem by setting a proxy for each user. I had to buy it, because the public didn't work in my case. But for those who are having similar problems with Web Scraping, other than for social networks, I will insert here one of my codes for return proxy elite of free sites that maybe can help.

    def free_proxy_list():
        options = Options()
        options.headless = True
        driver = webdriver.Chrome(options=options)
        driver.get('https://free-proxy-list.net/')
        # Show only Elite Proxies
        driver.find_element_by_xpath('//*[@id="proxylisttable"]/tfoot/tr/th[5]/select/option[3]').click()
        # Show only SSl
        driver.find_element_by_xpath('//*[@id="proxylisttable"]/tfoot/tr/th[7]/select/option[3]').click()
        proxies = []
        # Paginate 1
        for i in range(1, 21):
            xpath_ip = driver.find_element_by_xpath('//*[@id="proxylisttable"]/tbody/tr[%s]/td[1]' % i).text
            xpath_port = driver.find_element_by_xpath('//*[@id="proxylisttable"]/tbody/tr[%s]/td[2]' % i).text
            proxy = xpath_ip + ":" + xpath_port
            proxies.append(proxy)
        # Paginate 2
        driver.find_element_by_xpath('//*[@id="proxylisttable_paginate"]/ul/li[4]/a').click()
        try:
            for i in range(1, 21):
                xpath_ip = driver.find_element_by_xpath('//*[@id="proxylisttable"]/tbody/tr[%s]/td[1]' % i).text
                xpath_port = driver.find_element_by_xpath('//*[@id="proxylisttable"]/tbody/tr[%s]/td[2]' % i).text
                proxy = xpath_ip + ":" + xpath_port
                proxies.append(proxy)
        except NoSuchElementException:
            return proxies
    
        return proxies