Search code examples
pythonweb-scrapingbeautifulsoup

Doesnt price compare after add merchant column

My scraper bot working normally till I add merchant column in database. Scraper.py file scraping success merchant and recorded in database correctly but compare bot not compare after add merchant column

enter image description here

My original code:(compare success)

# The max_date a product can have in order to be considered "old"
        limit = datetime.datetime.strptime(products[-1][1], "%Y-%m-%d %H:%M:%S.%f") + \
            datetime.timedelta(minutes=OLDER_PRODUCTS_RANGE)

        # Separate "old" from "new" products
        for i, product in enumerate(products[::-1]):
            date = datetime.datetime.strptime(
                product[1], "%Y-%m-%d %H:%M:%S.%f")

            index = len(products) - i - 1
            if date > limit:
                old_products = products[index+1:]
                new_products = products[:index+1]
                break

        # If we have only one or even none of the lists, return
        if len(new_products) == 0 or len(old_products) == 0:
            c.close()
            self.save_db(db)

            return True

        older_product = min(old_products, key=lambda x: x[5])
        current_product = min(new_products, key=lambda x: x[5])

        first_price = older_product[5]
        current_price = current_product[5]
        current_date = current_product[1]

        url = current_product[6]

        price_difference = first_price - current_price
        percentage = price_difference / first_price
        percentage_str = str("%.2f" % (percentage * 100))

        # If the drop in the price was greater then the expected percentage, warn the user
        if percentage >= self.percentage:
            rowid = current_product[0]
            product_name = current_product[4]
            product_id = current_product[3]

            print(
                f"[Bot] [{current_date}] Price of \"{product_name}\" is {percentage_str}% off")

            message = product_name + "\n\n" + \
                str(first_price) + " TL >>>> " + \
                str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
                MAIN_URL + url + "\n\n" + \
                MAIN_URL + "ara?q=" + product_id

            context.bot.send_message(
                chat_id=CHANNEL_ID,
                text=message
            )

            c.execute(
                "INSERT INTO deleted SELECT rowid FROM products WHERE product_id = %s AND rowid != %s;",
                (product_id, rowid)
            )

        c.close()
        self.save_db(db)

        return True

My edited Code: (doesnt price compare)

# The max_date a product can have in order to be considered "old"
        limit = datetime.datetime.strptime(products[-1][1], "%Y-%m-%d %H:%M:%S.%f") + \
            datetime.timedelta(minutes=OLDER_PRODUCTS_RANGE)

        # Separate "old" from "new" products
        for i, product in enumerate(products[::-1]):
            date = datetime.datetime.strptime(
                product[1], "%Y-%m-%d %H:%M:%S.%f")

            index = len(products) - i - 1
            if date > limit:
                old_products = products[index+1:]
                new_products = products[:index+1]
                break

        # If we have only one or even none of the lists, return
        if len(new_products) == 0 or len(old_products) == 0:
            c.close()
            self.save_db(db)

            return True

        older_product = min(old_products, key=lambda x: x[5])
        current_product = min(new_products, key=lambda x: x[5])

        first_price = older_product[5]
        current_price = current_product[5]
        current_date = current_product[1]
        
        url = current_product[6]

        merchant = current_product[7]

        price_difference = first_price - current_price
        percentage = price_difference / first_price
        percentage_str = str("%.2f" % (percentage * 100))

        # If the drop in the price was greater then the expected percentage, warn the user
        if percentage >= self.percentage:
            rowid = current_product[0]
            product_name = current_product[4]
            product_id = current_product[3]
            

            print(
                f"[Bot] [{current_date}] Price of \"{product_name}\" is {percentage_str}% off")

            message = product_name + "\n\n" + \
                + "Satici Adi:" + merchant + "\n\n" + \
                str(first_price) + " TL >>>> " + \
                str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
                MAIN_URL + url + "?magaza=" + merchant + "\n\n" + \
                MAIN_URL + "ara?q=" + product_id

            context.bot.send_message(
                chat_id=CHANNEL_ID,
                text=message
            )

            c.execute(
                "INSERT INTO deleted SELECT rowid FROM products WHERE product_id = %s AND rowid != %s;",
                (product_id, rowid)
            )

        c.close()
        self.save_db(db)

        return True

I use python with beautifulsoup4

Whats wrong?

Solution

  • I suppose the problem is you have extra + symbol here:

    message = product_name + "\n\n" + \
              + "Satici Adi:" + merchant + "\n\n" + \
              str(first_price) + " TL >>>> " + \
              str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
              MAIN_URL + url + "?magaza=" + merchant + "\n\n" + \
              MAIN_URL + "ara?q=" + product_id

    You should replace it by:

    message = product_name + "\n\n" + \
              "Satici Adi:" + merchant + "\n\n" + \
              str(first_price) + " TL >>>> " + \
              str(current_price) + f" TL - {percentage_str}%" + "\n\n" + \
              MAIN_URL + url + "?magaza=" + merchant + "\n\n" + \
              MAIN_URL + "ara?q=" + product_id