How to bypass AKAMAI bot detection for data scraping using requests_html, Beautiful Soup

Question

I am scraping data from the Rakuten Japanese e-commerce website. I am using requests-html and Beautiful soup. And the problem is when I request from my local pc (127.0.0.1) it's working fine. But when I request from my ec2 server getting Reference #<esi:vars name="AKAMAI_DEBUG_STRING"/> this message and no data or HTML page is found. And another case when I use wget and request from the server the page URL I get a full page. But my script doesn't work.

Here is my code sample:

from bs4 import BeautifulSoup
from requests_html import HTMLSession

def get_search_url(query_keyword):
    base_url = 'https://search.rakuten.co.jp/search/mall/'
    headers = {
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    }
    session = HTMLSession()
    session.headers.update(headers)

    request_url = base_url + query_keyword
    resp = session.get(request_url)
    soup = BeautifulSoup(resp.text, "lxml")
    return soup

def feature_product_details(url):
    output_list = []
    for i in url.find_all("div", attrs={"class": "dui-card searchresultitem"}):
        product_title = i.find("div", attrs={"class": "content title"})
        if product_title is not None:
            product_title = product_title.getText()
        else:
            product_title = ""

        output = {
            'title': product_title,
        }
        output_list.append(output)
    print(output_list)
    return output_list

def main_rakuten_product_search(query):
    query_keyword = query
    page = get_search_url(query_keyword)
    product_lists = feature_product_details(page)
    return product_lists

if __name__ == '__main__':
    queries = '【レビュー特典中】スマートキー 電波遮断ケース 電波遮断ボックス リレーアタック防止用 キーケース '
    main_rakuten_product_search(queries)

Sample output when running local server:

[
    {
        "title": "【レビュー特典中】スマートキー 電波遮断ケース 電波遮断ボックス リレーアタック防止用 キーケース リレーアタック対策 ボックス 箱　電波遮断ケース RFIDブロッキング 高級PUレザー 高級車盗難防止 カーセキュリティ 高級感溢れ レクサス（グレー）",
    }
]

But don't get any response when running it on my server: Just show this message Reference #<esi:vars name="AKAMAI_DEBUG_STRING"/>

If anyone has any idea on how this could be done, I would be grateful to hear.

Answer 1

I've tried your code on an EC2 in ap-northeast-1 (Tokyo) and I'm getting the sample output.

So, here are few things to check:

• make sure your EC2 has the right ports open
• double check the headers (I've modified yours a bit - see code below)
• check your query input; maybe some of them are malformed?
• don't spray the rakuten server with too many requests from one EC2; maybe they're blocking you already

Here's your code after some slight tuning:

from bs4 import BeautifulSoup
from requests_html import HTMLSession


def get_search_url(query_keyword):
    base_url = 'https://search.rakuten.co.jp/search/mall/'
    headers = {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36',
        'referer': 'https://www.rakuten.co.jp/'
    }
    session = HTMLSession()
    session.headers.update(headers)
    return BeautifulSoup(session.get(base_url + query_keyword).content, "lxml")


def feature_product_details(url):
    output_list = []
    for i in url.find_all("div", attrs={"class": "dui-card searchresultitem"}):
        product_title = i.find("div", attrs={"class": "content title"})
        if product_title is not None:
            product_title = product_title.getText()
        else:
            product_title = ""
        output_list.append({'title': product_title})
    return output_list


def main_rakuten_product_search(query):
    return feature_product_details(get_search_url(query))


if __name__ == '__main__':
    queries = '【レビュー特典中】スマートキー 電波遮断ケース 電波遮断ボックス リレーアタック防止用 キーケース '
    print(main_rakuten_product_search(queries))