Search code examples
pythonweb-scrapingscrapywebscarab

Unable to scrape data looks encrypted

I am a beginner web scraping I am trying to get the phone number from this page. https://www.quickerala.com/listings?q=Healthcare&location=Alleppey

Under this element tag

<span data-qk-el-trackcontact="1" data-trackdata="{&quot;business&quot;:&quot;427151&quot;,&quot;address&quot;:&quot;430820&quot;,&quot;number&quot;:&quot;653252%6252%3252%553252%6252%3252%653252%6252%3252%843252%6252%3252%653252%6252%3252%553252%6252%3252%753252%6252%3252%843252%6252%3252%653252%6252%3252%753252%6252%3252%&quot;,&quot;type&quot;:&quot;mobile&quot;,&quot;page&quot;:&quot;businessListings&quot;}" data-qk-el-unobfuscate="1" data-unobfuscate-text="653252%6252%3252%553252%6252%3252%653252%6252%3252%843252%6252%3252%653252%6252%3252%553252%6252%3252%753252%6252%3252%843252%6252%3252%653252%6252%3252%753252%6252%3252%">9809780878</span>

I can view the phone number "9809780878" but I am not sure how to get it.

I have tried

response.xpath('//div[@class="listContacts brtop-10"]//data-trackdata') 

response.xpath('//span[@data-qk-el-trackcontact=1]/data-trackdata') 

response.xpath('//span[@data-qk-el-trackcontact=1]').extract()

without any luck.

Solution

  • Mobile number is encrypted in the response in a specific area.

    Though numbers are avialable in other area too. But in this solution I have focused on copy there(website) decryption JS function and execute in our python script

    # Library Import
from bs4 import BeautifulSoup as Soup
import requests as rq
import js2py
import json

    Find & copy their JS decryption function & excute using js4py

    # JS function to decode mobile number
script = '''
function(e) {
    var t = e.split("").reverse().join(""),
        n = "",
        o = decodeURIComponent(decodeURIComponent(t)).split("#&#");
    for (i = 0; i < o.length; i++) "" != o[i] && (n += String.fromCharCode(o[i]));
    return n
}
'''
number_fun = js2py.eval_js(script)

    Rest of the code is what as usual

    # Process Data
res = rq.get(
    "https://www.quickerala.com/listings?q=Healthcare&location=Alleppey")
res_data = Soup(res.text, features="html.parser")
rows = res_data.findAll("button", {"data-el": "view-location"})
for row in rows:
    try:
        json_data = json.loads(row["data-map"])
        name = json_data["name"]
        location = json_data["location"]
        phone = number_fun(json_data["phone"][0])
        print(name, location, phone)
    except:
        pass

    Output

    Nediyathu Speech & Hearing Aid Centre Thrissur Mavelikara, Alleppey 9446477258
Mypharma Laboratories Alleppey, Alleppey 9809780878
Air Rescuers World Wide Pvt Ltd Chengannur, Alleppey 9870001118
Abhaya Ayurveda Hospital Ennakkad, Alleppey 9539297062
SOUPARNIKA AYURVEDA CLINIC Mavelikara, Alleppey 8075803773
Thottikuzhiyil Medicals Alappuzha, Alleppey 9447470538
Aawaaz Speech & Hearing Care Centre Cherthala, Alleppey 9995822386
DANA GYM Alleppey, Alleppey 9567448535
Upasana Yoga and Reiki Clinic Ambalapuzha, Alleppey 9947260352
SIDHA DEEPAM Cherthala, Alleppey 8714233349
SANTHISUKHAM Alleppey, Alleppey 8111928007
SAMANGA AYURVEDA & PHYSIOTHERAPY REHAB. Alleppey, Alleppey 90618 60702
KRV AYURVEDA KENDRAM Alleppey, Alleppey 9447145738
School of Life Skills Alappuzha, Alleppey 9895458500
Kripa Wellness Clinic & Diabetic Research Center Kayamkulam, Alleppey 0479 2446789
Dr. Oommens Eye Hospital & Microsurgery Center Chengannur, Alleppey 0479-2453416
Santhigiri Ayurveda & Siddha Vaidyasala, Chengannur, Alleppey 479-2452582
Santhigiri Ayurveda & Siddha Hospital Alleppey, Alleppey 0478-2879734
Santhigiri Ayurveda & Siddha Hospital Thiruvambady, Alleppey 0477-3200724
Tricare Diagnostics Chengannur, Alleppey 0479-2456664
Dhathri Ayurveda Hospital & Panchakarma Center Kayamkulam, Alleppey 0479-2431403
Pranala Diagnostics Haripad, Alleppey 9495603511
Sankar's Healthcare Diagnostics Pathirappally, Alleppey 9961234488