Search code examples
javablackberryweb-scraping

Java Web Scraping for blackberry apps


I am trying to find a way to collect information of all application from Blackberry App World. The Process I followed for Apple was simple because it had api and also for android/windows I got the ids of app from app listing page and than went to that page and collected information from html elements using JSoup library. For Blackberry, they load new application list using jquery web service call. I tried to debug the page using Firefox Firebug Plugin to check what events are fired but couldn't figure out how to programmatically fire button click events in Java and capture the returned data. Can anyone guide me how I can proceed ?


Solution

  • I looked into more detail in the source page and found that they have a catalog API javascript which in turn call a URL which gives reply in JSON format similar to Apple Search API. The url is: "_http://appworld.blackberry.com/cas/content/*app_id*"

    But to get the list of IDs I used Selenium in Java which opened the web page and clicked the Load More Button until approx number of times and than saved the html page and extracted ids of application from it. Than used those ids with above url to get the required data. I could have used the same api for the list of ids too. But I already did this step before I found the search URL :)

    Hope this helps someone