dealsgoround.com and citypockets.com access user accounts and their daily deals from livingsocial.com and other daily deal websites. These websites ask users to give their credentials to sign them in to respective websites and then import account details. LivingSocial and other deal website don't provide API access to user accounts.
I also want such access to other websites and import data from there. But I am unable to find out the backend process by which dealsgoround.com and citypockets.com import data from Groupon and LivingSocial etc.
By now I am able to find out that web crawlers/spiders can be used to scrap data from web pages. But I am not sure web crawlers can be useful in a scenario where we need user sign in and the page urls are encrypted or at least are dynamically generated urls.
Please help me and suggest me a way to do this. If crawlers are the solution to this please provide links to some web crawler APIs that I can use in my .net application.
Thanks
Atif
I was able to find the answer... Yes web crawler is the solution in this scenario.
We can use PHP, ASP.net or any other server side scripting language to send an httppostrequest with post parameters (user_name/password). This will authenticate (sign in) the user. After sign-in we can read the contents of any url with user information who has been signed-in.
Note: In my case this isn't unauthorized access to user accounts as users provide their credentials themselves to import their data from deal websites as in dealsgoround.com and citypocket.com