Search code examples
phpweb-crawlersocial-networkingaffiliate

Stop social networks from crawling affiliate links


I have searched whole web for solution but didnt found what i need . heres my question

I have a unique link which is shared by user on social networks like fb , twitter etc . When a unique ip click is triggered on that link i will award user some points , but as soon as user share link i see 7-8 database entries and this entries are from social network crawler . So when user share fb crawls and generates false database entries . Not just first time it crawls and crawls everytime after a fixed delay and creating problem . My link like below

kodeinfo.com/user/token/network

Please let me know how i can stop this networks from crawling the url


Solution

  • Mabye check if Facebook or twitter crawler use a specific User-Agent and ignore those agents.