Search code examples
botsip-addressuser-agentwhoisspoofing

Verify that user agent facebookexternalhit is really Facebook


I'm seeing a lot of hits on my website with the following user agent:

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

I'm trying to verify whether it really is from Facebook or a spoofing bot.

Using these these Facebook directions, I ran the following to get the real IPs:

whois -h whois.radb.net -- '-i origin AS32934' | grep ^route

That got me more than 800 IPs. They look a lot like the IPs I'm seeing, but not exactly.

For example, my website hits include:

2a03:2880:12ff:2::face:b00c
2a03:2880:13ff:f::face:b00c
2a03:2880:22ff:4::face:b00c

The whois list includes a lot that start the same, but no exact matches, for example:

2a03:2880:f310::/48
2a03:2880:f132::/48

Is that... close enough? Is there another way to check?


Solution

  • Yes, that is enough to check. The list entries you quote are routing prefixes, showing ownership of an entire block of IP addresses. For example, 2a03:2880:f310::/48 means every IPv6 IP address beginning 2a03:2880:f310:... (the 48 comes from the fact that the first 48 bits are fixed, the rest are open). The routes list you quote from also includes e.g. 2a03:2880::/32 (which is every single IPv6 address beginning 2a03:2880) and 2a03:2880:1000::/36 (which is every address under 2a03:2880:1...); so the IP addresses you quote are all real Facebook IP addresses. [I am also seeing a lot of hits from this bot, all real FB addresses, multiple hits for the same web page in the same second, thousands of requests for unchanging images, it's quite tedious.]