Search code examples
system-administrationweb-development-serverblacklistawstatswebsite-admin

How do I safely blacklist these IPs I found in AWStats?


My site has performance issues, partly caused by bots. I checked my AWStats and found some IP addresses eating up tons of bandwidth. How can I tell whether it's safe to blacklist these? A few are eating up 2 GB per month or more, consistently.

See this screenshot: https://ibb.co/hJ6hmR

I tried IP lookup on these IPs, but how can I tell whether it's something I should keep, like known crawlers?

I also checked the Crawlers in AWStats, but I don't see an IP address so I can't tell what it came from. One unknown robot ate up 11 GB, whereas Googlebot only uses 1 GB. https://ibb.co/cWDCmR

How do I single out bad crawlers?


Solution

  • You can try banning the robots in your .htaccess if you are using Apache

    BrowserMatchNoCase thisisthebotname bad_bot
    Order Deny,Allow
    Deny from env=bad_bot
    

    But you need the bot's user-agent