What is the best way to ban DDOS attacks for Apache, on a shared hosting platform?

Question

We have Ubuntu 14.04 Apache servers that host thousands of websites in a shared hosting environment, and occasionally one or another websites get the attention of DDOS attackers.

Question:

What is the best apache module or method for detecting and banning IP addresses that are too active?

Requirements:

1. Do not ban Googlebot and other legit search engines.
2. If Googlebot is indexing multiple client websites (at different domains) we don't want to count the aggregate hits from that IP (range) we need to count on a per domain basis (of course with a maximum overall hit count as well).

Answer 1

Fail2ban can help with that, you will need to configure it to fit your Requirements

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs -- too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).