Search code examples
mysqlapachedebianxml-rpc

Attack on MySQL - and how to prevent it?

I have been experiencing an attack on my server, and have blocked xmlrpc access by including the following in the apache conf file: 

<files xmlrpc.php>
    order allow,deny
    deny from all
</files>

Now, instead of crushing mysql service every couple hours, it crushes once/twice a day. Which is still a problem. fail2ban bans 2 failed ssh attempts for an hour. However, I do see about 100+ of the following entries in the daemon.log:

mysqld[18852]: 2016-10-13 3:06:40 139773247216384 [Warning] Access denied for user 'root'@'139.196.28.237' (using password: YES)

All those 100+ attempts happen within a minute, and about an hour later I see several messages like:

mysqld[18852]: 2016-10-13 3:32:52 139773325777664 [Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT. The statement is unsafe because it uses a LIMIT clause. This is unsafe because the set of rows included cannot be predicted. Statement: DELETE FROM wp_generic_options WHERE option_name LIKE 'jetpack\_nonce\_%' AND CAST( option_value AS UNSIGNED ) < 1476340372 ORDER BY option_id LIMIT 100

Some time later, it seems that the mysql service gets restarts, but then fails, and the following lines are found in the error log:

InnoDB: Initializing buffer pool, size = 256.0M
InnoDB: mmap(281542656 bytes) failed; errno 12
InnoDB: Cannot allocate memory for the buffer pool
Plugin 'InnoDB' init function returned error.
Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
Unknown/unsupported storage engine: InnoDB
Aborting

Can someone please explain to me what is happening? And how to stop and prevent it from happening again?

Solution

  • The geolocation for the IP address 139.196.28.237 reports that it originates from Hangzhou, China. Is that where you expect MySQL clients to be originating from when they connect to your MySQL instance? If not, you may have an external attacker. You probably should not allow external clients to reach your MySQL server at all. You need a firewall to block MySQL's port from external requests.

    The warning message about the unsafe statement in your binlog may be unrelated. I recall "jetpack" is a popular WordPress plugin, and the SQL query may be a normal part of the code for jetpack. You can fix these warnings by changing your MySQL binlog format to MIXED or ROW. To understand more about unsafe statement, read https://dev.mysql.com/doc/refman/5.7/en/replication-rbr-safe-unsafe.html

    The errors about InnoDB failing to allocated 256MB of memory for its buffer pool indicates that processes on your server are using more memory than the server has. InnoDB won't allocate memory unless physical memory is available (it won't allocate memory that is already swapping). This may also be unrelated to the attempted logins. You need to increase the memory on your server. If you're using AWS, you need to increase the instance size.