I have a MySQL database with ~27 million rows in one of its tables (analyses). I indexed two of the variables, time and ID, with the statements
ALTER TABLE `analyses` ADD INDEX(`time`);
ALTER TABLE `analyses` ADD INDEX(`ID`);
Each statement took ~4 minutes. I then attempted to delete redundant rows, with the statement
DELETE a1 FROM analyses a1 INNER JOIN analyses a2 WHERE a1.log_id > a2.log_id
AND a1.ID= a2.ID AND a1.time=a2.time;
This statement has been executing for ~2 days without an error.
Well, just for fun, you could try creating a new table with just the records that you want to retain...
First get rid of your existing indexes, and instead create a compound index on (id,time).
Then do this...
CREATE TABLE my_new_table AS
SELECT x.*
FROM analyses x
JOIN
( SELECT id
, time
, MAX(log_id) log_id
FROM analyses
GROUP
BY id
, time
) y
ON y.ID = x.ID
AND y.time = x.time
AND y.log_id = x.log_id;