Magento Product Price re-index hangs on DELETE query

The price reindexing for my Magento Enterprise 1.13 installation is not finishing any more when I run it from SSH. All other indexes seem fine. The longest I have left it to run is 4 days.

Server Specs:

6 cores
15GB RAM
200GB HDD
CentOS 6

I can see from using "SHOW PROCESSLIST;" that it hangs with a status of "Sending Data" on this query:

 DELETE `index_price`  FROM `catalog_product_index_price` AS `index_price` LEFT JOIN `catalog_product_index_price_idx` AS `ip_tmp` ON index_price.entity_id = ip_tmp.entity_id AND index_price.website_id = ip_tmp.website_id WHERE (ip_tmp.entity_id IS NULL)

I have checked both table's properties (while this query is running):

catalog_product_index_price = 16,437,361 rows, type INNODB and size 2.4GB
catalog_product_index_price_idx = 16,314,589 rows, type INNODB and size 2.1GB

Here are the table structures and indexes:

CREATE TABLE IF NOT EXISTS `catalog_product_index_price` (
  `entity_id` int(10) unsigned NOT NULL COMMENT 'Entity ID',
  `customer_group_id` smallint(5) unsigned NOT NULL COMMENT 'Customer Group ID',
  `website_id` smallint(5) unsigned NOT NULL COMMENT 'Website ID',
  `tax_class_id` smallint(5) unsigned DEFAULT '0' COMMENT 'Tax Class ID',
  `price` decimal(12,4) DEFAULT NULL COMMENT 'Price',
  `final_price` decimal(12,4) DEFAULT NULL COMMENT 'Final Price',
  `min_price` decimal(12,4) DEFAULT NULL COMMENT 'Min Price',
  `max_price` decimal(12,4) DEFAULT NULL COMMENT 'Max Price',
  `tier_price` decimal(12,4) DEFAULT NULL COMMENT 'Tier Price',
  `group_price` decimal(12,4) DEFAULT NULL COMMENT 'Group price'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog Product Price Index Table';

ALTER TABLE `catalog_product_index_price`
 ADD PRIMARY KEY (`entity_id`,`customer_group_id`,`website_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_CUSTOMER_GROUP_ID` (`customer_group_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_WEBSITE_ID` (`website_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_MIN_PRICE` (`min_price`), ADD KEY `IDX_CAT_PRD_IDX_PRICE_WS_ID_CSTR_GROUP_ID_MIN_PRICE` (`website_id`,`customer_group_id`,`min_price`);

CREATE TABLE IF NOT EXISTS `catalog_product_index_price_idx` (
  `entity_id` int(10) unsigned NOT NULL COMMENT 'Entity ID',
  `customer_group_id` smallint(5) unsigned NOT NULL COMMENT 'Customer Group ID',
  `website_id` smallint(5) unsigned NOT NULL COMMENT 'Website ID',
  `tax_class_id` smallint(5) unsigned DEFAULT '0' COMMENT 'Tax Class ID',
  `price` decimal(12,4) DEFAULT NULL COMMENT 'Price',
  `final_price` decimal(12,4) DEFAULT NULL COMMENT 'Final Price',
  `min_price` decimal(12,4) DEFAULT NULL COMMENT 'Min Price',
  `max_price` decimal(12,4) DEFAULT NULL COMMENT 'Max Price',
  `tier_price` decimal(12,4) DEFAULT NULL COMMENT 'Tier Price',
  `group_price` decimal(12,4) DEFAULT NULL COMMENT 'Group price'
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog Product Price Indexer Index Table';

ALTER TABLE `catalog_product_index_price_idx`
 ADD PRIMARY KEY (`entity_id`,`customer_group_id`,`website_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_IDX_CUSTOMER_GROUP_ID` (`customer_group_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_IDX_WEBSITE_ID` (`website_id`), ADD KEY `IDX_CATALOG_PRODUCT_INDEX_PRICE_IDX_MIN_PRICE` (`min_price`);

I have tried running the mysql command "OPTIMIZE" on both tables and run checks for any foreign key issues both didnt help.

I have also tried changing my.cnf as it has been suggested maybe the "innodb_lock_wait_timeout" limit has been reached. See my my.cnf below:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
symbolic-links=0

thread_concurrency=12
thread_cache_size=64
#wait_timeout=600
#wait_timeout=60
#table_cache=2048
table_cache=1024
query_cache_type=1
query_cache_size=1024M
#query_cache_limit=16M
query_cache_limit=32M
key_buffer_size=256M
max_allowed_packet=2048M
#max_connections=1000
max_connections=600
tmp_table_size=1024M
max_heap_table_size=1024M
table_definition_cache=4000
table_open_cache=4000
#sort_buffer_size=1M
#read_buffer_size=1M
sort_buffer_size=8M
read_buffer_size=4M
join_buffer_size=16M
#below files to help with crons
#open_files_limit=131070
wait_timeout=9000
connect_timeout=9000

innodb_thread_concurrency=12
innodb_file_per_table=1
#innodb_buffer_pool_size=4G
#innodb_buffer_pool_size=3G
innodb_buffer_pool_size=7G
innodb_lock_wait_timeout=9000
innodb_flush_log_at_trx_commit=1
innodb_additional_mem_pool_size=24M
#innodb_fast_shutdown=0
innodb_log_buffer_size=8M
innodb_log_file_size=128M

#log_error=/var/log/mysql/mysql-error.log
#log_queries_not_using_indexes=1
#slow_query_log_file=/var/log/mysql/mysql-slow.log
#log_slow_queries=ON

Solution

I fixed this by modifying the product re-indexing to delete product price entries in the "catalog_product_index_price" table using the full primary key for the table. To do this you need to override this file: /app/code/core/Enterprise/Catalog/Model/Index/Action/Product/Price/Abstract.php and change

$select = $this->_connection->select()
        ->from(array('index_price' => $this->_getTable('catalog/product_index_price')), null)
        ->joinLeft(
            array('ip_tmp' => $this->_getIdxTable()),
            'index_price.entity_id = ip_tmp.entity_id AND index_price.website_id = ip_tmp.website_id',
            array()
        )
        ->where('ip_tmp.entity_id IS NULL');

to...

$select = $this->_connection->select()
        ->from(array('index_price' => $this->_getTable('catalog/product_index_price')), null)
        ->joinLeft(
            array('ip_tmp' => $this->_getIdxTable()),
            'index_price.entity_id = ip_tmp.entity_id AND index_price.website_id = ip_tmp.website_id AND index_price.customer_group_id = ip_tmp.customer_group_id',
            array()
        )
        ->where('ip_tmp.entity_id IS NULL');

Do this at your own peril, so far everything seems ok. The only downside to this that I can see is that some redundant rows may be left in your "catalog_product_index_price" table. Other than that I havent had any issues.