run PHP script from cli : php.ini memory_size not taken into account

Question

I am trying to run a php script (magento reindexer script) from the command line. The script consumes a lot of memory so I am getting the following error: PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /home/karanta/www/karanta.fr/lib/Zend/Db/Adapter/Abstract.php on line 691

To fix this issue, I edited the /etc/php5/cli/php.ini file and set memory_limit = 2048M.

To check the configuration, I run a script containing phpinfo(); from the cli and I see: memory_limit => 2048M => 2048M, so it seems the configuration is taken into account properly. ini_get('memory_limit); also returns 2048M.

However when I rerun the reindex script, I still get PHP Fatal error: Allowed memory size of 536870912 bytes exhausted as if the memory_limit was still 512M. The script is unable to complete, I am out of ideas how to augment the memory_limit to allow the script to complete.

Edit: I also tried adding the instruction ini_set("memory_limit", -1); directly into the PHP script and it still hangs with the same PHP Fatal error: Allowed memory size of 536870912 bytes exhausted.

Extra information:

Server is a dedicated machine at ovh running Debian GNU/Linux 7.5 with 64GB RAM!

php -v returns:

PHP 5.6.12-1 (cli) Copyright (c) 1997-2015 The PHP Group Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

free -k -h during script execution returns:

               total        used        free      shared  buff/cache     available
Mem:            62G        1,8G         48G         84M          12G         60G
Swap:          1,0G          0B        1,0G

ps -aux returns:

karanta  25568  5.6  0.0 229484 41824 pts/1    S    07:54   0:04 php shell/indexer.php --reindex catalog_url

---

Following @IgorGreg's recommandation, I tried setting the memory limit using Zend_Memory. I wrote this php script that I run from the cli in replacement of the shell/indexer.php script.

<?php
require_once 'app/Mage.php';
$app = Mage::app('admin');
umask(0);
$memoryManager = Zend_Memory::factory('none');
$memoryManager->setMemoryLimit(-1);
$process = Mage::getModel('index/process')->load(3);
$process->reindexAll();
?>

Still getting the same error.

Answer 1

Magento in CLI mode parses the .htaccess file and if it finds a memory limit, or any other PHP settings there, applies them (yes, seriously....)

The responsible code is in shell/abstract.php:

/**
 * Parse .htaccess file and apply php settings to shell script
 *
 */
protected function _applyPhpVariables()
{
    $htaccess = $this->_getRootPath() . '.htaccess';
    if (file_exists($htaccess)) {
        // parse htaccess file
        $data = file_get_contents($htaccess);
        $matches = array();
        preg_match_all('#^\s+?php_value\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
        preg_match_all('#^\s+?php_flag\s+([a-z_]+)\s+(.+)$#siUm', $data, $matches, PREG_SET_ORDER);
        if ($matches) {
            foreach ($matches as $match) {
                @ini_set($match[1], str_replace("\r", '', $match[2]));
            }
        }
    }
}

Solution

Do not set memory limit and execution time via .htaccess, use the php.ini or the virtual host configuration.