Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 2
Product Model Save Memory Leak
 
chris12295
Jr. Member
 
Total Posts:  7
Joined:  2010-02-26
 

I am looping through approx. 10,000 xml elements and either creating or updating a product in the magento database.  The problem is, each iteration through the loop uses more and more memory especially after loading a product and saving a product.  This memory is never freed.  Also, it is very slow, so I am wondering if there is a better way to do this.

//in class Import...
$i=0;
$total count($xml->Item);
$this->product Mage::getModel('catalog/product');

foreach(
$xml->Item as $xmlItem)  {

                $log 
"\n";
        
$log .= "Base memory " memory_get_usage() . " ...\n";
        
$log .= "Entering element " $i " of " $total "...\n";
              
        
$sku $xmlItem->item_id;
        
        if( empty(
$sku) ) {
            $log 
.= "Error processing element " $i "\n";
            
Mage::log($lognull"Import_" strval(floor($i/500)) . ".log");
            return;
        
}
        
        $log 
.= "Searching database for " $sku "...\n";

                
$this->product->reset();
        
$this->product Mage::getModel('catalog/product');
        
        
$productId $this->product->getIdBySku($sku);
        
        if (
$productId{
            $log 
.= "Product " $sku " exists, updating...\n";
            
$this->product->load($productId);
        
}
        
        $name 
$xmlItem->description;
        
$desc strip_tags($xmlItem->description_long'<br>');
        
$sdesc $name;
        
$price $xmlItem->price;
        
$weight $xmlItem->weight;
        
$baseImg $xmlItem->image_large;
        
$thumb $xmlItem->image_small;
        
$brand $xmlItem->brand_description;
        
$sizenote $xmlItem->sizenote;
        
$shipping $xmlItem->shipping;
        
$unit $xmlItem->unit;
        
$package $xmlItem->package;
        
$per $xmlItem->per;
        
$otherTxt $xmlItem->rebatetext;
        
        
$this->product->setSku($sku)
        ->
setName($name)
        ->
setDescription($desc)
        ->
setShortDescription($sdesc)
        ->
setPrice($price)
        ->
setData('image_large'$baseImg)
        ->
setData('image_small'$thumb)
        ->
setData('brand'$brand)
        ->
setData('sizenote'$sizenote)
        ->
setData('shipping'$shipping)
        ->
setData('unit'$unit)
        ->
setData('package'$package)
        ->
setData('per'$per)
        ->
setData('other_txt'$otherTxt)
        ->
setTypeId('simple')
        ->
setAttributeSetId(4)
        ->
setCategoryIds("0")
        ->
setWeight($weight)
        ->
setTaxClassId(2// taxable goods
        
->setVisibility(4// catalog, search
        
->setStatus(1// enabled
         
        // assign product to the default website
        
->setStoreId('default');
        
        
$stockData $this->product->getStockData();
        
$stockData['qty'1;
        
$stockData['is_in_stock'1;
        
$stockData['manage_stock'0;
        
$stockData['use_config_manage_stock'0;
        
        if(!
$productId$log .= "Adding product " $sku "...\n";
        
$this->product->setStockData($stockData)->save();
        
        
Mage::log($lognull"Import_" strval(floor($i/500)) . ".log");

//end foreach
 
Magento Community Magento Community
Magento Community
Magento Community
 
tzyganu
Mentor
 
Avatar
Total Posts:  2205
Joined:  2009-11-18
Bucharest, Romania
 

try unseting $this->product after each iteration.

unset($this->product);
 
Magento Community Magento Community
Magento Community
Magento Community
 
chris12295
Jr. Member
 
Total Posts:  7
Joined:  2010-02-26
 

I have tried that with no luck.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cooldar
Sr. Member
 
Total Posts:  95
Joined:  2008-05-14
 

I need a solution for this as well,
my scripts are all hit by memory exhausted error

DO SOMETHING MAGENTO!!!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
babybidou
Jr. Member
 
Avatar
Total Posts:  15
Joined:  2010-04-26
 

Bump !

It seems to be a big problem…

 
Magento Community Magento Community
Magento Community
Magento Community
 
coldlamper
Member
 
Total Posts:  35
Joined:  2009-02-01
NC, USA
 

I have the same problem. I upgraded to php 5.3.3 so i could use gc_collect_cycles to force garbage collection.  It does work and collects a lot of real leaks and saves memory but Magento must be increasing the size of variables on load and save.  These don’t show up as garbage and don’t get cleared out with gc_collect_cycles.  I really wish the Magento team would look into this.

 
Magento Community Magento Community
Magento Community
Magento Community
 
coldlamper
Member
 
Total Posts:  35
Joined:  2009-02-01
NC, USA
 

your exactly right

I have spent weeks discovering a huge memory leak in magento.  It duplicates an array of products over and over.  So for example you tried to import 1000 products you would have an array containing 1000 products 1000 times.  It eats up memory but also a lot of cpu parsing the info. I have cut down my script import time dramatically. The other memory leaks that deal with object clean up I haven’t been able to pin-point but I upgraded to php 5.3 and use garbage collection as you stated.  The fix is only 1 line of code.  If you are willing to compensate me I will give you the fix.  I usually would just post it but I really had to spend a lot of time and compile php into debug mode and run it thru valgrind and then use a php debugger.

If you import more than 100 products this fix really helps and it speeds up regular magento operation as well.  I contacted magento about this and they did not reply.

message me or email if interested

I also sell a couple of modules at http://ewtechnologies.com/store/magento

 
Magento Community Magento Community
Magento Community
Magento Community
 
coldlamper
Member
 
Total Posts:  35
Joined:  2009-02-01
NC, USA
 

I have decided to post my memory leak findings on my blog.
http://blog.ewtechnologies.com/magento-2/magento-memory-leak-1-4-1-1/

 
Magento Community Magento Community
Magento Community
Magento Community
 
johjoh
Member
 
Total Posts:  39
Joined:  2010-02-25
 
coldlamper - 27 October 2010 12:03 PM

I have decided to post my memory leak findings on my blog.
http://blog.ewtechnologies.com/magento-2/magento-memory-leak-1-4-1-1/

i tested your fix today and it works!!!!! no increasing memory-consumption any more, constant runtimes of aorund 1 sec per product during the hole process. thanks a lot!

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Working on testing it out. At first I thought my export had an early termination, but then I remembered how fast 1.3.1.1 was. Now if I can edge my memory_limit from 512MB back down to 256MB, I’ll be pretty happy! Anybody running this on live sites yet?

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Well, here’s dataflow export results on an export job I run from the command line.

Dell Dual Core Server: run time of 2 hrs, 28 minutes, Varien’s best unpatched 1.4.1.1

Virtual PC Single Core: run time of 23 minutes on 1.4.1.1 with Brian’s one line patch to disable the array of death (tm).

Notice the reduction in time on what would otherwise be highly inadequate virtual equipment in comparison to a medium weight server with disk i/o optimized hardware and 6GB memory at its disposal

Read it and weep for past lost processor cycles and wasted time waiting for exports that either took all night or never even completed.

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

For yuk’s sake, just ran export_all_products on a command line script.

Back in the day, this was a 6MB export that took 35 minutes under 1.3.1.1.

Just finished exporting 10.5MB in 25 minutes under 1.4.1.1 --> NOT 3 1/2 HOURS!!!

Resident Memory for the PHP process running the export hung around 185MB towards the end, NOT 380-410MB.

I’m dropping memory_limit to 256MB on my test server. If I extrapolate the memory curve out right, you probably don’t have to worry about bumping it up to 512MB till you hit 12,500 products.

Now going over to GUI profile run to see what I get with memory_limit at 256MB

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Last time I was able to run export_all_products from within Magento? 1 year, 7 days, 7 hours and 39 minutes ago.

Been running command line ever since November 5th of last year to just get it to complete, and with 512MB memory_limit.

memory_limit reduced to 256MB and running from System->Import/Export->Profile stock Export All Products profile no. 1

Run time 29 minutes, so the GUI costs you about 4 minutes time, not half bad. And this on Virtual PC instance of Ubuntu Workstation.  LOL

Now to hammer test it so I can get it over on the staging server.  cool smile

And for those who worry about that blank screen, it’s going to be that way for 28 1/2 minutes. 1.4.1.1 doesn’t seem to give you the “Starting profile execution, please wait..” until after it’s scanned through all your product, stuffed all the information into the dataflow_batch_export table and then starts giving you output as it scans through dataflow_batch_export, does all its data conversion junk and then creates the export file in the last 30 seconds.

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.xml

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        if (!
in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes)) {
            self
::$_joinTypes[] self::STRAIGHT_JOIN_ON;
            
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
        
}
    }

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.xml

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        
self::$_joinTypes[] self::STRAIGHT_JOIN_ON;
        
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
    
}
 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Ok, Magento officially is one weird jive turkey of a system.

Running on a dual core server, 6MB at work, with the above patch, I MUST keep memory_limit at 512MB, the export terminates with anything less, -->BUT<-- the export_all_products profile took slightly less than 8 minutes to complete.

Less capable Virtual PC server, 29 minutes on 256MB memory.

More capable Heavy Iron, 7 minutes 45 seconds on 512MB memory.

 
Magento Community Magento Community
Magento Community
Magento Community
 
bineamagento
Jr. Member
 
Total Posts:  2
Joined:  2011-08-13
 

I\’ve had some luck loading the product by SKU instead of loading it by id.

I too was looping through about 12k products in an import/update script. Loading by id crashed my script at about product 10500 and a memory consumption hitting 512 Mb. Loading by SKU, my memory rose to 240 Mb and then stopped there, which of most was from just initializing Magento.

Loading and saving 10k products amounts to about 240 Mb RAM used, not increasing

$[b]productToEnable Mage::getModel("catalog/product")->loadByAttribute("sku"$sku);[/b]
$productToEnable
         
->setStatus(Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
         ->
setDescription($description)
         ->
save();

Loading 10k products amounts to about 512 Mb RAM used, and increasing

$productToEnable Mage::getModel("catalog/product")->load($id);
$productToEnable
         
->setStatus(Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
         ->
setDescription($description)
         ->
save();
 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 2