I am looping through approx. 10,000 xml elements and either creating or updating a product in the magento database. The problem is, each iteration through the loop uses more and more memory especially after loading a product and saving a product. This memory is never freed. Also, it is very slow, so I am wondering if there is a better way to do this.
//in class Import... $i=0; $total = count($xml->Item); $this->product = Mage::getModel('catalog/product');
I have the same problem. I upgraded to php 5.3.3 so i could use gc_collect_cycles to force garbage collection. It does work and collects a lot of real leaks and saves memory but Magento must be increasing the size of variables on load and save. These don’t show up as garbage and don’t get cleared out with gc_collect_cycles. I really wish the Magento team would look into this.
I have spent weeks discovering a huge memory leak in magento. It duplicates an array of products over and over. So for example you tried to import 1000 products you would have an array containing 1000 products 1000 times. It eats up memory but also a lot of cpu parsing the info. I have cut down my script import time dramatically. The other memory leaks that deal with object clean up I haven’t been able to pin-point but I upgraded to php 5.3 and use garbage collection as you stated. The fix is only 1 line of code. If you are willing to compensate me I will give you the fix. I usually would just post it but I really had to spend a lot of time and compile php into debug mode and run it thru valgrind and then use a php debugger.
If you import more than 100 products this fix really helps and it speeds up regular magento operation as well. I contacted magento about this and they did not reply.
i tested your fix today and it works!!!!! no increasing memory-consumption any more, constant runtimes of aorund 1 sec per product during the hole process. thanks a lot!
Working on testing it out. At first I thought my export had an early termination, but then I remembered how fast 1.3.1.1 was. Now if I can edge my memory_limit from 512MB back down to 256MB, I’ll be pretty happy! Anybody running this on live sites yet?
Well, here’s dataflow export results on an export job I run from the command line.
Dell Dual Core Server: run time of 2 hrs, 28 minutes, Varien’s best unpatched 1.4.1.1
Virtual PC Single Core: run time of 23 minutes on 1.4.1.1 with Brian’s one line patch to disable the array of death (tm).
Notice the reduction in time on what would otherwise be highly inadequate virtual equipment in comparison to a medium weight server with disk i/o optimized hardware and 6GB memory at its disposal
Read it and weep for past lost processor cycles and wasted time waiting for exports that either took all night or never even completed.
For yuk’s sake, just ran export_all_products on a command line script.
Back in the day, this was a 6MB export that took 35 minutes under 1.3.1.1.
Just finished exporting 10.5MB in 25 minutes under 1.4.1.1 --> NOT 3 1/2 HOURS!!!
Resident Memory for the PHP process running the export hung around 185MB towards the end, NOT 380-410MB.
I’m dropping memory_limit to 256MB on my test server. If I extrapolate the memory curve out right, you probably don’t have to worry about bumping it up to 512MB till you hit 12,500 products.
Now going over to GUI profile run to see what I get with memory_limit at 256MB
Last time I was able to run export_all_products from within Magento? 1 year, 7 days, 7 hours and 39 minutes ago.
Been running command line ever since November 5th of last year to just get it to complete, and with 512MB memory_limit.
memory_limit reduced to 256MB and running from System->Import/Export->Profile stock Export All Products profile no. 1
Run time 29 minutes, so the GUI costs you about 4 minutes time, not half bad. And this on Virtual PC instance of Ubuntu Workstation.
Now to hammer test it so I can get it over on the staging server.
And for those who worry about that blank screen, it’s going to be that way for 28 1/2 minutes. 1.4.1.1 doesn’t seem to give you the “Starting profile execution, please wait..” until after it’s scanned through all your product, stuffed all the information into the dataflow_batch_export table and then starts giving you output as it scans through dataflow_batch_export, does all its data conversion junk and then creates the export file in the last 30 seconds.
Ok, Magento officially is one weird jive turkey of a system.
Running on a dual core server, 6MB at work, with the above patch, I MUST keep memory_limit at 512MB, the export terminates with anything less, -->BUT<-- the export_all_products profile took slightly less than 8 minutes to complete.
Less capable Virtual PC server, 29 minutes on 256MB memory.
More capable Heavy Iron, 7 minutes 45 seconds on 512MB memory.
I\’ve had some luck loading the product by SKU instead of loading it by id.
I too was looping through about 12k products in an import/update script. Loading by id crashed my script at about product 10500 and a memory consumption hitting 512 Mb. Loading by SKU, my memory rose to 240 Mb and then stopped there, which of most was from just initializing Magento.
Loading and saving 10k products amounts to about 240 Mb RAM used, not increasing
