I am looping through approx. 10,000 xml elements and either creating or updating a product in the magento database. The problem is, each iteration through the loop uses more and more memory especially after loading a product and saving a product. This memory is never freed. Also, it is very slow, so I am wondering if there is a better way to do this.
//in class Import... $i=0; $total = count($xml->Item); $this->product = Mage::getModel('catalog/product');
I have the same problem. I upgraded to php 5.3.3 so i could use gc_collect_cycles to force garbage collection. It does work and collects a lot of real leaks and saves memory but Magento must be increasing the size of variables on load and save. These don’t show up as garbage and don’t get cleared out with gc_collect_cycles. I really wish the Magento team would look into this.
I have spent weeks discovering a huge memory leak in magento. It duplicates an array of products over and over. So for example you tried to import 1000 products you would have an array containing 1000 products 1000 times. It eats up memory but also a lot of cpu parsing the info. I have cut down my script import time dramatically. The other memory leaks that deal with object clean up I haven’t been able to pin-point but I upgraded to php 5.3 and use garbage collection as you stated. The fix is only 1 line of code. If you are willing to compensate me I will give you the fix. I usually would just post it but I really had to spend a lot of time and compile php into debug mode and run it thru valgrind and then use a php debugger.
If you import more than 100 products this fix really helps and it speeds up regular magento operation as well. I contacted magento about this and they did not reply.
Working on testing it out. At first I thought my export had an early termination, but then I remembered how fast 188.8.131.52 was. Now if I can edge my memory_limit from 512MB back down to 256MB, I’ll be pretty happy! Anybody running this on live sites yet?
Last time I was able to run export_all_products from within Magento? 1 year, 7 days, 7 hours and 39 minutes ago.
Been running command line ever since November 5th of last year to just get it to complete, and with 512MB memory_limit.
memory_limit reduced to 256MB and running from System->Import/Export->Profile stock Export All Products profile no. 1
Run time 29 minutes, so the GUI costs you about 4 minutes time, not half bad. And this on Virtual PC instance of Ubuntu Workstation.
Now to hammer test it so I can get it over on the staging server.
And for those who worry about that blank screen, it’s going to be that way for 28 1/2 minutes. 184.108.40.206 doesn’t seem to give you the “Starting profile execution, please wait..” until after it’s scanned through all your product, stuffed all the information into the dataflow_batch_export table and then starts giving you output as it scans through dataflow_batch_export, does all its data conversion junk and then creates the export file in the last 30 seconds.
Ok, Magento officially is one weird jive turkey of a system.
Running on a dual core server, 6MB at work, with the above patch, I MUST keep memory_limit at 512MB, the export terminates with anything less, -->BUT<-- the export_all_products profile took slightly less than 8 minutes to complete.
Less capable Virtual PC server, 29 minutes on 256MB memory.
More capable Heavy Iron, 7 minutes 45 seconds on 512MB memory.
I\’ve had some luck loading the product by SKU instead of loading it by id.
I too was looping through about 12k products in an import/update script. Loading by id crashed my script at about product 10500 and a memory consumption hitting 512 Mb. Loading by SKU, my memory rose to 240 Mb and then stopped there, which of most was from just initializing Magento.
Loading and saving 10k products amounts to about 240 Mb RAM used, not increasing