Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Export All Products V1.4.0.1 timeout vs 1.3x
 
james123456
Jr. Member
 
Total Posts:  15
Joined:  2010-05-19
 

After patching it gives me a fatal error of not enough memory.. Which is good as it normally just hung for hours and hours.. this gave me that error after about 3 mins.

Adjusted memory limit to 512 and trying again…

IT WORKED!!!!!!! 9852 records exported in about 5 mins!!!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
james123456
Jr. Member
 
Total Posts:  15
Joined:  2010-05-19
 
james123456 - 18 November 2010 07:12 AM

After patching it gives me a fatal error of not enough memory.. Which is good as it normally just hung for hours and hours.. this gave me that error after about 3 mins.

Adjusted memory limit to 512 and trying again…

IT WORKED!!!!!!! 9852 records exported in about 5 mins!!!!

Note this was for a 1.4.0 install of magento

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Note that /lib/Varien/Db/Select.php has far reaching results beyond just making DataFlow work. MySQL Select statements are used every time you hit the database, so I’ve noticed almost a 1 second reduction in most page loads. It most shows where you are doing a category listing and have 25 products loading onto that page where multiple items are being retrieved.

Running on a dual core server, 6MB at work, with the above patch, I MUST keep memory_limit at 512MB, the export terminates with anything less, -->BUT<-- the export_all_products profile took slightly less than 8 minutes to complete for 9500 products.

Less capable Virtual PC server, 29 minutes on 256MB memory.

More capable Heavy Iron, 7 minutes 45 seconds on 512MB memory.

 
Magento Community Magento Community
Magento Community
Magento Community
 
imwy2cool
Sr. Member
 
Total Posts:  164
Joined:  2009-05-07
 

Hmm . . . my site is on a dedicated box Quad Core, 16GB Ram, etc., so it’s more than capable, but it won’t export and I have memory up to 1024MB. And I’ve let it run for an hour with 7,000 products after making that change.

 
Magento Community Magento Community
Magento Community
Magento Community
 
MathB76
Jr. Member
 
Total Posts:  3
Joined:  2010-09-16
 

Hello,
Just to say if you choose only one store, it works ok!
Sincerly,
Mathieu B.

 
Magento Community Magento Community
Magento Community
Magento Community
 
emmenko
Sr. Member
 
Total Posts:  96
Joined:  2010-10-20
 

Hi everyone,

I tested the script....I need it to export Product Stocks...it works but there is a little problem: I’m testing it with only 30 rows and it takes like 30 sec to run the entire script. I use Magento 1.4.1.1
I’m running it from the shell and I see that the export itself is fast (1-2 sec) becasue I get the message “Export Complete. BatchID ....” but then it takes the remaining sec to end the script.
I think this time is taken by the “DB cleaning part”, this one

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host']$db['user']$db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());
    
/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());

Is it really necessary? Is there no problem if the script runs for such a long time, althought the export is done in a few sec??
I also correct the Select.php script and set the memory_limit to 512M

ini_set('memory_limit''512M');
ini_set('max_execution_time''180000');

Thx in advance…

Nicola

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host']$db['user']$db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());
    
/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());

Is necessary to clean out the dataflow_batch tables. Otherwise you have an unending growth of contents in these tables that has been seen to rise to a pretty astronomical figure. However, in 1.4.1.1, once this patch has been applied, when you run exports from the import/export profiles in admin, the system starts cleaning out the dataflow_batch_export table. Running from the command line, I have found that I have to manually clean out the contents.

I find it humorous that you’re worried about 30 seconds, having had to allow this script to run for 2 1/2 to 3 hours to get any results for 9500 products, until applying the above mentioned 1.4.2.0rc1 patch which allowed it to finish in 7-9 minutes. I still don’t find 30 seconds to be much of a worry, but you might find that if you change the line

sleep(30);
to
sleep(5);
, it might tighten up your execution time a bit. I added the delay to allow Magento’s post export database operations a little time to settle down before clearing the dataflow_batch table.
 
Magento Community Magento Community
Magento Community
Magento Community
 
emmenko
Sr. Member
 
Total Posts:  96
Joined:  2010-10-20
 
chiefair - 23 November 2010 10:51 AM

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host']$db['user']$db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());
    
/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());

Is necessary to clean out the dataflow_batch tables. Otherwise you have an unending growth of contents in these tables that has been seen to rise to a pretty astronomical figure. However, in 1.4.1.1, once this patch has been applied, when you run exports from the import/export profiles in admin, the system starts cleaning out the dataflow_batch_export table. Running from the command line, I have found that I have to manually clean out the contents.

I find it humorous that you’re worried about 30 seconds, having had to allow this script to run for 2 1/2 to 3 hours to get any results for 9500 products, until applying the above mentioned 1.4.2.0rc1 patch which allowed it to finish in 7-9 minutes. I still don’t find 30 seconds to be much of a worry, but you might find that if you change the line
sleep(30);
to
sleep(5);
, it might tighten up your execution time a bit. I added the delay to allow Magento’s post export database operations a little time to settle down before clearing the dataflow_batch table.

Ahah ok...I just thought that this time will be increased with a lot of rows (as I said I only tested with 30 rows)...but this is just to clean, so there is no problem. Sorry for the stupid question...I’m still a noob in magento and I have a lot of stuff to learn yet.

Thx for the answer btw… ^^

Nicola

 
Magento Community Magento Community
Magento Community
Magento Community
 
jaywebguy
Jr. Member
 
Total Posts:  26
Joined:  2010-09-01
 

I’m so glad to find this post. this last page explains what I’ve been dealing with today. I’ll try applying the patch in the morning and see if I can get back out my 3500 products.

Thanks guys

Jason

 
Magento Community Magento Community
Magento Community
Magento Community
 
jaywebguy
Jr. Member
 
Total Posts:  26
Joined:  2010-09-01
 

Just wanted to add. I made the correction (see page 3) to the Select.php and sure enough the script ran and exported my products.

Thanks again.

Jason

 
Magento Community Magento Community
Magento Community
Magento Community
 
imwy2cool
Sr. Member
 
Total Posts:  164
Joined:  2009-05-07
 
chiefair - 12 November 2010 10:31 AM

So, anyone want to experiment with making 1.4.x.x DataFlow instead of being DataSlow?

Curing a Magento Memory Leak in 1.4.1.1

While Brian mentions it being used for reducing memory bloat during import, it probably affects export as well.

Addition of one line of code:

if (!in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes))
seems to eliminate an ever increasing array of death that’s probably what makes you assign ever increasing memory_limit amounts in the hopes that imports and exports will work.

Wow . . . it finally worked. I don’t know why it didn’t the first few times. I ran it for hours and hours . . . but I decided to try again (I changed nothing, as I had the patch in place ever since I read this), and within 3 minutes I had 10K products out! YAY!

Thank you SO MUCH! I’ve also noticed that I have not had a duplicate order issue anymore either. Several issues where things were timing out are no longer there. As was said earlier, this had pretty reaching effects.

Thank you again

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

@imwy2cool

The only reason I can think of that it wasn’t working at first is if you had the old Select.php code stuck in the Magento Cache, etc.

Since Magento’s EAV data storage scheme hammers the database with a bunch of self-joins on select, having the database select query develop a memory leak would affect just about every operation in the Magento system. Uncached page loads here lost about a second of wait time. From 4 to 3 seconds is a nice jump.

 
Magento Community Magento Community
Magento Community
Magento Community
 
nafnaf1000
Sr. Member
 
Total Posts:  209
Joined:  2008-02-21
 

Thank you so much for this script. It is a big help as I have to export all my items often....

I am trying to filter some data but it seems to be giving me all items. i want to do where UPC > 0 (not nill) but it is giving me all items anyway have any ideas?

 
Magento Community Magento Community
Magento Community
Magento Community
 
dotprkirk
Member
 
Total Posts:  50
Joined:  2008-04-22
 

That patch did the trick for me, 3100 products in 30 or so seconds.

Congrats, smart people. smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

@nafnaf1000

Having similar issues and needing to filter by more than the simple selection provided, I came up with this. You could use it to create a custom filter for your UPC attribute.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top