Magento

chiefair - 16 November 2010 11:42 PM

The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.php

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    {
        parent::__construct($adapter);
        if (!in_array(self::STRAIGHT_JOIN_ON, self::$_joinTypes)) {
            self::$_joinTypes[] = self::STRAIGHT_JOIN_ON;
            self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
        }
    }

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.php

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    {
        parent::__construct($adapter);
        self::$_joinTypes[] = self::STRAIGHT_JOIN_ON;
        self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
    }

EDIT: File name corrected.

I was about to post about how this code does not fix the long export issue in PE 1.9.0, however as I was typing it… the export finished! Under 5 minutes for ~5000 products. :D

Nice, isn’t it? The one thing you do have to watch out for is running your export during normal customer viewing hours. That 5-8 minutes will cause some intense CPU usage during the time it runs, but it’s not doing it for hours now. 

chiefair - 23 December 2010 09:37 AM

Nice, isn’t it? The one thing you do have to watch out for is running your export during normal customer viewing hours. That 5-8 minutes will cause some intense CPU usage during the time it runs, but it’s not doing it for hours now. 

I just checked our server stats, the export did create a peak in CPU usage, however it was not enough to throw an alert. Overall a nice fix

Same goes for imports as to being careful on when you do it. Just did an import and watched with slight misgivings as top started showing 135-148% usage during the post import reindex. People probably felt that, guessing 200% is two cores fully occupied out of 4.

Thank you very much chiefair, I am now able to export products quickly.

Have you or anyone else been able to import files using this method?

I can import files (slowly) using Magento\’s built-in tool. I used your method to run my import profile, and when I manually visit my import.php (containing your script, with the ID for my import job), the page loads, then displays the success message. But nothing has been added to the database.

This memory leak (ever expanding array) is at a far lower level than import and export profiles. It is in the Select routine being passed to mysql to collect data and so therefore globally affects any data selects, whether it’s doing an export, or a simple product page view. In fact one of the side affects besides being able to once again use the export_all_products profile from the GUI was an across-the-board reduction of page load by at least a second.

Imports involve inserts and updates, not something addressed in the Select routine patch. One of the major misunderstandings I’ve seen with importing is the total misunderstanding of the Number of Records field. This is not the total number of records you are importing (total in the file) but the number of records updated as a subset of the number of records in the import file. It is a chunking factor. Set it to 10 and you will update a 500 record file 10 records at a time until they’re all done.

This is very important to understand because you create a User Defined Memory Leak ™ if you set it to the total number of records in the file. Set it high enough and the import crashes somewhere around 400 to 600 records unless you’ ve set your memory_limit to some god-awful level in the 1-2GB range.  Basically you have to fiddle with the Number of Records, with settings in the 5-25 record range being a place to start. With memory_limit set to 256MB, 10 would probably be appropriate, 512MB = 25. Basically fiddle with it until you raise it to the point you start noticing a drop-off in speed or memory exhaustion commences.

Click thumbnail to see full-size image

I just applied this, and was able to get my export down from 3h49m to 6 minutes. 

9800 products.

Greate work to whomever found this solution.

That would be Brian Kieth, and soon thereafter Magento in time to get included in the 1.4.2.0 update.

For anyone out there who asks, it is patched in 1.4.2.0 and if you have slowness, you have other problems to track down.

james123456 - 18 November 2010 08:12 AM

After patching it gives me a fatal error of not enough memory.. Which is good as it normally just hung for hours and hours.. this gave me that error after about 3 mins.

Adjusted memory limit to 512 and trying again…

IT WORKED!!!!!!! 9852 records exported in about 5 mins!!!!

hello,
Please help as I do not understand what needs to be done step by step :( I’m new with magento and with programming...please help and tell me what need to be done in order to export all products (I’m using magento 1.4.0.1)
Thx

This Fix worked for me too, BUT after an successfull export (the created csv file is there and it is complete), i get an error:

any ideas why?

Just reporting in that this worked for me in 1.4.0.1.

Before this patch, would take hours (if at all?). Now, 2500 product stocks exported in seconds! Thanks to chiefair and Brian Kieth!

@bastic

Your system is barfing on the dataflow_batch_export table cleanup. This log table monitor can tell you how many records are stuck in the table. Probably way more than you want to know. Nexcess has a nice little script that truncates the dataflow_batch_import and dataflow_batch_export tables.

Any solution for Ver1.5?

As to 1.5.x.x, not really sure where to go with that one.

The problem with a drastic slowing of export when upgrading from 1.3.x.x to a 1.4 version lower than 1.4.2.0 was a severe memory leak in the Select.php function. It was repaired in the 1.4.2.0 code base.

If you’re having problems with 1.5.x.x, it’s a whole new issue, barring any regression errors which means you probably need to post a new forum thread specific to 1.5.x.x. Otherwise it gets buried and out of sight on an older non-related problem.

I’m running 1.4.1.1 with the modification which was great when exporting sub 10k products. Now I’m trying to export 20k and it runs and runs before stopping with no error. I really don’t think this is a server issue since I’m on a VPS with extremely high settings. I can still export around 5k grouped items but the other 15k simple products won’t budge…