Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 6 of 6
Export All Products V1.4.0.1 timeout vs 1.3x
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

I’m currently stress testing a LiteSpeed webserver for import in 1.4.1.1 because the export once again is running out of memory somehow.

768M memory_limit and on running an export profile, Magento dies with a malloc error indicating that there’s some internal problem with a 128M memory limit being exceeded. DESPITE EVERYTHING TELLING YOU IT’S 768M LOCAL AND SYSTEM.

Here’s the resulting digital spew:

Fatal error: Out of memory (allocated 134479872) (tried to allocate 4096 bytes) in /home/x/public_html/app/code/core/Mage/Eav/Model/Entity/Attribute/Source/Table.php on line 62

Fatal error: Out of memory (allocated 134742016) (tried to allocate 24 bytes) in /home/x/public_html/lib/Zend/Db/Select.php on line 924

Fatal error: Out of memory (allocated 133431296) (tried to allocate 5541 bytes) in /home/x/public_html/lib/Zend/Db/Statement/Pdo.php on line 228

Fatal error: Out of memory (allocated 133169152) (tried to allocate 24 bytes) in /home/x/public_html/lib/Zend/Db/Select.php on line 924

Having been through four different Linux installs with Magento, with different iterations of Apache/Mod_php5, Apache/FastCGI, LiteSpeed/lsphp5, you will find on high volume production sites, that there’s always something dropping a wrench in the transmission. You get one system tuned up to where it works, and decide you need more performance, only to bring yet another fresh set of problems. Apache/FastCGI seems to work the best with the least resource hogging (LiteSpeed = fast with issues), but you’ll need some nice heavy iron to run it with any speed. Thankfully, exports are not Magento’s main function, or I’d have put a bullet through its brain a long time ago.

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

@Quantifier

Sounds like you’ve hit the next memory_limit level and are silently running out of memory (nothing odd about this on exports).

Have you considered running your exports from the command line (Eliminates horrific amounts of overhead)?

This currenly is the only thing that works for me, as our LiteSpeed system has developed a strong aversion to allocating memory for DataFlow Export despite the memory_limit being set above 512M.

This is on 1.4.1.1 patched for the Select.php memory leak with 9800+ products.

 
Magento Community Magento Community
Magento Community
Magento Community
 
YasoKuuhl
Jr. Member
 
Avatar
Total Posts:  5
Joined:  2010-02-22
Germany, Hanau
 
chiefair - 16 November 2010 10:42 PM

The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.php

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        if (!
in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes)) {
            self
::$_joinTypes[] self::STRAIGHT_JOIN_ON;
            
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
        
}
    }

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.php
public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        
self::$_joinTypes[] self::STRAIGHT_JOIN_ON;
        
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
    
}

EDIT: File name corrected.

this sloved my problem, thx on chiefair!

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

You’re welcome! It really helps.

Addendum to the Lightspeed saga. It really helps to have Hosting Providers that know their stuff. Soon after the memory post above, we got everything hammered out and I think nothing of running profile exports during business hours. Lightspeed’s the way to go and if you’re running 4 or more cores, one core takes all the load for the export.

Imports are another issue altogether as the reindex invoked at the end can interfere with your customer’s user experience.

 
Magento Community Magento Community
Magento Community
Magento Community
 
MageKing
Member
 
Avatar
Total Posts:  54
Joined:  2010-08-28
Ahmedabad
 
YasoKuuhl - 20 December 2011 08:05 PM

chiefair - 16 November 2010 10:42 PM
The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.php
public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        if (!
in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes)) {
            self
::$_joinTypes[] self::STRAIGHT_JOIN_ON;
            
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
        
}
    }

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.php
public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        
self::$_joinTypes[] self::STRAIGHT_JOIN_ON;
        
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
    
}

EDIT: File name corrected.

HAS SOLVED MY PROBLEM TO SOME EXTENT.

 
Magento Community Magento Community
Magento Community
Magento Community
 
elegente
Jr. Member
 
Total Posts:  5
Joined:  2011-09-29
 

at the begin, i can export all products, but when my products exceed to 3 thousands it can’t export, anyone can tell me how to fix it.

Thanks

 
Magento Community Magento Community
Magento Community
Magento Community
 
myladeybugg
Member
 
Total Posts:  32
Joined:  2013-01-30
 

I’m running 1.7.0.2 and am attempting to export all products. I have been attempting to export them through the regular export, not dataflow. I’m not sure if this has anything to do with it, but I keep getting an internal 500 error. I increased my PHP memory and time limit, but am still getting the 500 error. Here’s my PHP settings:

max_execution_time = 900
;Increased from 90
max_input_time = 900
;Increased from 90
memory_limit = 512M
;Increased from 128M

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 6 of 6