Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Export All Products V1.4.0.1 timeout vs 1.3x
 
doddy
Jr. Member
 
Total Posts:  21
Joined:  2008-08-15
 

same problem here.

export won`t run and no file is created.
There ist a bug report thread already opened

 
Magento Community Magento Community
Magento Community
Magento Community
 
markw87
Jr. Member
 
Total Posts:  27
Joined:  2010-06-28
 

Im also trying to use the CrobJob method but Im not gonna get my hopes up, Its really frustrating im trying to export 3500 products. Works fast on smaller quantities I dont quite understand why its taking so long, how has this issue not been addressed seems so odd.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mizpah
Member
 
Avatar
Total Posts:  54
Joined:  2009-12-09
Birmingham
 

Have just run into this problem myself - unfortunately we have upgraded from 1.3 to 1.4 - and managed that transation fine, after recoding templates etc - now however I have got to the first price update for products since the upgrade to live - and I have till Friday to update the prices on 21,000 products.

So my first task is to export everything into one csv, ready to drop into a prepared spredsheet to vlookup all the changes into - and I cant!  (white screen, continually loading, no file in var/export) This process however worked fine on 1.3.x.

Does anyone have any ideas for a route forwards ? As from now, we are effectively in a position where we run the risk of selling certain items at a loss - so I really need to be able to do this immediately!

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

1.4.1.1 -> hours for a whole lot of nothing. In 1.3.1.1, the answer was to jack the memory limit up to 512M. At 1G, NSL here

One of the major downfalls with Magento is the lack of exporting SKU ranges. They never planned beyond a 20 shirt boutique on this one.

Update:: There may be a separate php.ini file for your php command line operations. Set the memory limit there to 1024M and run the php script here. Be patient, it will take hours. The export I’m running is a subset of the product info, includes only necessary items like sku, store, type, name, price, weight, and whatever other items need to be reviewed in Access for changes.

I’m considering doing a SQL query outside of Magento to extract this information. When in doubt bypass the restriction…

Magento Data Dribbles 1.4.1.1 tongue rolleye

Magento Data Dribbles 1.4.1.1 Update:: Imports and Exports slowly fill these tables, shrinking them may help get back some speed

Add logging to your export scripts so you can monitor completion and runtime. Make it log to your own export_profile.log instead of filling system.log with non-system information.

Somewhere towards the top add the line

Mage::log('Start Run Profile No. x'null'export_profile.log');
and
Mage::log('End Run Profile No. x'null'export_profile.log');
towards the bottom after the main parts of the script have run. The profile number is what’s in the first column of your import/export profiles. You get nice date/time stamped entries in your own log for manual and chron scripts.

To use SQL -> Information for sql queries

For less than 30 seconds to results, this dumps pricing for store 0, you’ll have to check your appropriate attribute id’s to make sure they match. Assumes your database is named magento:

SELECT p_entity.sku AS skup_varchar.value AS namep_decimal.value AS price
FROM magento
.catalog_product_entity p_entitymagento.catalog_product_entity_decimal p_decimalmagento.catalog_product_entity_varchar p_varchar
WHERE p_entity
.entity_id p_varchar.entity_id AND p_entity.entity_id p_decimal.entity_id AND ((p_entity.type_id='simple') AND (p_varchar.attribute_id=56) AND (p_varchar.store_id=0) AND (p_decimal.attribute_id=60) AND (p_decimal.store_id=0))
 
Magento Community Magento Community
Magento Community
Magento Community
 
doddy
Jr. Member
 
Total Posts:  21
Joined:  2008-08-15
 

Dear guys,

does anyone found a solution to export products in 1.4?

greats

doddy

 
Magento Community Magento Community
Magento Community
Magento Community
 
Aerostah
Jr. Member
 
Total Posts:  15
Joined:  2009-02-18
 

I too have a store with 3500+ products and we’ve been suffering from stalling product exports since upgrading to 1.4.0.1.

It is a disappoinment to see how such an essential feature remains broken for months.  The quality of our store’s products continue to deteriorate, with a contanstly growing web catalog —it’s been challenging for our team to keep up with manufacturer price updates—all maintenance is manual.

Can anyone recommend a third party extension that can be used while the devs work towards a solution?

Thanks,

Dan

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

For posterity, here’s a script that will dump base pricing for simple product in Store 0 (admin) as a CSV file.

5 Seconds for 10,000 items

fields are store, sku, name, price

<?php
/* To ensure this will only be run from the server, uncomment block, include server IP address
 * and run from cron job or ssh session
 */ 
// if ($_SERVER['REMOTE_ADDR'] !== 'SERVER IP ADDRESS HERE') {
//    die("Run from terminal window.");
// }

/* Include Mage and run as admin in store 0 */

require_once 'app/Mage.php';
umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
$userModel Mage::getModel('admin/user');
$userModel->setUserId(0);
$storeName Mage::app()->getStore()->getWebsite();

/* Set variables, since in EAV the entity and attribute id are sequential and uniquely assigned, 
 * you'll need to look them up in the eav_attribute table for your magento installation to make
 * sure you have the right value
 */

$read         Mage::getSingleton('core/resource')->getConnection('core_read');
$pEntity     Mage::getSingleton('core/resource')->getTableName('catalog_product_entity');
$pVarchar    Mage::getSingleton('core/resource')->getTableName('catalog_product_entity_varchar');
$pDecimal    Mage::getSingleton('core/resource')->getTableName('catalog_product_entity_decimal');
$nameAttrId  56;
$priceAttrId 60;

/* SQL query to get base prices on simple product for store 0 */

$query 'SELECT p_entity.type_id AS type, p_entity.sku AS sku, p_varchar.value AS name, p_decimal.value AS price
      FROM ' 
$pEntity ' p_entity, ' $pDecimal ' p_decimal, ' $pVarchar ' p_varchar
          WHERE p_entity.entity_id = p_varchar.entity_id AND p_entity.entity_id = p_decimal.entity_id AND ((p_entity.type_id="simple") AND (p_varchar.attribute_id=' 
$nameAttrId ') AND (p_varchar.store_id=0) AND (p_decimal.attribute_id=' $priceAttrId ') AND (p_decimal.store_id=0))
      ORDER BY p_entity.sku'
;

$results $read->fetchAll($query);

/* straight dump to screen without formatting */

// print_r($results);

/* output to CSV file */

$fh fopen('export_base_price.csv''w');

foreach (
$results as $row{
    fputcsv
($fh$row);
}

?>
 
Magento Community Magento Community
Magento Community
Magento Community
 
imwy2cool
Sr. Member
 
Total Posts:  164
Joined:  2009-05-07
 

I’m going to bump this. I’ve resorted to disabling a few hundred at a time and then exporting all “disabled” to get a few hundred out at a time. Very much a pain.

Someone on the Magento team, please respond to this issue.

 
Magento Community Magento Community
Magento Community
Magento Community
 
fbctech
Jr. Member
 
Total Posts:  19
Joined:  2010-03-29
 

I gave up on the Magento export import for products awhile ago.

Using Store Manager I\’ve had some success and it has a progress bar.

I use http://www.magneticone.com/store/Magento.Modules

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Taking the script posted here...

combining it with the knowledge that there’s a housekeeping problem...

and mashing this log maintenance script designed to help alleviate the problem with the import/export script...

we come up with a command line script that can run an import or an export profile, be run from cron for off-hours use and that can do it’s own housekeeping once the import or export is done by flushing its respective dataflow_batch_(import|export) table.

<?php

/***********************
 * Import/Export Script to run Import/Export profile 
 * from command line or cron. Cleans entries from dataflow_batch_(import|export) table
 ***********************/

$mageconf './app/etc/local.xml';  // Mage local.xml config
$mageapp  './app/Mage.php';       // Mage app location
$logfile  'export_data.log';       // Import/Export log file

/* uncomment following block when moved to server - to ensure this page is 
 * not accessed from anywhere else 
 */

//if ($_SERVER['REMOTE_ADDR'] !== '<your server ip address>') {
//   die("You are not a cron job!");
//}


/* System -> Import/Export -> Profiles get profile ID from 
 * Magento Import/Export Profiles
 */

$profileId 9;

/* Post run housekeeping table bloat removal
 * imports use "dataflow_batch_import" table
 * exports use "dataflow_batch_export" table
 */

$table 'dataflow_batch_export';

/* Scan Magento local.xml file for connection information */

if (file_exists($mageconf)) {

$xml 
simplexml_load_file($mageconfNULLLIBXML_NOCDATA);

$db['host'$xml->global->resources->default_setup->connection->host;
$db['name'$xml->global->resources->default_setup->connection->dbname;
$db['user'$xml->global->resources->default_setup->connection->username;
$db['pass'$xml->global->resources->default_setup->connection->password;
$db['pref'$xml->global->resources->db->table_prefix;



else {
    Mage
::log('Export script failed to open Mage local.xml'null$logfile);
    exit(
'Failed to open Mage local.xml');
}


/* Initialize profile to be run as Magento Admin and log start of export */

require_once $mageapp;
umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

$profile Mage::getModel('dataflow/profile');
$userModel Mage::getModel('admin/user');
$userModel->setUserId(0);
Mage::getSingleton('admin/session')->setUser($userModel);
$profile->load($profileId);
if (!
$profile->getId()) {
    Mage
::getSingleton('adminhtml/session')->addError('ERROR: Incorrect profile id');
}

Mage
::log('Export ' $profileId ' Started.'null$logfile);

Mage::register('current_convert_profile'$profile);
$profile->run();
$recordCount 0;
$batchModel Mage::getSingleton('dataflow/batch');

Mage::log('Export '.$profileId.' Complete. BatchID: '.$batchModel->getId(), null$logfile);

echo 
"Export Complete. BatchID: " $batchModel->getId() . "\n";

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host']$db['user']$db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());
    
/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());


?>
 
Magento Community Magento Community
Magento Community
Magento Community
 
Petrache Nicolae
Jr. Member
 
Total Posts:  29
Joined:  2010-03-09
 
chiefair - 06 November 2010 06:21 PM

Taking the script posted here...

combining it with the knowledge that there’s a housekeeping problem...

and mashing this log maintenance script designed to help alleviate the problem with the import/export script...

we come up with a command line script that can run an import or an export profile, be run from cron for off-hours use and that can do it’s own housekeeping once the import or export is done by flushing its respective dataflow_batch_(import|export) table.

<?php

/***********************
 * Import/Export Script to run Import/Export profile 
 * from command line or cron. Cleans entries from dataflow_batch_(import|export) table
 ***********************/

$mageconf './app/etc/local.xml';  // Mage local.xml config
$mageapp  './app/Mage.php';       // Mage app location
$logfile  'export_data.log';       // Import/Export log file

/* uncomment following block when moved to server - to ensure this page is 
 * not accessed from anywhere else 
 */

//if ($_SERVER['REMOTE_ADDR'] !== '<your server ip address>') {
//   die("You are not a cron job!");
//}


/* System -> Import/Export -> Profiles get profile ID from 
 * Magento Import/Export Profiles
 */

$profileId 9;

/* Post run housekeeping table bloat removal
 * imports use "dataflow_batch_import" table
 * exports use "dataflow_batch_export" table
 */

$table 'dataflow_batch_export';

/* Scan Magento local.xml file for connection information */

if (file_exists($mageconf)) {

$xml 
simplexml_load_file($mageconfNULLLIBXML_NOCDATA);

$db['host'$xml->global->resources->default_setup->connection->host;
$db['name'$xml->global->resources->default_setup->connection->dbname;
$db['user'$xml->global->resources->default_setup->connection->username;
$db['pass'$xml->global->resources->default_setup->connection->password;
$db['pref'$xml->global->resources->db->table_prefix;



else {
    Mage
::log('Export script failed to open Mage local.xml'null$logfile);
    exit(
'Failed to open Mage local.xml');
}


/* Initialize profile to be run as Magento Admin and log start of export */

require_once $mageapp;
umask(0);
Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

$profile Mage::getModel('dataflow/profile');
$userModel Mage::getModel('admin/user');
$userModel->setUserId(0);
Mage::getSingleton('admin/session')->setUser($userModel);
$profile->load($profileId);
if (!
$profile->getId()) {
    Mage
::getSingleton('adminhtml/session')->addError('ERROR: Incorrect profile id');
}

Mage
::log('Export ' $profileId ' Started.'null$logfile);

Mage::register('current_convert_profile'$profile);
$profile->run();
$recordCount 0;
$batchModel Mage::getSingleton('dataflow/batch');

Mage::log('Export '.$profileId.' Complete. BatchID: '.$batchModel->getId(), null$logfile);

echo 
"Export Complete. BatchID: " $batchModel->getId() . "\n";

/* Connect to Magento database */

sleep(30);

mysql_connect($db['host']$db['user']$db['pass']) or die(mysql_error());
mysql_select_db($db['name']) or die(mysql_error());
    
/* Truncate dataflow_batch_(import|export) table for housecleaning */

$querystring "TRUNCATE ".$db['pref'].$table;

mysql_query($querystring) or die(mysql_error());


?>

can you post details on how do use this? thanks

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

Import or Export?

Most basic instructions, running as an export, go to your import/export profiles, find the export you want to run and look in the first column for the profile ID. In this case my profile ID is 9

/* System -> Import/Export -> Profiles get profile ID from 
 * Magento Import/Export Profiles
 */

$profileId 9;

Save the script as export09.php in your document root and execute from the command line in your document root with php -f export09.php, put it into the background with nohup php -f export09.php & or set up a line in your crontab to run it at the proper time.

You will have to set up your export through System->Import/Export->Profiles to customize it to what you specifically want to export.

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

So, anyone want to experiment with making 1.4.x.x DataFlow instead of being DataSlow?

Curing a Magento Memory Leak in 1.4.1.1

While Brian mentions it being used for reducing memory bloat during import, it probably affects export as well.

Addition of one line of code:

if (!in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes))
seems to eliminate an ever increasing array of death that’s probably what makes you assign ever increasing memory_limit amounts in the hopes that imports and exports will work.
 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

The Array Of Death(tm) has been noted and here’s the official repair in the new version. Maybe Data Will Flow again!

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.php

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        if (!
in_array(self::STRAIGHT_JOIN_ONself::$_joinTypes)) {
            self
::$_joinTypes[] self::STRAIGHT_JOIN_ON;
            
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
        
}
    }

Excerpt from 1.4.1.1 /lib/Varien/Db/Select.php

public function __construct(Zend_Db_Adapter_Abstract $adapter)
    
{
        parent
::__construct($adapter);
        
self::$_joinTypes[] self::STRAIGHT_JOIN_ON;
        
self::$_partsInit = array(self::STRAIGHT_JOIN => false) + self::$_partsInit;
    
}

EDIT: File name corrected.

 
Magento Community Magento Community
Magento Community
Magento Community
 
evnetwork
Jr. Member
 
Avatar
Total Posts:  13
Joined:  2010-07-20
 
chiefair - 16 November 2010 10:42 PM

Excerpt from 1.4.2.0rc1 /lib/Varien/Db/Select.xml



Excerpt from 1.4.1.1 /lib/Varien/Db/Select.xml

I think you meant Select.php rasberry

Thanks for the effort in trying to figure this out, we badly needed a solution for this.

I’m gonna try this patch and the script you pasted earlier and report here smile

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top