Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 3 of 3
Bulk Import UNIMAGINABLY SLOW! Constant freezing during import, way too slow. 
 
Watercooling
Jr. Member
 
Total Posts:  28
Joined:  2008-07-24
 
bolasevich - 26 April 2010 04:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

Thanks for that tweak bolasevich,

it reduces import from 3-4 second to 0-5 seconds per product for me (importing ~ 8000 products).
Anyway it does not make me sleep well touching core functions. Do you have an idea if this functions are necessary?

Kind regards
Florian

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mammon
Jr. Member
 
Avatar
Total Posts:  20
Joined:  2009-08-24
 
bolasevich - 26 April 2010 04:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}


I would recommend taking a copy of that file and then placing it in the matching path in the app/core/local and you can also checkout my extensions at http://www.commerceextensions.com/

Thanks for tweak! It works on 1.4.0.1 and improves speed greatly. Can you explain what risks of using this tweak?

 
Magento Community Magento Community
Magento Community
Magento Community
 
bolasevich
Sr. Member
 
Avatar
Total Posts:  235
Joined:  2008-03-04
Newington, CT USA
 

no there are no issues with using this. In short by default it appears to clear the cache and reindex on every product that is imported everytime. So thats why its faster you can easily do both manually once after all products are imported. As for the core issue, its actually not if you just take that core file and copy it to the same path in your local folder and then comment out those few lines

Scott

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

check this topic : http://www.magentocommerce.com/boards/viewthread/201210/

this is an attempt for providing mass product import at reasonable speed

 
Magento Community Magento Community
Magento Community
Magento Community
 
pgulikers
Jr. Member
 
Total Posts:  27
Joined:  2009-02-21
 

I also experienced very slow import, expecially with simple products, the $product->save(); statement takes 8 seconds to complete.
Debugged SQL and saw a separate delete query per category on the catalog_category_product_index table, like:

## 21989 ## QUERY
SQL: DELETE FROM `catalog_category_product_index` WHERE (category_id=4259 AND store_id=1 AND product_id IN (’6690’))
AFF: 0
TIME: 0.0002

The query is quick but it is run for every existing category in the store.

I found out that the reason is I don’t have a category set for simple products but only for the associated configurable product and these import much quicker.

Solution:
Before $product->save(); add this line:

$product -> setCategoryIds('1');

and BANG! Up to speed again!!! I went from 1 record / 8 seconds to 3.85 records per second, that’s a 3000% performance gain!

Tried $product -> setCategoryIds(array()); as well, but that didn’t help.

 
Magento Community Magento Community
Magento Community
Magento Community
 
dweeves
Enthusiast
 
Total Posts:  877
Joined:  2010-06-26
FRANCE
 

preaching for my own church but:

for speedy import on simple products, take a look at magmi smile

on this post:

 
Magento Community Magento Community
Magento Community
Magento Community
 
AkelaVlk
Jr. Member
 
Total Posts:  1
Joined:  2010-10-20
 

There is one ultimate solution what works under any PHP version without any Magento patch.

Instead of

for ($id $i 1$i <= 1000$i++, $id++) {
    $productCollection 
Mage::getModel(\'catalog/product\')->getCollection()
            ->
addFieldToFilter(\'entity_id\'$id);
}

use

for ($id $i 1$i <= 1000$i++, $id++) {
   $comm 
\"php -q ./your_thread.php -i $id\";
    
$main_thr proc_open($comm$descriptorspec$pipes);
}

And put

<?php
    $params 
getopt(\'i:\');
    
$id $params[\'i];

    require_once($shopPath.\"/app/Mage.php\");
    Mage::app(\'
admin\');

        $descriptorspec = array(
            0 => array(\"pipe\", \"r\"),  // stdin is a pipe that the child will read from
            1 => array(\"pipe\", \"w\"),  // stdout is a pipe that the child will write to
            2 => array(\"file\", \"/tmp/error-output.txt\", \"a\") // stderr is a file to write to
        );

    $productCollection = Mage::getModel(\'
catalog/product\')->getCollection()
            ->addFieldToFilter(\'
entity_id\', $id);

Into separate file with name your_thread.php
At each iteration php call another php script. At the end of second php file garbage collertor free up all memory. So there will be no memory leak at all.

 
Magento Community Magento Community
Magento Community
Magento Community
 
sophiasstyle09
Member
 
Total Posts:  45
Joined:  2010-01-11
 

I haven’t read through all comments.  However from the little I have read I think RapidFlow is a great extension http://www.unirgy.com/products/urapidflow/ you might find useful.  My company has dealt with slow imports and impossible exports for a year now and we just installed the RapidFlow extension.  It is well worth the money and customer service will make sure that you get the extension successfully installed on your site.

Sincerely,
Gena

 
Magento Community Magento Community
Magento Community
Magento Community
 
cvs601
Jr. Member
 
Total Posts:  1
Joined:  2009-04-04
 

I am looking to import a lot of products.  I am going to try to increase my “number of records” to 10.  Someone said that 100 was too much, but I am wondering just how close you can get to that number.  If anyone else has information about the effect and/or success of changing this value please share!

 
Magento Community Magento Community
Magento Community
Magento Community
 
Bing Dicklepuss
Sr. Member
 
Total Posts:  91
Joined:  2010-07-26
 
cvs601 - 06 August 2011 01:23 PM

I am looking to import a lot of products.  I am going to try to increase my “number of records” to 10.  Someone said that 100 was too much, but I am wondering just how close you can get to that number.  If anyone else has information about the effect and/or success of changing this value please share!

I would advise you against using dataflow import at all.  Use the new magento import function.  You can import 10,000 new products in a few seconds.  With dataflow, it will take you at least 0.5 seconds PER PRODUCT.

 
Magento Community Magento Community
Magento Community
Magento Community
 
perspres
Jr. Member
 
Total Posts:  11
Joined:  2012-04-20
 
bolasevich - 26 April 2010 04:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}


I would recommend taking a copy of that file and then placing it in the matching path in the app/core/local and you can also checkout my extensions at http://www.commerceextensions.com/

This worked for me on 1.7, I’m currently getting about 2 uploads per second now :D Thanks bolasevich. Your extensions also look interesting. I’m certainly considering investing in your bulk uploader!

Dave

 
Magento Community Magento Community
Magento Community
Magento Community
 
Annushka
Jr. Member
 
Total Posts:  6
Joined:  2012-08-19
 

I usually use file2cart to import data. The fastest thing I’ve ever tried. Hope this service will help someone www.file2cart.com

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mirasvit
Guru
 
Avatar
Total Posts:  639
Joined:  2009-08-22
 

Hello there,

Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.

Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension of http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html.

 
Magento Community Magento Community
Magento Community
Magento Community
 
tortora
Member
 
Total Posts:  31
Joined:  2014-01-18
 

We have an application that offers an easier way to bulk import and addresses some of these issues.  You can get more info on it at www.ClaraStream.com and feel free to email me at info@clarastream.com if you have any questions.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 3 of 3