Magento

bolasevich - 26 April 2010 05:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

Thanks for that tweak bolasevich,

it reduces import from 3-4 second to 0-5 seconds per product for me (importing ~ 8000 products).
Anyway it does not make me sleep well touching core functions. Do you have an idea if this functions are necessary?

Kind regards
Florian

bolasevich - 26 April 2010 05:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}


I would recommend taking a copy of that file and then placing it in the matching path in the app/core/local and you can also checkout my extensions at http://www.commerceextensions.com/

Thanks for tweak! It works on 1.4.0.1 and improves speed greatly. Can you explain what risks of using this tweak?

no there are no issues with using this. In short by default it appears to clear the cache and reindex on every product that is imported everytime. So thats why its faster you can easily do both manually once after all products are imported. As for the core issue, its actually not if you just take that core file and copy it to the same path in your local folder and then comment out those few lines

Scott

check this topic : http://www.magentocommerce.com/boards/viewthread/201210/

this is an attempt for providing mass product import at reasonable speed

I also experienced very slow import, expecially with simple products, the $product->save(); statement takes 8 seconds to complete.
Debugged SQL and saw a separate delete query per category on the catalog_category_product_index table, like:

## 21989 ## QUERY
SQL: DELETE FROM `catalog_category_product_index` WHERE (category_id=4259 AND store_id=1 AND product_id IN (’6690’))
AFF: 0
TIME: 0.0002

The query is quick but it is run for every existing category in the store.

I found out that the reason is I don’t have a category set for simple products but only for the associated configurable product and these import much quicker.

Solution:
Before $product->save(); add this line:

and BANG! Up to speed again!!! I went from 1 record / 8 seconds to 3.85 records per second, that’s a 3000% performance gain!

Tried $product -> setCategoryIds(array()); as well, but that didn’t help.

preaching for my own church but:

for speedy import on simple products, take a look at magmi

on this post:

There is one ultimate solution what works under any PHP version without any Magento patch.

Instead of

use

And put

Into separate file with name your_thread.php
At each iteration php call another php script. At the end of second php file garbage collertor free up all memory. So there will be no memory leak at all.

I haven’t read through all comments.  However from the little I have read I think RapidFlow is a great extension http://www.unirgy.com/products/urapidflow/ you might find useful.  My company has dealt with slow imports and impossible exports for a year now and we just installed the RapidFlow extension.  It is well worth the money and customer service will make sure that you get the extension successfully installed on your site.

Sincerely,
Gena

I am looking to import a lot of products.  I am going to try to increase my “number of records” to 10.  Someone said that 100 was too much, but I am wondering just how close you can get to that number.  If anyone else has information about the effect and/or success of changing this value please share!

cvs601 - 06 August 2011 02:23 PM

I am looking to import a lot of products.  I am going to try to increase my “number of records” to 10.  Someone said that 100 was too much, but I am wondering just how close you can get to that number.  If anyone else has information about the effect and/or success of changing this value please share!

I would advise you against using dataflow import at all.  Use the new magento import function.  You can import 10,000 new products in a few seconds.  With dataflow, it will take you at least 0.5 seconds PER PRODUCT.

bolasevich - 26 April 2010 05:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}


I would recommend taking a copy of that file and then placing it in the matching path in the app/core/local and you can also checkout my extensions at http://www.commerceextensions.com/

This worked for me on 1.7, I’m currently getting about 2 uploads per second now :D Thanks bolasevich. Your extensions also look interesting. I’m certainly considering investing in your bulk uploader!

Dave

I usually use file2cart to import data. The fastest thing I’ve ever tried. Hope this service will help someone www.file2cart.com

Hello there,

Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.

Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension of http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html.

We have an application that will allow you to easily;

-migrate large amounts of product data into your Magento storefront
-run bulk uploads / imports
-easily update existing products on your Magento storefront.

If interested you can get more info at www.clarastream.com or you can email me at tortora@clarastream.com