Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Reindexing after product import AKA catalog_product_import_after
 
maryanov
Jr. Member
 
Total Posts:  5
Joined:  2009-03-15
Ukraine Nikolaev
 

Hi folks.
Can somebody explain why magento reindexes(layered navigation, full text etc.) for ALL products in store, not just for products that were inserted/updated while product import?
Well I have about 700 products and I have huge problems with update or insert products via import, because reindexation after product import loads 100% CPU and takes about 30 minutes.
So I investigated the problem and found out:

Mage::dispatchEvent('catalog_product_import_after', array());
in app/code/core/Mage/Catalog/Model/Convert/Adapter/Product.php.
This magic string calls full reindexing for all products in store after product import. Well I do not like this thing and commented this string. Ok, now layered navigation and other indexing things are not updated after products import.
Then I asked my self why in product administration, when I edit 1 product and then press Save, why it does not call full reindexation of all products and at once it updates layered navigation and other indexing things. Why we can’t do same thing in product import. And I found out why layered navigation does not get updated while product import:
$product->setIsMassupdate(true);
$product->setExcludeUrlRewrite(true);
These two lines are at the end of Mage_Catalog_Model_Convert_Adapter_Product::saveRow(), before $product->save();
If I comment line
$product->setIsMassupdate(true);
, then layered navigation updates on each product that imports, i.e. not on ALL products in store, but only on products that are imported. Well I do not know what second line is for, but I think it disables urlRewrites update for separate product while saving.
So these my changes fixed the issue, when all store products are get reindexed. Now only products that are updated/inserted are reindexed.
Can somebody tell me, could I comment these lines or it will cause some system errors in future?
I think these were added for speedup importing products, well it does. But AFTER import it speed down whole my server.
 
Magento Community Magento Community
Magento Community
Magento Community
 
bcelenza
Member
 
Avatar
Total Posts:  44
Joined:  2008-08-27
 

Interesting find!

I tried what you recommended above and, while it did make the import slower (due to re-indexing each product), the overall import was much faster and did not result in a confusing lockup of the importer’s session.

I’ve attached a patch file for those interested in researching/trying this method. I understand the need for a full re-index after doing a product import (especially on imports > 200 or 300 products), but this method at least makes the import status % more accurate by removing the background re-indexing process.

Another possible solution would be to queue up a cron job to re-index the full catalog after a product import. While it will have a minor delay in product visibility in the catalog, it would allow the user importing to continue work without having to wait for the index to complete or open a new browser (to get a new apache thread).

 
Magento Community Magento Community
Magento Community
Magento Community
 
bcelenza
Member
 
Avatar
Total Posts:  44
Joined:  2008-08-27
 

Apologies, attachments don’t seem to be working for me. See the following pastie link for the patch:

http://pastie.org/627854

Indexapp/code/core/Mage/CatalogIndex/Model/Observer.php
===================================================================
--- 
app/code/core/Mage/CatalogIndex/Model/Observer.php    (revision 1558)
+++ 
app/code/core/Mage/CatalogIndex/Model/Observer.php    (revision 1560)
@@ -
223,+223,@@
      */
     public function 
catalogProductImportAfter(Varien_Event_Observer $observer)
     
{
-        $this->_getIndexer()->plainReindex();
+        
//$this->_getIndexer()->plainReindex();
         
$this->_getAggregator()->clearCacheData();
         return 
$this;
     
}
Index
app/code/core/Mage/Catalog/Model/Convert/Adapter/Product.php
===================================================================
--- 
app/code/core/Mage/Catalog/Model/Convert/Adapter/Product.php    (revision 1558)
+++ 
app/code/core/Mage/Catalog/Model/Convert/Adapter/Product.php    (revision 1560)
@@ -
699,+699,@@
             catch (
Exception $e{}
         }
 
-        $product->setIsMassupdate(true);
+        
//$product->setIsMassupdate(true);
         
$product->setExcludeUrlRewrite(true);
 
         
$product->save();
 
Magento Community Magento Community
Magento Community
Magento Community
 
maryanov
Jr. Member
 
Total Posts:  5
Joined:  2009-03-15
Ukraine Nikolaev
 

Hi bcelenza!
I did a bit more investigations and changes in catalog reindexing after import. Here is the patch http://pastie.org/628700. What these changes doing:
a) updates indexes(except catalog rules) only for products that were imported;
b) for Layered Navigation - updates only that attributes that were in import file.
These changes are very raw, I did them only for my self as temporary solution, if it will be interesting for somebody or may be magento team would like to put these changes to their repository(it will be awesome), then I will find time to improve these changes for you wink

 
Magento Community Magento Community
Magento Community
Magento Community
 
cstef
Jr. Member
 
Avatar
Total Posts:  10
Joined:  2008-06-17
 

bcelenza and maryanov,

Thank you so much!  This solution worked perfectly for me.  Our client updates their products frequently through Dataflow, and they have 19000+ products, so layered navigation was getting killed every time a profile was run.  Good work!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
moomoo
Sr. Member
 
Avatar
Total Posts:  192
Joined:  2009-07-10
 

Sorry for being slow, I don’t get the whole diff thing (or I have a vague idea but I can’t find that particular class in the product file).

Is it possible to paste your actual files? Is it possible to just put them into the /local route of app so that upgrades won’t break it (or maybe they should break it if they will import products differently?).

Am I right in thinking that this patch will speed up product upload? I have 1700 products and it takes a looong time, any speedup must be good. A particular problem I had was too many URLrewrites being made in the backend, then when I deleted them and re-imported products, none were being generated.

Thanks!

 
Magento Community Magento Community
Magento Community
Magento Community
 
maryanov
Jr. Member
 
Total Posts:  5
Joined:  2009-03-15
Ukraine Nikolaev
 
f3ze - 06 November 2009 02:11 AM

Sorry for being slow, I don’t get the whole diff thing (or I have a vague idea but I can’t find that particular class in the product file).

Is it possible to paste your actual files? Is it possible to just put them into the /local route of app so that upgrades won’t break it (or maybe they should break it if they will import products differently?).

Am I right in thinking that this patch will speed up product upload? I have 1700 products and it takes a looong time, any speedup must be good. A particular problem I had was too many URLrewrites being made in the backend, then when I deleted them and re-imported products, none were being generated.

Thanks!

Hi f3ze. Take a look to attachment. There are patches in it and already patched files, I dunno if these patches can be applied to your magento version, but I created them on 1.3.2.2.
P.S. of course these changes will be lost after any magento core update.

File Attachments
product_import_diff.tgz  (File Size: 14KB - Downloads: 335)
 
Magento Community Magento Community
Magento Community
Magento Community
 
moomoo
Sr. Member
 
Avatar
Total Posts:  192
Joined:  2009-07-10
 

Thanks a lot for that, I can understand the actual files much better smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
DaveGrex
Jr. Member
 
Total Posts:  12
Joined:  2009-11-30
 

Hey maryanov,
I incorporated this into my 1.4.1.1 installation of magento and it works perfectly except the url_rewrites are not being indexed.  Are your url_rewrites being indexed properly. If so have you done any updates to your code since your last post? I would really appreciate a solution to this.

Thanks!

 
Magento Community Magento Community
Magento Community
Magento Community
 
Jagat Kothari
Jr. Member
 
Avatar
Total Posts:  27
Joined:  2009-06-04
INDIA
 

Hi DaveGrex,

I had also found the same problem with Magento Product Import that after importing products, Catalog Url Rewrites are not executed as expected.

You can run the following Cronjob in order to reindex the URL Rewrites and generate the new Catalog Url Rewrites ::

php /path/to/magento/shell/indexer.php --reindex catalog_url

This worked for me.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mirasvit
Guru
 
Avatar
Total Posts:  639
Joined:  2009-08-22
 

Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.

Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html.

Even in case you place online shop even on a good hosting, sometimes the products, categories and attributes are saved slowly. This is due to the need to clear the cache and run re-indexing of stored items.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top