Magento Forum

Page 3 of 3
Super Slow Product Import. 
 
pixelaté
Member
 
Total Posts:  36
Joined:  2007-09-06
 
batman - 06 November 2008 02:20 AM

I decided to “Disable” all of the cache settings while the import was running.

As soon as I did that, the import took off running.

Thankyou!!!

This solved my issue. Going from ~12 seconds per insert to ~0.3 seconds!

My catalogue search isn’t working properly. I’m not sure if that’s related or I just stuffed something up however.

I wrote a piece of code (based on the code in the Magento Core) to enable and disable the cache. Use it before you start importing and after you finish the import.

public static function setCache($doEnable{
    
if (!$doEnable{
      Mage
::app()->cleanCache();
    
}

    $enable 
= array();
    foreach (
Mage::helper('core')->getCacheTypes() as $type=>$label{
      $enable[$type] 
$doEnable;
    
}
    Mage
::app()->saveUseCache($enable);
  
}
(Note, it doesn’t need to be a static function, I just implemented it this way)

For example:

MyClass::setCache(false);
  
// do funky import of x number of records...
  
MyClass::setCache(true);

Cheers,

Ian.

 
Magento Community Magento Community
Magento Community
Magento Community
 
pixelaté
Member
 
Total Posts:  36
Joined:  2007-09-06
 

It’s probably worth noting that I did some more experimenting with turning off the cache. After turning it off and refreshing “Layered Navigation Indices” and then running a product import with the cache ‘on’, it was running quite quickly. However, after about 10 imports, it started slowing down again.

It got to around 1-1.5 seconds per import before I decided to stay with my little code snippet above which is working great. Products end up in the system within a fraction of a second, processor/memory usage is very low and they’re fully searchable and discoverable via the front-end interface (I was having an issue where they weren’t showing up but it was unrelated to this).

So, turning off the cache should be the absolute first thing a user should try if their imports are slow.

Cheers,

Ian.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jmoney
Jr. Member
 
Total Posts:  2
Joined:  2008-12-30
 
pixelaté - 05 March 2009 07:10 PM

So, turning off the cache should be the absolute first thing a user should try if their imports are slow.

Cheers,


Ian.

I would recommend also clearing all temporary internet files that are cached as well. I’ve personally found that using Chrome with freshly cleared temp files drastically reduces the time, especially between the time it says files imported and finished profile execution.

Best of luck…

Jmoney

 
Magento Community Magento Community
Magento Community
Magento Community
 
Sousa
Guru
 
Avatar
Total Posts:  348
Joined:  2007-09-02
Porto, Portugal
 

I need to make one test, my client says the importation is very fast, but when i test it on my pc is very slow.

For me is great when he say that, but is strange he has a private IP.

Any one have a private ip?

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

Product import speed should improve soon with this bug being assigned. Please keep the below bug updated so that magento team prioritizes this for the next immediate release. Also update the bug with any programmatic inputs you may have.

http://www.magentocommerce.com/bug-tracking/issue?issue=8892

 
Magento Community Magento Community
Magento Community
Magento Community
 
lesya
Jr. Member
 
Total Posts:  23
Joined:  2009-05-25
 

You can use store manager for magento application, it enables product import to csv or excel file and then export back to store manager - i believe its much faster. It’s described in detail here http://docs.google.com/View?docid=d8z5g3s_226ckgnfxfb#_2_6_1_Import_Products_0468213

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mirasvit
Guru
 
Avatar
Total Posts:  639
Joined:  2009-08-22
 

Hi,

Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.

Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension of http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html.

Even in case you place online shop even on a good hosting, sometimes the products, categories and attributes are saved slowly. This is due to the need to clear the cache and run re-indexing of stored items.

 
Magento Community Magento Community
Magento Community
Magento Community
 
karantan
Jr. Member
 
Total Posts:  8
Joined:  2010-11-04
 

Hi

I’ve developed easy product transfer in and out of Magento. I check it out. All suggestions are welcome

It allows you to:
- Export products from Magento to XML (at the moment it supports only smaller B2B shops).
- Transfer products between Magento shops (simple and configurable).
- Import products from XML file (again smaller B2B shops are supported atm) to Magento.

The site is available on http://hub.modulabs.com

NOTE: system is still under development. all suggestions are welcome.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 3 of 3