Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 2 of 3
Bulk Import UNIMAGINABLY SLOW! Constant freezing during import, way too slow. 
 
invaderflid
Member
 
Avatar
Total Posts:  59
Joined:  2008-10-21
Northport Long Island New York
 

I was having a lot of issues with the Magento import also. Products imported at 1 a second when it even ran. Most of the times the imports didn’t even finish. The company I do work with wrote a lot of custom code for the product import and really streamlined Magentos import. The products are importing at around 83 a second and the import doesn’t freeze up. I’ve uploaded files with over 65,000 products and they’ve finished very quickly.

The site has over 500,000 products and is growing. They were also able to speed up the search as once the product list got that large search was also suffering, but now its done in a few seconds. This is project started on Magento 1.1.6 and finished on 1.1.7 fyi. Check out Sysgen Media.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Johannes Lietz
Jr. Member
 
Total Posts:  12
Joined:  2009-05-12
Wuppertal, Germany
 

Currently I’m evaluating Magento for a new project. One of the requirements is that the store has to be capable of keeping 5,000,000 different products.

I’ve created a script to generate 5,000,000 products, currently resulting in a 7.5 GB .csv-file. But for now I’ve started off with importing 1000 products only, and importing even 1000 products is far too slow.

Are there any other ways to import products into Magento besides the csv-import function that are substantially faster?

I guess writing those products directly into MySQL through a custom import php + sql script will be much faster, but does anybody have a blueprint for such a script? Which tables have to be touched?

I must make sure Magento will still be running with good performance when containing 5,000,000 products…

 
Magento Community Magento Community
Magento Community
Magento Community
 
ckosny
Guru
 
Total Posts:  349
Joined:  2009-02-28
Luxembourg
 

Frankly I doubt that Magento can handle 5.000.000 products. There are ways to speed up the data import (although most likely it still would take quite a while for importing that many products), but I see a bigger issue in clearing caches or rebuilding stuff like the search index , the flat catalogue or the layered navigation.

If you really really need to use Magento with that many products I think you would be better off with the enterprise edition - this way you at least have the chance to get some direct help from the Magento guys (they are not very often seen in the forum). Although the price is quite steep…

Good luck

Claudia

 
Magento Community Magento Community
Magento Community
Magento Community
 
F. P.
Member
 
Total Posts:  55
Joined:  2007-12-09
 

That is indeed a very big problem with magento. A possible way would be to insert directly into the database and not with the dataflow. Dataflow seems to be nice for less than 100 products but not for thousands of products. But without exact documentation of the database it seems to be impossible to develop such a script.
I have one for oscommerce and zencart which works together with a distributor. It runs as a cron job. The first import takes some time to load all products, pictures, descriptions a.s.o.. But the daily import just takes a few minutes for about 60 000 products.
I did try to adapt it to work with magento, but it is a trial and error and I finally gave up.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Johannes Lietz
Jr. Member
 
Total Posts:  12
Joined:  2009-05-12
Wuppertal, Germany
 

Thanks for your replies!

I also had a look at the forum’s “Can Magento do...” section, but couldn’t find an answer, respectively there was a similar unanswered question on how many products Magento can handle.

Hopefully they’ll answer my support e-mail now.

 
Magento Community Magento Community
Magento Community
Magento Community
 
bolasevich
Sr. Member
 
Avatar
Total Posts:  235
Joined:  2008-03-04
Newington, CT USA
 

http://www.magentocommerce.com/extension/1894

my extension will do 25k + imports in one swing.. its still pretty slow but i imported 25k in less then a day.. no freezes.. no hangups.. nothing one swing worked perfectly.. will also do all types of products / tier pricing / categories / bundle products / configurable products / custom options.. etc etc

 
Magento Community Magento Community
Magento Community
Magento Community
 
edwardtilbury
Member
 
Total Posts:  66
Joined:  2008-08-01
Las Vegas, Nevada
 

Processed 10.3% 1898/18522 records

UGH!!!!! IT’s been going for like 8 hours!

I remember zencart was done with this in 5 minutes or less....

Much more stuff going on I suppose, but GEEEEEZZ… I’m going to be an old man before it’s done!

8 gig dual core machine ubuntu… still slow.

 
Magento Community Magento Community
Magento Community
Magento Community
 
danshields
Member
 
Total Posts:  32
Joined:  2009-03-11
 

my percentage never moves it just goes and then says done

 
Magento Community Magento Community
Magento Community
Magento Community
 
ArthurCatigbe
Member
 
Total Posts:  34
Joined:  2009-06-02
 

Does anyone have any idea how long it would take to upload 5k simple products with no custom options in Magento 1.3.2.4?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Johannes Lietz
Jr. Member
 
Total Posts:  12
Joined:  2009-05-12
Wuppertal, Germany
 
ArthurCatigbe - 16 February 2010 08:44 AM

Does anyone have any idea how long it would take to upload 5k simple products with no custom options in Magento 1.3.2.4?

This really depends on your system- and database configuration. With default settings I would estimate 1-10 seconds per product.

What can really speed up things here is tuning your MySQL database. Have a look for the following parameters in your my.cnf or add them (with some useful sample settings, but this really depends on your server):

query_cache_type 1
query_cache_size 
256M
innodb_buffer_pool_size 
512M
innodb_additional_mem_pool_size 
12M

BTW: My Magento installation currenly contains about 2.000.000 SKUs/products, but this is only possible with a custom import script that writes all data directly to the database. You can achieve many products per second here! grin

 
Magento Community Magento Community
Magento Community
Magento Community
 
capncaveman
Jr. Member
 
Total Posts:  22
Joined:  2009-10-24
 
Johannes Lietz - 16 February 2010 03:34 PM

ArthurCatigbe - 16 February 2010 08:44 AM
Does anyone have any idea how long it would take to upload 5k simple products with no custom options in Magento 1.3.2.4?

This really depends on your system- and database configuration. With default settings I would estimate 1-10 seconds per product.

What can really speed up things here is tuning your MySQL database. Have a look for the following parameters in your my.cnf or add them (with some useful sample settings, but this really depends on your server):

query_cache_type 1
query_cache_size 
256M
innodb_buffer_pool_size 
512M
innodb_additional_mem_pool_size 
12M

BTW: My Magento installation currenly contains about 2.000.000 SKUs/products, but this is only possible with a custom import script that writes all data directly to the database. You can achieve many products per second here! grin

Ok...so I am having the same problem.  It is taking longer than i would like to upload products.  If I create a my.cnf file with the parameters listed above...where would it be uploaded to the server?

Here is my specs for my dedicated server

Operating System: CentOS 5
Processor: Intel Core 2 Duo - 2.13 GHz
RAM: 2 GB

Would I be able to modify your suggestions to make it even faster?

 
Magento Community Magento Community
Magento Community
Magento Community
 
partscash
Member
 
Total Posts:  38
Joined:  2009-09-11
Indiana
 
Johannes Lietz - 16 February 2010 03:34 PM

ArthurCatigbe - 16 February 2010 08:44 AM
Does anyone have any idea how long it would take to upload 5k simple products with no custom options in Magento 1.3.2.4?

This really depends on your system- and database configuration. With default settings I would estimate 1-10 seconds per product.

What can really speed up things here is tuning your MySQL database. Have a look for the following parameters in your my.cnf or add them (with some useful sample settings, but this really depends on your server):

query_cache_type 1
query_cache_size 
256M
innodb_buffer_pool_size 
512M
innodb_additional_mem_pool_size 
12M

BTW: My Magento installation currenly contains about 2.000.000 SKUs/products, but this is only possible with a custom import script that writes all data directly to the database. You can achieve many products per second here! grin

This custom import script that writes all data directly to the database do you have one?  If so I will gladly pay you for it - please let me know.

 
Magento Community Magento Community
Magento Community
Magento Community
 
bolasevich
Sr. Member
 
Avatar
Total Posts:  235
Joined:  2008-03-04
Newington, CT USA
 

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}

I would recommend taking a copy of that file and then placing it in the matching path in the app/core/local and you can also checkout my extensions at http://www.commerceextensions.com/

 
Magento Community Magento Community
Magento Community
Magento Community
 
davidstrinity
Jr. Member
 
Total Posts:  7
Joined:  2009-10-03
 

Thank you for the “number of records” hint. I thought I was supposed to put how many lines I was trying to import!!! I had it set to 4900!! LOL.

Thank you so much. It is importing very quickly now (I set it to 5).

 
Magento Community Magento Community
Magento Community
Magento Community
 
Watercooling
Jr. Member
 
Total Posts:  28
Joined:  2008-07-24
 
bolasevich - 26 April 2010 04:50 AM

I’ve got a speed tweak that will shave a few off the import.

/public_html/app/code/core/Mage/Catalog/Model/Product.php

protected function _beforeSave()
{
//-----------------------fffffffffffff $this->cleanCache();

and

public function afterCommitCallback()
{
parent::afterCommitCallback();
//-----------------------fffffffffffff Mage::getSingleton(’index/indexer’)->processEntityAction(
//-----------------------fffffffffffff $this, self::ENTITY, Mage_Index_Model_Event::TYPE_SAVE
//-----------------------fffffffffffff );
return $this;
}

Thanks for that tweak, it reduces import from 3-4 seconds per product to 0.5 seconds. (I am currently importing ~ 8000 products).
Anyway removing code from the core functions does not make me sleep well :-( Are you sure that the mentioned functions are not necessary?

Any idea why bulk import slows down after the first 100-200 products?

Kind Regards
Florian

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 2 of 3