Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Magento 1.7.0.2: Very slow Productimport using Soap or Mage
 
kaiszy
Jr. Member
 
Total Posts:  1
Joined:  2012-08-16
 

Hi!

For a customer i compare sevceral shop systems to find out the best one. The last three days i test magento 1.7.0.2 CE.

I try to import round about 500.000 Products in a Testshop to measure if its possible to use Magento with this number of articles.

First i try to import the Articles with the soapApi. Because it was very very slow (round about 3-6 Seconds per Product with enabled cache) i change it to use direct the Mage api2 (and 1).

Its a little bit faster (approx. 2-3 Secondes per Product) but much to slow (with this speed we need 11 or more Days to import 500.000 Products...and i also have to import Attributes and Images...).

I use a new server for testing (2 Xeon 3.2 GHz, 64 GB Ram, very fast SAS-HDs) and also read the Article about “Tuning innodb performance for Meganto). I also install apc-Cache, tmpfs (for /var).

Why is the import sooo slow (makes it unusable fin my eyes) ? I don’t wan’t to think about the Shop Performance after import....

Is there something i can do to speed it up ? In my eyes the concept of magento (overloading is great solved) is very good. But the perfomance makes it unusable for my customer.

Thanks for any hint.

Best regards,
Kai.

 
Magento Community Magento Community
Magento Community
Magento Community
 
BelVG
Guru
 
Avatar
Total Posts:  306
Joined:  2011-02-16
Minsk, Belarus
 

Hi,

For import you may try Magmi - Magento Mass Importer
http://sourceforge.net/projects/magmi/
http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Wiki

 
Magento Community Magento Community
Magento Community
Magento Community
 
Mirasvit
Guru
 
Avatar
Total Posts:  639
Joined:  2009-08-22
 

Saving products / attributes takes too much time because of two things: re-indexing and clearing the cache. In both cases, we recommend you re-index and clearing the cache asynchronously, i.e. to process the queue in cron. Please pay attention to the extension which solve this task (Fast Asynchronous Re-Indexing)

 
Magento Community Magento Community
Magento Community
Magento Community
 
johnbullac
Jr. Member
 
Total Posts:  17
Joined:  2011-12-12
 

Kai, I can suggest you to look through this post about Magento upgrades
Just because I see you have lots of items to import, I guess it will help you to minimize the time looses and routine

 
Magento Community Magento Community
Magento Community
Magento Community
 
Store_Manager
Enthusiast
 
Avatar
Total Posts:  867
Joined:  2012-07-09
 

Hello Kai!

As I know most of web-tools cannot handle such big amount of products from one file in one procedure, as often at some point or it is better say at some thousands of products import can fail.
Here is the article that describes the aspects that should be taken into account importing huge amounts of products -
http://mag-manager.com/useful-articles/tipstricks/how-many-products-it-is-possible-to-import-to-magento-will-it-work-for-a-million-of-products
There you will also find the solution that handles import of million of products in one procedure from one file. It is possible due to the fact that it is done using pc resources rather than server ones.
In any case, my personal opinion is that having to import huge data sets, server stability and avoiding import crashes are more important then the speed of import.

 
Magento Community Magento Community
Magento Community
Magento Community
 
menoskedos
Jr. Member
 
Total Posts:  2
Joined:  2011-01-15
 

I use magmi and you must make sure that the url_keys of categories and products are unique.
If it is a url repeated, the end of the category or product urls, you’ll see some numbers that Magento put it there. You must include url_key field in the import file.

http://sourceforge.net/projects/magmi/
http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Wiki

I have magento 1.7 with 73.000 products.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top