Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Import CSV fast
 
Dirty1Harry
Jr. Member
 
Total Posts:  4
Joined:  2012-09-09
 

Hello guys,

I need to import 1 million products (CSV) to magento as fast as possible.

Advise me something not very expensive please.

 
Magento Community Magento Community
Magento Community
Magento Community
 
KeatonOFF
Jr. Member
 
Total Posts:  5
Joined:  2012-09-11
 

I suggest you to try this service. Not to promote it, but because it really helps. And it is not expensive. You can estimate the price which depends on the quantity of data you’d like to import. As for me it is very useful and very fast.

I think it is good for your situation.
Hope it will help you.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Store_Manager
Enthusiast
 
Avatar
Total Posts:  867
Joined:  2012-07-09
 

Hi there,

If you need to run one-time import, I can offer you trial version of Store Manager for Magento -
http://www.magentocommerce.com/magento-connect/store-manager-for-magento.html
trial version is free and fully functional, so if you need to run import only once, it won\’t cost you a penny.

The application has built-in import tool that handles import of unlimited number of products of different types - simple, configurable, bundled, grouped with images (small, base, thumbnails). Also it allows on a fly to generate categories and update inventory. In case you need to make mark-ups, it can be done directly during import by expressions.

If you need to regularly update your store, then you can buy the software - you pay only once and get the live-time license without having to pay additional fee as most online services do.

Hope it was helpful smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
pbosch
Member
 
Avatar
Total Posts:  46
Joined:  2008-09-01
Enschede, The Netherlands
 

To be honest, I doubt importing one million of products is idealistic on the standard data adapter, nor this will be possible with storemanager. There are solution on the market for a free fast importer http://sourceforge.net/projects/magmi/ .This one is opensource and developed.

However, there is no promise the importer will work in future versions when Magento releases a new one. Store manager I doubt it will help here when importing a CSV with millions of products.

The process:
Importing a CSV (each CSV is different) involves many things such as recalculations, adding weight options, extra options or combining colums or CSV’s.  I have uploaded a sample MS XML file which can be created also from CSV. If you want to import you need to process your CSV and optimize it for Magento e.g. with scripts. Just open the MS XML and you will see the basis template of Magento which you can interpretate as a CSV or a MS XML file. (data headers) We created such a tool at www.magdeveloper.com

Advice:
-Do not try to import a million products on the standard dataflow of Magento . It will simply take too long…
-If you are going to do this how would you update the products daily? ( you need raw SQL injection to make a fast script)
-You need processing of the CSV before uploading and also updating or checking if a product has already been added.
-Have a look at some importers at the market that offer you flexible management and setup a direct connection between your supplier and update it daily e.g. via a cronjob.

Let me know if you need anymore help.

File Attachments
Sampledata.xml  (File Size: 317KB - Downloads: 198)
 
Magento Community Magento Community
Magento Community
Magento Community
 
davenurave
Jr. Member
 
Total Posts:  11
Joined:  2012-09-14
 

Thanks for the details, pbosch. but I can’t help wondering, how folks manage 1 milloin products…

 
Magento Community Magento Community
Magento Community
Magento Community
 
pbosch
Member
 
Avatar
Total Posts:  46
Joined:  2008-09-01
Enschede, The Netherlands
 

Hi, i am glad to hear that you find my answer useful because else i woudn\\\’t contribute.

I think that one million is durable and Magento users have this but there are some limits in re-indexing.
If you want a million products make sure you have a right server environment (database max limit), good caching system.
You definitively want to check if the magento indexing is working correctly because if you have a full database you might get troubles in the future. For this kind of problems (depending on you business process of importing daily etc) I would recommend to use SLA\\\’s with Magento.

So the main dependent factors are you cache and Indexing besides a good hosting package. Also importing from a CSV would not be very practical if you need to open the file on your local pc. But I have given you some directions where to go. Still it depends where you get the CSV from and if there are better options besides CSV (input format) and which data parser, extension you use.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Store_Manager
Enthusiast
 
Avatar
Total Posts:  867
Joined:  2012-07-09
 

Pbosch, thank you for your post. I will clarify some points you\’ve mentioned for other forum users.

I doubt importing one million of products is idealistic on the standard data adapter, nor this will be possible with storemanager.

We haven\’t mentioned that you should run the initial data input in one import with Store Manager. What is more, you can do it for free and we\’ve provided a direct URL to check it. Also we do provide support free of charge, so we gladly assist in such complex tasks and we find the exact solution that fit our customer\’s needs.

However, there is no promise the importer will work in future versions when Magento releases a new one. Store manager I doubt it will help here when importing a CSV with millions of products.

Store Manager DOES support latest version of Magento approximately withing 1 month after official release, free solutions cannot afford it in most cases, so you need to be either very technical to be able to update it or need to look for another solution.

If you are going to do this how would you update the products daily? ( you need raw SQL injection to make a fast script)

The Store Manager has an option to add new products only, thus duplicated products will be ignored
Also Store Manager allows you to update one or two fields only - you can update price and quantity without updating any other fields (there\’s no need to update description or reload images each time
There is not need to override the catalogue rebuild and cache clearing. Store Manager can do it for you once import is done.

You need processing of the CSV before uploading and also updating or checking if a product has already been added

Store Manager will make it for you automatically. Simply setup one import for price and quantity update and another task for adding new products, thus skipping the updating of existing ones.

I have uploaded a sample MS XML file which can be created also from CSV. If you want to import you need to process your CSV and optimize it for Magento e.g. with scripts. Just open the MS XML and you will see the basis template of Magento which you can interpretate as a CSV or a MS XML file.

The main feature of Store Manager is that you do not need to adjust the file in any way! You can do it on a fly while import. Including adding some data or skipping some unnecessary columns. Basis Magento template has lots of columns like type_id and others.
Usual customer will never know what to input there, while the .csv file will not be imported correctly without the data.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top