Magento Forum

Page 1 of 3
Bulk Import UNIMAGINABLY SLOW! Constant freezing during import, way too slow. 
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

Hi, I am attempting to migrate my catalog of 11,000 products over from MonsterCommerce via a .CSV import. I started out by attempting a small 2 product import which completed successfully. I am now trying to import a slightly larger csv containing 75 products and the import is taking WAYY too long. I have aborted the import several times due to it freezing/hanging randomly. The farthest that DataFlow has gotten is about 42.5% which took over a half hour, so that is about 1 minute per product that it imported.  At this rate it will take 8 days to import all of my products. This is unacceptable, the Data Import tool offered by MonsterCommerce takes about 2 minutes to import 2,000 products. This file is 99 bytes and contains only 75 products, there is no way that this is considered an acceptable import speed.

Is there a way to increase import speed that I am not seeing? If not this is a major bug. Also the field mapping is flat out broken on both import and export functionality.

I dont really have a problem reformatting my .csv files and arranging the columns in the way that Magento likes them, but I cannot have the import take that long. There are times where I need to import thousands of products multiple times per day. The DataFlow needs to be much more reliable. I can’t have constant freezing and week long import times. Is there a fix for this?

Any help would be much appreciated.

Thanks.

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

hmm ...very true ....the import needs a revisit in terms of functionality ... I have been able to import 900 products at a stretch .. but not without it hogging a lot of my memory and for all that importing only 6 products in a minute :( ....but still have no alternative as of now ..

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

from what I have seen of it thus far, Magento is not yet ready to be used as the foundation of a successful eCommerce site with a large rapidly expanding product catalog.

It is a shame because in theory it has all of the features I am looking for. It just can’t seem to deliver them in a reliable, time/cost effective manner.

I had all of the confidence in the world that I had found the solution I was looking for in Magento, but now I am finding more and more that I am probably not done looking for a suitable eCommerce solution.

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

common Jike !! you must have realized by now that you donot get everything you way always in opensource .... for your requirement of speed there are a thousand ways to improve a magento store and a lot depends on your catalog ... do keep in mind that Magento has just released its version 1.0 and there is a long way to go ...so the priority has been on rolling out a feature rich product and the idea is to improve the performance as we move on .... of course in case you need to improve the performance now, then take Magento’s professional help and get it done ...

 
Magento Community Magento Community
Magento Community
Magento Community
 
Serial Killa
Sr. Member
 
Avatar
Total Posts:  76
Joined:  2008-03-22
Malaysia
 

yeah , im looking for a solution as well , i was trying to import 45,000 data,
i left it overnight and i only manage only to import 24% of it ,
this is local machine , windows based server ,

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 
srinigenie - 07 May 2008 09:27 AM

common Jike !! you must have realized by now that you donot get everything you way always in opensource .... for your requirement of speed there are a thousand ways to improve a magento store and a lot depends on your catalog ... do keep in mind that Magento has just released its version 1.0 and there is a long way to go ...so the priority has been on rolling out a feature rich product and the idea is to improve the performance as we move on .... of course in case you need to improve the performance now, then take Magento’s professional help and get it done ...

My only turn off thus far is the rediculous slowness of it. The features are great, I just can’t say to a client “This does everything you need and more… but it takes 3 days to upload what right now is only taking 2 minutes.” or “The attributes feature is awesome, it’s totally what you need. Just sit here for the next 5 minutes while this loads and you’ll see what i mean.”

It is not worth it for people to sacrifice productivity for presentability when the productivity handicap is this severe. From a business aspect I dont think it is worth it to migrate to a solution that takes away this much productivity. Think about it, it takes 2 minutes right now for them to upload 2,000 products. It would take Magento 33 Hours to accomplish the same thing. I can’t present this as a solution to a client yet until that is fixed. If you are pulling the open source card I have to ask what is the point of even releasing it at all if you use that kind of excuse for it’s unpractical sluggishness. What niche is it filling if it is unusably slow and hangs/freezes constantly?

I would rather have less features in a quick/reliable solution than a ton of features in a product that creeps along at this torturous of a pace.

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

smile ...nobody can understand your plight better than myself as I am faced with importing a catalog of 1 lakh products .... and am myself working on improving the speed of import ...the speed can only improve from here ....and I can assure you that ... but magento -default installation is surely not for huge catalogs and needs quite a bit of customizations ...so the call is yours!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
gabrielk
Sr. Member
 
Avatar
Total Posts:  83
Joined:  2007-11-30
 

A little more info on your dataflow settings would probably be helpful, such as whether you’re uploading, importing from localhost or a remote server, mapping the field names or letting dataflow handle the mapping, etc.

I’ve only done testing with up to 20 products at a time, importing from a localhost connection, using the default mapping (using Magento’s field names in my csv), and it works in a reasonable amount of time without any hangups.  Eventually I’ll be uploading around 15,000 products in small batches (starting at 100 products per batch, going up from there as memory/speed/etc/ allows), so I’m curious about this topic myself, I just haven’t experienced any inherent slowness personally.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Laurent.J
Jr. Member
 
Total Posts:  5
Joined:  2007-12-24
 

i had a few trys with importing from oscommerce with the updated version of the oscommerce import tool.
i have 15000 customers to import.
3500 products
Muhahahahahahahaha!!!!
guess what....

 
Magento Community Magento Community
Magento Community
Magento Community
 
atang
Member
 
Total Posts:  59
Joined:  2008-07-10
 

I was only importing about 50 products, and that was taking over 60 minutes without completing.  I then tried setting the “Number of records:” field in the profile wizard for the Import profile to 3, and that seemed to speed it up quite a bit (finished in less than a minute.  I would imagine setting it higher will batch more rows at a time, at the expense of memory, but I haven’t had time to play around with it, and I don’t have thousands of products.  Maybe you should try setting it to “100” (make sure you specify enough memory for PHP)?

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

@atang oh! which version are you using? I dont see the ““Number of records:” field (am using version 1.0.9XXX)

 
Magento Community Magento Community
Magento Community
Magento Community
 
atang
Member
 
Total Posts:  59
Joined:  2008-07-10
 

I’m using Magento ver. 1.0.19870.4.  The field is in “Profile Information” under “Name”, “Entity Type”, “Direction”, and “Store”

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

ah! no wonder, I am with 1.0.19700 :(

 
Magento Community Magento Community
Magento Community
Magento Community
 
atang
Member
 
Total Posts:  59
Joined:  2008-07-10
 

It looks like it was added in release 1.0.19870.2.  The release notes list these changes:

# Improved products import speed, decreased product saving/updating time.
# Batch import records number is made configurable and can be changed in import profile.

 
Magento Community Magento Community
Magento Community
Magento Community
 
joesixpack
Member
 
Total Posts:  35
Joined:  2008-06-23
 

Thanks atang.

I set the Number of Records to 5 and was able to import about 1000 records at less than 1 second per record on a slow shared server. Not blazingly fast, but way better than what i was getting before (nowhere) when i had the number of records set at 1000. Didn’t realize that the Number of Records field is not the number of records being imported, but the number of records that Magento processes in chunk as it imports. If Number of Records is set to a large number like 1000, the server tries to process too many records at once and chokes.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ktsixit
Member
 
Total Posts:  68
Joined:  2008-08-20
 

can someone tell me where to find documentation about batch import. I dont even know where that thing is in the admin panel, cant find it. Answer someone pls

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 3