Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 3
Super Slow Product Import. 
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

I cant use magento until the import feature is respectable. The current implementation of this feature is worthless for business purposes. The current eCommerce software my client uses imports 13,000 products in less than 15 minutes. The current version of Magento can accomplish this same task, but cannot do it in less than 12 hours! I cannot possibly sell my services to a client when they are unable to add a list of products in 1 day’s time using an automated tool. Isn’t this what computers are for? I mean, I can probably add 13000 products in by hand in less than 12 hours. If you have had this same problem with a bulk magento product import and have found a fix PLEASE let me know. I am very desperate to find an answer to this problem because it is the only thing standing in my way at this point.

Please help!

Thanks.

Craig

 
Magento Community Magento Community
Magento Community
Magento Community
 
srinigenie
Guru
 
Avatar
Total Posts:  539
Joined:  2008-02-04
 

Hi Craig,

Your observations are quite true!! Am facing the same issue .. infact I have a larger catalog (presently of around 50K and that is bound to grow a lot )!! .... I have managed to get the import to run through a Cron as browsers get timed out and inconsitencies in network connection at my end doesnot allow the import to run beyond a certain time limit.

On the Cron, the best speed I have observed so far is 600 records an hour which means more than 20 hours for a catalog of 13K records :( ... wish Magento team is working seriously on this as this is most desired and would save a lot of time for many!!

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

I have the same problems with about 2500 items in my catalog. Dataflow worked fine in Magento 1.0 but in the crurrent release is not more usable.

I started production with Magento 1.0 but now I have to stop my business after an upgrade! With new Magento release expect in 1-2 months and without a word of englightning from the Team about this (a part in the bug tracking section where looks like the feature is in debugging) is very diffcult to plans any professional activity.

Any workaround would be warmly welcomed, i am even considering downgrading to Magento 1.0 (is it possible/reccomandable or is it totally crazy thought?)

 
Magento Community Magento Community
Magento Community
Magento Community
 
easyc
Jr. Member
 
Total Posts:  9
Joined:  2008-01-16
 

It took me around 12 hours to get 13000 records in, and that was after an agonsising amount of hours trying to get the Dataflow Import to work.  I eventually found this guys site:

http://biz.metrofindings.com/main.page/magento_modules.html

Which fixes the export bug and allows you to actually export a product and use the file as a template for a decent import.  I had to further modify another file to stop the import process from timing out as the file was uploaded to magento.  Once it did start to import it went smoothly until the magento session timed out (while I’d left the PC - leaving around 800 items missed from the import) until I logged back in to refresh the session.

As far as magento goes for professionals, it just doesn’t cut the mustard at this stage.

I’m too far in to abandon my current project now (stayed with version 1.0) but I won’t use it again until I’m convinced the important issues have been resolved.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

I would be happy to import 13000 records in 12 hours!
That would means around 1000 product every hours. Not the greatest perfomance, but good enough to keep my store alive until the Dataflow goes 100%. I managed to import 1 product in 40 minutes.

What puts me down is the fact that Magento team hasn’t realized the gravity of this bug (ie, the answer here: http://www.magentocommerce.com/bug-tracking/issue?issue=2068

I guess because the bug doesn’t happens in every enviroments (?)

On other hand I have not experienced an export bug, tried the patch at metrofindings out of curiosity, but it did not help. I share you final thoughts about Magento, btw.

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

I cant believe they closed that bug. This is astounding. The import tool runs slower than it should if we were using a GPRS or 28k Modem. If this were 1989 I’d still be complaining about how slowly the products are being imported. That is sad.

The Magento Team seems more concerned with adding new features than making sure the ones they have now are rock solid. I have encountered this in the past on other web based software. I call it “The Beta State of Mind”. It happens when a development team gets sucked into working and thinking in a certain way during the beta stage of development, and when the product reaches a release version they just keep working the way they did when the product was in beta. When a software is put out as a release version the way that you work on that software must change to fit that new stage in development. New features should fall to a lower priority as the final product’s core functionality and stability becomes a major concern. Attaining and sustaining a userbase to your product only happens when your product is stable and functional in its current state. The second you lose sight of maintaining a stable, functional product and start focusing mainly on adding features is the second you achieve “The Beta State of Mind”. Keeping your product stable and bug free should be your main priority at all times, but when you have a release version this is even more true. I dont understand how the team can even have a list of bugs like that when so many people are working on this project. Someone must be in charge of squashing them, closing open bugs is not a squashed bug, its avoiding a bug you think is “icky”.

I dont get it. I am about to abandon all hopes of using Magento in a professional atmosphere. Maybe I’ll recommend it to my grandma for selling the quilt or two she knits in a year. Magento, in its current state, would fit that bill perfectly. Anything else is questionable.

Do you have over 100 products? Yes? hmmm… try something else. Too many for Magento.

Like I said in my previous bug report, I’m going to work on solving the energy crisis.... I’ll probably have that all worked out before the team fixes the giant hole in their product called DataFlow.

Bugs like this don’t kill software. Developers who ignore bugs like this do.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

Whatever Magento Developer’s “state of mind” is, at this stage the only thing I care is to find a solution.

If won’t be possible to get someone either fix this BUG or implementing a workaround (ie: injecting the .csv through MySQL queries) i will considering migrating to another platform.

But this is gonna be the last option since Magento is still good and especially because changing the software would mean trashing about 2 months work.

My question to the community are: can we join our efforts to finding a common solution? Can we hire a prefessional to get this fixed?
Any propositive solution will be discussed.

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

cibernoid:

I agree.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

easyc reported a possible solution here:

http://www.magentocommerce.com/bug-tracking/issue?issue=2068

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

My plans are:

1) waiting for easyc’ method. If it won’t works i will ask for support to Varien team requesting a quote. Hopefully they will take care of the problem or they will make an introduction to one partner.

Since it is CLEAR the dataflow bug does NOT occours in EVERY enviroment, I am curios to ask:

1) are you running multiple store?

2) Have you did - in your actual mysql db - many TESTING import before the last one?

3) Does it Dataflow worked in Magento 1.0? (in my case it worked and it stopped to to in 19700 release)

4) How big is your catalogindex_price table? (in my case is huge with about 3 million of records)

 
Magento Community Magento Community
Magento Community
Magento Community
 
cibernoid
Sr. Member
 
Avatar
Total Posts:  208
Joined:  2008-02-12
 

At least, I figured it out. In my case the problem was due to the presence of multiple stores.

I had 2 stores in my enviroment. I tried to deleted one of them and ran the import script again. He was fast as hell (compared to before).

Hope this will help people in trouble and also give some hints to the Magento team to find the BUG.

 
Magento Community Magento Community
Magento Community
Magento Community
 
dan_w
Sr. Member
 
Total Posts:  290
Joined:  2008-01-04
 

Likewise, with v1.0.19870, the product import came to a total standstill with dataflow. I even let it run overnight, 12 hours and it managed 5 products.

@cibernoid: Like you said, it was the multi-website/multi-store configuration that was causing the problem for me. I deleted the multiple websites and left three stores in tact to see if it was any better. 200 new products in under 5min. Not bad. I just hope I don’t have to delete and recreate websites everytime we want to update products.

 
Magento Community Magento Community
Magento Community
Magento Community
 
thirty4
Jr. Member
 
Avatar
Total Posts:  16
Joined:  2008-02-28
 

Howdy,

I posted a bug report tracking the problem down to the re-indexing function (for search, I assume) so find it strange that having one website speeds up the import process.

It is possible it is to do with having multiple stores in the re-indexing process though, I haven’t had the time to profile all of the code within that function to track down the exact bottle neck yet but it is definately to do with the indexing.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Tom Arnst
Jr. Member
 
Avatar
Total Posts:  22
Joined:  2007-12-20
Vanha Vaasantie 291, Noormarkku, 29600, Finland
 

Hi,

Same here, I had stores deleted one import was o.k. again.

Greetings,
Tom
www.printed-t-shirts.eu

 
Magento Community Magento Community
Magento Community
Magento Community
 
Sprout Collective
Member
 
Total Posts:  64
Joined:  2007-08-31
Portland, OR / Orange County, CA
 

When you say multiple stores, are you referring to Websites, Stores, or Store Views?

We’re having the same issues here, testing with only 3 store views in most cases.  Several different environments, Magento versions, servers, etc., same issues.

When we begin importing, everything seems fine, 1 or 2 seconds per record.  However, as things progress, each record takes longer and longer, and finally just completely died (overnight) at 1183 of around 1800.  Once it gets to around 15 seconds, you know something’s not right, and halfway there we’re at 30 seconds to a minute each.

Beyond importing, once we have 1k products in the application, another problem rears it’s ugly head; most actions that have to do with products, like simply saving even a *single* product, bulk updating the status of even 5 products, or refreshing catalog-related things.  It also has the tendency to bring the store to a halt when attempting to perform product-related updates.

We’d be incredibly grateful for any more ideas, even if it’s killing search, indexing, or some other feature for now, somehow.

I must add that we’re getting great performance in practically every other area, managing to run some of the fastest Magento environments we’ve seen so far.  Which is why this catalog issue is so surprising and show-stopping at this point.

- Cliff

 
Magento Community Magento Community
Magento Community
Magento Community
 
thirty4
Jr. Member
 
Avatar
Total Posts:  16
Joined:  2008-02-28
 
Sprout Collective - 06 June 2008 11:20 PM

When you say multiple stores, are you referring to Websites, Stores, or Store Views?
When we begin importing, everything seems fine, 1 or 2 seconds per record.  However, as things progress, each record takes longer and longer, and finally just completely died (overnight) at 1183 of around 1800.  Once it gets to around 15 seconds, you know something’s not right, and halfway there we’re at 30 seconds to a minute each.

Beyond importing, once we have 1k products in the application, another problem rears it’s ugly head; most actions that have to do with products, like simply saving even a *single* product, bulk updating the status of even 5 products, or refreshing catalog-related things.  It also has the tendency to bring the store to a halt when attempting to perform product-related updates.

- Cliff

This is because of the bug I reported in http://www.magentocommerce.com/bug-tracking/issue?issue=2097

The reason it affects saving in general is because the bug is in the product afterSave event, so anything that saves a product is affected when you have multiple stores.  What happens is that after the first iteration on the first store, the $products variable is overwritten and it begins rebuilding indexes for every product in the store.

Also the way it performs AJAX importing one product at a time also slows it down somewhat.  I patched the bug in my version and changed the AJAX importing to process 50 items at a time, and it improved performance to importing over 4,000 items last night in under an hour in a VM.

If you would like to try the following to patch it:

line 351 of code/core/Mage/CatalogIndex/Model/Indexer.php

if ($products instanceof Mage_Catalog_Model_Product{
                $collection
->addIdFilter($products->getId());
            
else if (is_array($products) || is_numeric($products)) {
                $collection
->addIdFilter($products);
            
}
            $productIds 
$collection->getAllIds();
            if (!
$productIds || empty($productIds))
                continue;

            if (
count($attributeCodes)) {
                $this
->_getResource()->reindexAttributes($productIds$attributeCodes$store);
            
}

            
if (count($priceAttributeCodes)) {
                $this
->_getResource()->reindexPrices($productIds$priceAttributeCodes$store);
                
$this->_getResource()->reindexTiers($productIds$store);
                
$this->_getResource()->reindexFinalPrices($productIds$store);
                
$this->_getResource()->reindexMinimalPrices($productIds$store);
            
}

for the AJAX batch processing you can update the file app/code/core/Mage/Adminhtml/Block/System/Convert/Profile/Run.php around line 238 by changing the 1 in array_chunk to a larger number, 10 works well but ramping it right up greatly improves the time it takes to process a large import.

$jsonIds array_chunk($importIds50);
                    foreach (
$jsonIds as $part => $ids{
                        $data 
= array(
 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 3