Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 2 of 3
Super Slow Product Import. 
 
Sprout Collective
Member
 
Total Posts:  64
Joined:  2007-08-31
Portland, OR / Orange County, CA
 

Ok, we modified our code based on your suggestions and things are much better. Thanks! smile

It still takes some time to process batches in the admin but they do complete without error or time outs. Also, I changed the amount to 50 and imported our 1500 items in no time without error.

Thanks again for the tip.

- TIM

 
Magento Community Magento Community
Magento Community
Magento Community
 
Michae1
Enthusiast
 
Total Posts:  826
Joined:  2007-08-31
 

Magento Version 1.0.19870.2 Available for Download, it has the fix for this problem. Please see the Release Notes

 
Magento Community Magento Community
Magento Community
Magento Community
 
thirty4
Jr. Member
 
Avatar
Total Posts:  16
Joined:  2008-02-28
 

EDIT: Magento ver. 1.0.19870.4 patch fixes all of the issues below.

Are you sure this fix works as it doesn’t seem to make much difference and the logic behind it doesn’t seem right either.. all the fix seems to have done is split the array into chunks for processing? maybe you can explain how the fix works?

What happens for me in this function is, products are all in the second store.. so when it loops through the first store it tries to find a product matching $product->getId() which none exist in that store so the $products variable gets overwritten with nothing and we fall in to continue;

Now on the second iteration, where all the products are, $products is nor an instance of Mage_Catalog_Model_Product, an array with any values, or a numeric variable so no Id filter gets added, so the collection just returns all products in the store that have a status of enabled and are visible.

Your fix also does a return false; after the first matching store is found too, but what if the products are in multiple stores?

foreach ($stores as $store{
            $collection 
Mage::getModel('catalog/product')
                ->
getCollection()
                ->
addAttributeToFilter('status'$status)
                ->
addAttributeToFilter('visibility'$visibility)
                ->
addStoreFilter($store);
            
/* @var $collection Mage_Catalog_Model_Resource_Eav_Mysql4_Product_Collection */

            
if ($products instanceof Mage_Catalog_Model_Product{
                $collection
->addIdFilter($products->getId());
            
else if (is_array($products) || is_numeric($products)) {
                $collection
->addIdFilter($products);
            
}
            $products 
$collection->getAllIds();

            if (!
$products)
                continue;

            
$step 1000;
            
$productCount count($products);
            for (
$i=0;$i<$productCount/$step;$i++) {
                $stepData 
array_slice($products$i*$step$step);

                if (
count($attributeCodes)) {
                    $this
->_getResource()->reindexAttributes($stepData$attributeCodes$store);
                
}

                
if (count($priceAttributeCodes)) {
                    $this
->_getResource()->reindexPrices($stepData$priceAttributeCodes$store);

                    
$this->_getResource()->reindexTiers($stepData$store);
                    
$this->_getResource()->reindexFinalPrices($stepData$store);
                    
$this->_getResource()->reindexMinimalPrices($stepData$store);
                
}
            }
            
return $this;
 
Magento Community Magento Community
Magento Community
Magento Community
 
es5812
Jr. Member
 
Total Posts:  11
Joined:  2008-07-10
 

Hey guys I am having the same slow import and freezing issue. It is taking about 10 seconds a product and it freezes anywhere between 100 and 300 every time. I am importing via an XML file and am using 35 fields including the import of three images sizes per product. My host right now is godaddy (which is probably the problem), but any insight into the matter would be greatly appreciated. If you need any more information from me just let me know. Thanks

Using: ver. 1.0.19870.4

 
Magento Community Magento Community
Magento Community
Magento Community
 
tartan
Jr. Member
 
Avatar
Total Posts:  25
Joined:  2008-05-09
 

1.1.6 single store

Ultra slow import of 4,500 products.  I’ll try to time it when I get the chance.

I hate to draw comparisons, but Zen stores I’ve implemented can import thousands upon thousands of products in mere minutes (or seconds), on lesser hardware.

<sigh>

 
Magento Community Magento Community
Magento Community
Magento Community
 
batman
Jr. Member
 
Total Posts:  8
Joined:  2008-07-30
 

My catalog has just under 6000 products. When I attempt to use dataflow to update inventory, price and status. The imports never finish. Sometimes 1000 products other times only 50 products would get imported and then the import would just hang.

I decided to “Disable” all of the cache settings while the import was running.

As soon as I did that, the import took off running.

I have since completed an update of my catalog, 5771 records using XML format with 25 records at a time. It ran in just under 46 minutes (2.1 records/second). The fields in my XML file are: store,websites,attribute_set,type,sku,status,qty,category_ids,is_in_stock,price,cost.

Not terribly fast, but I can live with that for now.

So there appears to be some issue with the import and caching.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Sousa
Guru
 
Avatar
Total Posts:  348
Joined:  2007-09-02
Porto, Portugal
 

But why this happen? why is very slow?

Maybe magento team can say something.

 
Magento Community Magento Community
Magento Community
Magento Community
 
cattyjin
Jr. Member
 
Total Posts:  17
Joined:  2008-10-28
 
JikeSpingleton - 27 May 2008 06:44 PM

I cant use magento until the import feature is respectable. The current implementation of this feature is worthless for business purposes. The current eCommerce software my client uses imports 13,000 products in less than 15 minutes. The current version of Magento can accomplish this same task, but cannot do it in less than 12 hours! I cannot possibly sell my services to a client when they are unable to add a list of products in 1 day’s time using an automated tool. Isn’t this what computers are for? I mean, I can probably add 13000 products in by hand in less than 12 hours. If you have had this same problem with a bulk magento product import and have found a fix PLEASE let me know. I am very desperate to find an answer to this problem because it is the only thing standing in my way at this point.

Please help!

Thanks.

Craig

Wow, I just import 400 products in 3 hours!!!!!!!!!
I have more than 180,000 products!!
I can’t bear it !

 
Magento Community Magento Community
Magento Community
Magento Community
 
cattyjin
Jr. Member
 
Total Posts:  17
Joined:  2008-10-28
 
batman - 06 November 2008 02:20 AM

My catalog has just under 6000 products. When I attempt to use dataflow to update inventory, price and status. The imports never finish. Sometimes 1000 products other times only 50 products would get imported and then the import would just hang.

I decided to “Disable” all of the cache settings while the import was running.

As soon as I did that, the import took off running.

I have since completed an update of my catalog, 5771 records using XML format with 25 records at a time. It ran in just under 46 minutes (2.1 records/second). The fields in my XML file are: store,websites,attribute_set,type,sku,status,qty,category_ids,is_in_stock,price,cost.

Not terribly fast, but I can live with that for now.

So there appears to be some issue with the import and caching.

Well, I have tried the method you provide, but it doesn’t work. It still imports slowly. Just about 3 products in 1 min! And it is running in the local machine!

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

its clear that this issue will never be fixed because it is not a priority of Varien’s to provide an efficient way to import products into Magento.

I have found, and been HAPPILY using alternate eCommerce software.

For all of you who are experiencing issues with Magento (most of you). Check out http://www.volusion.com. They are great. Excellent customer service, the software has EVERYTHING, i mean EVERYTHING. You have to host with them, but I quickly got past that because of how feature-packed the software is.

I imported 14,572 products in 47 seconds.

Good luck everyone.

 
Magento Community Magento Community
Magento Community
Magento Community
 
GreyWolf
Member
 
Total Posts:  46
Joined:  2008-07-31
 

Batman is correct…
Disable your cache in the admin>system>cache
Import speeds change for me, from 1 product every 6-9 seconds, to one product every second (or less) when I do this.

Its important to note that you must keep your import file under 2M in size. For me that’s about 1500 to 1800 products per file.

 
Magento Community Magento Community
Magento Community
Magento Community
 
JikeSpingleton
Jr. Member
 
Total Posts:  30
Joined:  2008-05-01
 

I’ve done that… thats one of the first things I did. If you’ve used other ecommerce software you’ll know that 1 second per product is unacceptably long.

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I am using a very nice dedicated server with rackspace, as well as a development server in house. Both are Quad Core 4GB RAM, Dell 6950’s with raid configurations. While we are not to the point of load balancing or separating our db from our application server, I would say our hardware is doing well. I have also invested precious time researching apache/php/mysql configuration settings. I haven’t installed apc for caching or anything, but I believe I have done a very good job of maxing our servers out.

I do not use dataflow, but I actually call the getCollection method on the product model and run my updates/imports manually. When I update a product, I can do about 1 per second. If I simply update qty, I can update thousands in seconds.

When I call getCollection on 2000 products, the product collection loads in a few seconds, but when I load 4700 products in the collection it takes about 1 min and 15 seconds. This is exponential growth :(…

Does this mean that when I finally get my 50,000 products in magento, this initial collection will take a very long time to load?

Does anyone else know why this getCollection takes so long?

Seriously… I am about ready to just do…

for( $i 0$i 4714$i++) //4714 is the amount of products that I know I have in magento
{
    $product 
Mage::getModel('catalog/product')->load($i);

    
/*
    ...here I can manipulate my product as such...
    */

}
 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I think I solved my problem, check out…

http://www.magentocommerce.com/boards/viewthread/30164/#t99176

I might help some of you doing custom imports

 
Magento Community Magento Community
Magento Community
Magento Community
 
invaderflid
Member
 
Avatar
Total Posts:  59
Joined:  2008-10-21
Northport Long Island New York
 

The company I work with just finished modifying the Magento import. They managed to improve the import from 1 product per second to about 83 products per second. If anyone is interested here is a link to the site Sysgen Media - Magento

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 2 of 3