Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 2
[SOLVED] dramatic performance issues with product->save()
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

Hi there,

I’m currently experiencing some massive performance problems with Mage_Catalog_Model_Product->save(). This method has ever been slow, but It seems like it becomes slower the more products are in my store. I need to import or update about 30k products at a time and due to my latest sightings, this would take about three days and my products count keeps on growing. While using a custom import script - which works fine - I figured the save() method of the product model to be the problem. To head this issue I have implemented a small benchmark thingy which simply profiles the following code snippet:

$product Mage::getModel('catalog/product');
$product->load($id);
$product->setIsMassupdate($massUpdate);
$product->setExcludeUrlRewrite($urlExclusion);
$product->save();
$product->save();

And the results are just inacceptable to me:
Having 100 products around results in an average saving time of 2,7 seconds. But having 1000 products results in an average saving time of 43,7 seconds! And if I expect this behaviour to be linear (which is wrong, I know, but shocking), having 30k products would take about 22 minutes per product!

Please note that there were three tests having the mass update and/or url rewrite exclusion toggled on and off. My results are attached to this post as PDF documents. I have ran this benchmark on a 1.3.2.4 and 1.4.0.0 Magento with and without Magento’s cache enabled or disabled. The caching functionality nor the version haven’t had a noticeable impact on this, thus I don’t post these PDF docs too.

About my server configuration:
- Debian 5
- 3.6 GHz Quadcore, 12MB L1 Cache
- 8GB DDR3 RAM
- 15krpm 500GB RAID10
- Optimized MySQL Query Cache
- eAccellerator to cache PHP OpCodes
- Optimized .htaccess
- Disabled Logging in the Back-End

I hope you guys have an idea since this is ruining my ecommerce business with Magento. Thanks in advance wink

EDIT: Another info I forgot: There are no extension modules used, but my own one. And this doesn’t inherit any core model that could affect the saving procedure.

File Attachments
100products.pdf  (File Size: 264KB - Downloads: 151)
1000products.pdf  (File Size: 264KB - Downloads: 190)
 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

Another small note. I have profiled the SQL Queries that get executed within a single product->save() and its hairy. See yourself:

Worst 10 Queries

Seconds               Query
1
,094528198242        TRUNCATE TABLE catalogsearch_r..
0,827792167664        TRUNCATE TABLE catalogsearch_r..
0,540891170502        UPDATE `catalog_product_entity..
0,292192935941        DELETE FROM 
`catalogindex_aggr..
0,112812042236        TRUNCATE TABLE catalogsearch_r..
0,108272075653        SELECT `e`.*, `link_table`.`pa..
0,059484958649        UPDATE 
`catalogsearch_query` S..
0,016631841660        UPDATE 
`catalogsearch_query` S..
0,011770963669        DELETE FROM 
`catalogindex_aggr..
0,010859012604        DELETE FROM `catalogindex_aggr..

Worst Types

Type        Calls    âˆ‘ Seconds
TRUNCATE    11       2
,065194607
UPDATE      6        0
,626586914
DELETE      24       0
,347634077
SELECT      120      0
,135942936
INSERT      12       0
,021828175
COMMIT      4        0
,000383854

I wonder why there are so many TRUNCATEs around? Is this really neccessary?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Incognito
Guru
 
Total Posts:  322
Joined:  2008-08-07
Michigan
 

it is truncating the catalogsearch_result table.  The problem is that it refreshes a lot of things on product->save() ex layered navigation, search indexes, etc.  This is necessary if you want the new info to appear in the results, but if you are uploading thousands of products it is more efficient to do it all at once as opposed to one at a time.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Incognito
Guru
 
Total Posts:  322
Joined:  2008-08-07
Michigan
 
Bastian Selonke - 28 January 2010 01:13 AM

About my server configuration:
...
- 8GB DDR3 RAM
...

How much of this is your mySQL server allowed?

 
Magento Community Magento Community
Magento Community
Magento Community
 
coolshop
Sr. Member
 
Total Posts:  121
Joined:  2009-08-28
 

Bastian,

This is just let you I am facing the same issue. I do not have a solution yet but I am looking for one. I have version 1.3.2.4

I setup my store on a shared hosting server and I was able to save a product in 3 sec through php script callled though ssh.
And I was complaining that this was slow since I needed to load 100 000 products through my save php script. I was able to save 30 products in about 1 minute.

But I had to move to a VPS hosted server because of the 30 sec php script execution limit set by the hosting company for the shared hosting server.
Since I moved to the VPS server it takes 15 sec to save a product via php script. I have both magento application running: the shared hosting and the VPS hosting. Trying to compare both. But no luck so far.

I know I can achieve 3 sec per product though not great, it is better than 15 sec. So obviously there is something that is causing it to take that long. I am searching the entire forum for a possible answer.

It appears that having multiple stores aggravate the issue, but I am not sure, I am still digging.

I will let you when I find something. In the meantime I hope someone has a clue about this issue.

 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

First of all, thank you for your answers. I haven’t been able to get this to work in any better way yet :/

Incognito - 29 January 2010 08:17 AM

Bastian Selonke - 28 January 2010 01:13 AM

About my server configuration:
...
- 8GB DDR3 RAM
...

How much of this is your mySQL server allowed?

It’s like 4GB and this should be far more than enough. But I doubt changing this will affect the product->save() since a single call will “only” eat like 12 megs. I wonder how the “enterprise guys” handle this? I can’t imagine someone paying 10k bucks per year for a shop with 10 products or something.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ShopGuy
Guru
 
Total Posts:  462
Joined:  2008-09-07
 

I thought this issue was going to be solved in 1.4.

It looks like the solution would be to create another table and mark it everytime there is a product save and thenevery xx minutes check the table and if products have been saved then run the various truncates and other cleanup commands.

 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

But that would require me to touch Magentos core and I really dislike this, since it would screw my ability to “update on the fly”. However, the TRUNCATE takes only a small part of the time so there must be something that eats far more performance. I figured that my CPU sleeps most of the time and read of several users that experienced the same behaviour. I wonder why Magento has so damn high hardware requirements but does not use them entirely?

 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

Found the solution. Disable Magentos cache! This lowered my saving time by 75% in average!

I commented out the content of cleanCache() in Mage_Catalog_Model_Abstract/Product and it resulted in the biggest performance boost I have ever seen in Magento. For instance, a single simple product that has been saved in 13secs is now beeing saved in 3secs! After all, I’ve undone the changes to cleanCache() and disabled every cache option in Admin -> System -> Cache.

Yet, the cache sucks for the backend, but rules the front end to save some JOINs. But does it really worth to save a second on the frontend, but lose one minute in the backend? No!

 
Magento Community Magento Community
Magento Community
Magento Community
 
coolshop
Sr. Member
 
Total Posts:  121
Joined:  2009-08-28
 

This is does not make any sense. I will try it. When I disable cache, the front end is slow, so is the back end. But I will play with this further.

I am happy this worked for you. I hope it solves my problem as well. So far I have managed to load 10 000 products, each product taking 15 sec to save. This is a pain.
I have 90 000 more products to load.

 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 
coolshop - 03 February 2010 03:06 AM

This is does not make any sense.

So far, it does, since it seems to rebuild the whole cache for everything on every product->save(), but I might be wrong.

coolshop - 03 February 2010 03:06 AM

each product taking 15 sec to save. This is a pain. I have 90 000 more products to load.

As from what I know about the Magento community, we will stay alone in the dark with this problem. I have located the problem by profiling the PHP code executed in the product->save() and figured out that Core_Model_Catalog_Product::_afterSave() is consuming most of the time. Maybe you can hook in your own profiler and see whats the bottleneck?
 
Magento Community Magento Community
Magento Community
Magento Community
 
coolshop
Sr. Member
 
Total Posts:  121
Joined:  2009-08-28
 

Unfortunately for me disabling the cache did not improve the saving time in the backend. Still 15 sec per product. So I am still on the hunt for some improvements.

 
Magento Community Magento Community
Magento Community
Magento Community
 
a4345ergdfg45
Jr. Member
 
Total Posts:  24
Joined:  2008-09-09
 

Well, what about profiling?

 
Magento Community Magento Community
Magento Community
Magento Community
 
bugi
Jr. Member
 
Total Posts:  1
Joined:  2010-07-27
 

Hi all,
It is very interesting topic because I have the same problems like You guys. I my case I see that I have very big cpu load ~100% and I see that first records are added quite fast. I wonder that anyone resolve this problem? Cache disabling does nothing in my case.

I use Magento 1.4.1.1.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ladle
Sr. Member
 
Avatar
Total Posts:  173
Joined:  2009-06-11
 
bugi - 20 September 2010 10:17 PM

Hi all,
It is very interesting topic because I have the same problems like You guys. I my case I see that I have very big cpu load ~100% and I see that first records are added quite fast. I wonder that anyone resolve this problem? Cache disabling does nothing in my case.

I use Magento 1.4.1.1.

The cause I think is Catalog Price Rules.  When they are all turned off, products save fast.  When turned on, every time you save a product, it reapplies rules.  They need to have a setting to put catalog price rules on a cron schedule.

 
Magento Community Magento Community
Magento Community
Magento Community
 
chiefair
Mentor
 
Avatar
Total Posts:  1848
Joined:  2009-06-04
 

@ShopGuy

I thought this issue was going to be solved in 1.4.

Tears in my eyes, not sure whether I\’m laughing or crying…

@bugi

My performance matches your description, running 1.4.1.1 here.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 2