Magento Forum

   
Model saving ridiculously slow? 
 
parkin_m
Jr. Member
 
Total Posts:  18
Joined:  2008-12-01
 

I have been doing some benchmarking on an import script I have been writing. The steps are:

1) open a text file
2) parse it into an array
3) create some simple products using models and save.

As an average I get around 0.39 seconds per product to load the model, add the data and then save (with some try/catch). 0.35 seconds of this time is the $model->save() method - which is clearly ridiculous! Magento is supposed to be enterprise level software - how on earth can it take 0.35 seconds to save a model?

I am now thinking about writing custom SQL to do the import - however if in future releases Magento is updated my scripts will break which is a bit of a pain.

Am I missing something or is there something really clever magento is doing under the hood which is time consuming but necessary? Because this is clearly why the import script is taking forever when loading from a CSV file.

Version is 1.17 by the way.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Discovery
Enthusiast
 
Total Posts:  767
Joined:  2007-12-29
 

I feel your pain. Computers are now thousands of times faster than they used to be, memory is nowadays measured in gigabytes instead of kilobytes, bandwidth is no longer given as ‘baud’, yet, time old tasks like adding some text to database tables takes forever and a day with Magento. Such simple tasks should happen in femto-seconds, if they don’t then something is going wrong.

Despite everything being Open Source it is not obvious why the slowness or what the hidden benefits of slowness are.

I shouldn’t grumble, perhaps I need to compile my own cart in 6502 assembler… Which reminds me - did you see the Javascript ZX Spectrum emulator? Slower than the 2Mhz 8 bit Z-80 of yesteryear, even on a dual-core Xeon. There’s a lot wrong with computers.

 
Magento Community Magento Community
Magento Community
Magento Community
 
LeeSaferite
Guru
 
Avatar
Total Posts:  322
Joined:  2007-08-31
Lake City, FL
 

It may be due to the fact that Magento is performing LOTS of INSERTS for each product saved.  The EAV system is highly flexible, but highly inefficient as well.  There is a reason they flattened some tables in the Mage_Sales module.  You really see the problem on adding new products becasue you are inserting ALL of the sttributes at once, versus an update where you may only change 2 or 3 attributes.  Profile it, look at what your DB is doing every time you add a product. 

I don’t think you could write an SQL statement that put an entire product into the DB in one fell swoop (using the EAV system that Magento currently uses). 

But I’ve been wrong before.  Someone, please prove me wrong.

 
Magento Community Magento Community
Magento Community
Magento Community
 
parkin_m
Jr. Member
 
Total Posts:  18
Joined:  2008-12-01
 

Hmm I understand that an EAV system is going to have overheads - however 0.35s is really unacceptable for a model save. 1100 products is taking around 7 minutes - god forbid I might need to import 50,000.

Is this something that the magento team is going to attempt to improve? Or is ‘enterprise’ software just a synonym for packed with features but bloated and slow?

 
Magento Community Magento Community
Magento Community
Magento Community
 
parkin_m
Jr. Member
 
Total Posts:  18
Joined:  2008-12-01
 

Is this not something that a team member can give a response on please? smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
parkin_m
Jr. Member
 
Total Posts:  18
Joined:  2008-12-01
 

*Bump

 
Magento Community Magento Community
Magento Community
Magento Community
 
millerav
Jr. Member
 
Total Posts:  2
Joined:  2009-02-10
 

0.35s is really unacceptable for a model save. 1100 products is taking around 7 minutes - god forbid I might need to import 50,000

.

Yup it’s painful. I’ve got those 50,000 products to import (with it keeping in sync the variables in the live store elsewhere) and the import times are killing me. So I’ve broken down and started writing it so my import script is completely separate from Magento (batch INSERT INTOs directly to the database). It’s totally inflexible but after I get the bugs out will be fast enough for me (so far testing seems to come in at about 150 products a second on a slow testing box).

The priority needs to be on getting the system working efficiently as that will drive adoption of the platform faster than anything else.

 
Magento Community Magento Community
Magento Community
Magento Community
 
blakew
Sr. Member
 
Total Posts:  107
Joined:  2008-06-20
 

I am right there with all of you… I can’t imagine Magento was made for web sites with this many products. I’m still fighting through this slow product creation time. I have another problem now though. Every time I blink Magento has changed the database. My $product->save() yields an undefined index when I try to create a product. This problem occurs in 1.2.1.2 but not in 1.2.0.3. I have not tested anything in between. Do any of you have any idea why I can’t create a product in 1.2.1.2?

Here is a thread I started to see if I can get some resolution:
http://www.magentocommerce.com/boards/viewthread/35310/

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
    Back to top