Enterprise Sales: +1.310.775.2674 (N. America) +44 20.3286.4137 (UK)
Performance is Key! - Notes on Magento’s Performance
Magento's performance is key.
Over the past few months, we've dedicated significant resources to diagnose and optimize Magento's performance. Now that Magento's 1.0 feature set is mostly complete, we are glad to share some of our work and the effort that has gone towards the goal of making Magento fast and furious. We are extremely happy with the results of our testing and benchmarks.
With such extensive features, functionality and flexibility performance may sometimes take a hit, and Magento is no exception to this rule. But there are performance gains in every incremental improvement, and we're starting to see significant results in the latest release. Below are a few of the performance improvements that appear in Magento's latest release.
- Top menu join queries rewritten, top menu block is cached now on per store view level for every active category
- Added catalog indexes for layered navigation and search
- Added dynamic merge and whitespace remove during JavaScript load
- Footer block is cached
- Implemented <remove name=""> directive in layout xml to avoid instantiating of unwanted block objects, skipped left/right column blocks generation for pages with 1-column template</remove>
- Most of sales order attributes became static and were moved to one raw table
- Quote, quote address and quote items entities are stored separately
- Greatly improved popular tags block generation time
- Quote optimization for using in top links block and in the sidebar
- Implemented caching API for using in Mage_Core_Model_Abstract descendant models
- Removed unnecessary queries from catalog pages and homepage (e.g. reduced total number of database queries from 700 to 60 for homepage, and from 500 to 100 on product listing page on a database of 1000 products and 300 categories).
From our performance and load tests we have found that the latest Magento release outperforms version 0.8.17240 by almost double. Below we show a graph of one of the load tests we ran using Apache Benchmark (ab) comparing the two Magento versions. We experienced similar results throughout our tests and comparisons of the two versions.
DYI (Do it Yourself) Performance Enhancements
While we continue to improve Magento's core performance, there are additional improvements that Magento users can implement to yield performance gains in a single-server environment:
- Install a PHP opcode cache such as APC or XCache. These can deliver significant improvements to PHP's responsiveness by caching PHP code in an intermediate bytecode format, which saves the interpreter from recompiling the PHP code for each and every request.
- Make sure your Apache configuration has KeepAlives enabled. KeepAlives are a trick where multiple HTTP requests can be funneled through a single TCP connection. Since the setup of each TCP connection incurs additional time, this can significantly reduce the time it takes to download all the files (HTML, JavaScript, images) for a website.
- Use a memory-based filesystem for Magento's var directory. Magento makes extensive use of file-based storage for caching and session storage. The slowest component in a server is the hard drive, so if you use a memory-based filesystem such as tmpfs, you can save all those extra disk IO cycles by storing these temporary files in memory instead of storing them on your slow hard drive.
- Modify the configuration for your MySQL server to take better advantage of your server's RAM. Most Linux distributions provide a conservative MySQL package out of the box to ensure it will run on a wide array of hardware configurations. If you have ample RAM (eg, 1gb or more), then you may want to try tweaking the configuration. An example my.cnf is below, though you will want to consult the MySQL documentation for a complete list of configuration directives and recommended settings.
Example my.cnf:
Time to Cluster?
As any successful online retailer knows, a growing business means growing pains. If your store is a huge success, then you'll undoubtedly require more horsepower to keep your customers happy. This means leaving the world of single-server websites and moving to a clustered environment with two or more servers.
There are varying ways to accomplish this, but a typical first step is to add more web servers and place them all behind a load balancer, which will distribute the load fairly among all web servers. This method is popular because (a) web servers are cheaper than database servers; and (b) there's really no limit to the amount of web servers you can add.
Note: One current limitation in Magento is that there is no official method to propagate catalog images to each web node. If you're not comfortable setting up some sort of facility to do this, then you may want to hire a professional or try a different method of scaling.
A second method involves a performance tactic called “database replication.” When using replication, you will have two or more database servers in your cluster. One will be the “master” and the rest will be called “slaves.” The master is the only database server that accepts any sort of write-based queries. These write queries are then replicated by each of the slaves in real-time (or very close to it). The advantage of this setup is that Magento can issue read queries to any of the slave servers, saving all the write queries for the master database. A database is traditionally a difficult component to scale horizontally, so replication is a happy compromise.
The initial setup and configuration for MySQL replication is out of the scope of this document, but we can refer you to the experts. Here is a tutorial that explains all the details.
Once you have MySQL replication functioning, configuring Magento to use multiple database servers is a relatively simple task. The only file you will have to edit is app/etc/local.xml. An example is shown below. As you can see, there are two database connections configured, one called default_setup and one called default_read. The default_setup connection will be used for all write-based queries, and the default_read connection will be used for all read-based queries.
Example app/etc/local.xml:
This article has only covered a few of the well-known tactics for improving site performance. However, every website's scenario is slightly different, so it's often worth hiring a professional to appraise the needs of your site and provide you with possible upgrade options. But if you're comfortable with Linux and confident when working with configuration files, try out a few of our suggestions above. You may be pleasantly surprised.
Have any ideas of your own? We'd love to hear your thoughts. Leave a comment or join the Magento Performance and Optimization Group.
1WebAddict from Chandler, AZ|posted March 20 2008
These performance improvements will make a big difference on a large site. I am looking forward to see how much better this can get.
2i960 from Bakersfield, CA|posted March 20 2008
Are there any tmpfs alternatives available for Windows? I am already using APC with great results and I have my MySQL configuration pretty well optimized. If I could also use something like tmpfs that would be great.
3mytechfactory from New York, NY|posted March 20 2008
I can see the performance difference on my site. Thanks a lot Varien.
4Steve Yan |posted March 20 2008
Great to see Magento is able to run in a clustered environment so easily. The read-only databases (slaves) offers significant speed improvements while improving load balancing.
I am now wondering about the viability of a single box solution. Magento make quite a number of round trips to the database per page. It’s 60 on the home page and 100 on the product listing. The contention between the webserver and database would lead one to speculate that there exists a good possibility of creating a bottleneck.
5Dylan |posted March 20 2008
I agree with Steve on this. I was a bit shocked to see how many calls are made to the database per page, as well as the amount of function calls that are made (upwards of 70k+ in 8.1 for some pages).
Hopefully page caching will help reduce this, but it would be cool if some of those select statements got cached themselves to avoid this many trips to the db.
I’m glad to see Varien has an interest in performance, keep up the good work guys, and thanks for keeping us informed.
6Raymond Brigleb |posted March 20 2008
Wow. Thanks a million for that helpful post. Excellent tips, and the performance is looking much improved.
7Ross from Scarborough, North Yorkshire, UK|posted March 20 2008
Thanks for taking the time to write up some performance info, the DIY tips will come in very handy!
I was just reading over on Yahoo Developer Network about some new ideas they have come up with for front-end performance improvements, which is also a good read (for theme developers).
http://developer.yahoo.net/blog/archives/2008/03/yahoos_latest_p.html
Maybe once you folk at Varien have finished optimising the back-end, you could also look at some front-end optimisation with the default theme? (Also, it would be great to have a minimalist core ‘interface’ as a base for skinning/theming - I find the default one quite heavy)
8YoavKutner |posted March 20 2008
@Ross - Thanks for the link. We have already started work on a lighter interface that will be an alternative to the default one. We should make it available right after the release of 1.0
Thanks
yoav
9mascker from Portugal|posted March 21 2008
Hi,
Wat about caching templates (at least the catalog)?
will certainly increase disk space, but i think every template could cached for about 3 minutes?
Don’t know if is a viable alternative, but…
Regards
10PhiLho |posted March 22 2008
Interesting. A quick test showed the previous version was a bit slow, it is nice to see you care and work on this problem. Looks like you know what you are doing…
11oshipper from New York, NY|posted March 26 2008
Looking good but there are still a number of major performance issues to work out before a 1.0. release. Hope this lighter interface contains a much lighter and less complicated checkout process. Will be reporting some bugs and looking forward to putting the next version through it’s paces right away.
12wflorian from Germany|posted April 2 2008
Even if there are performance improvements, I still have major performance problems on my testserver. I really hope that there will be more big performance improvements coming soon.
13Max_Schreck from Czech Republic, Decin|posted April 3 2008
It’d be good to mention MySQL Query Cache in this article. http://dev.mysql.com/doc/refman/5.0/en/query-cache.html For example it can speed up the response time from 5 sec to 1 sec in some cases.
14Max_Schreck from Czech Republic, Decin|posted April 3 2008
I’m sorry for the previous post… it’s there.
15MagKen |posted April 14 2008
Just installed on my vps and would agree that ver 1 appears to be to slow to use on a live site. I have to point out however that I am not a techy user. Installation was very smooth, I ran the server check and the installer prompted me to change file permissions, no complaints there.
Logging in to Admin takes about 6 seconds, loading the shop home page with no sample products takes about 5 seconds.
I believe my vps has minimum 256 mb ram and software such as Expression Engine and LiteCommerce run well.
16YoavKutner |posted April 14 2008
@MagKen - What host are you using?
admin panel is slow to load due to the dashboard. We are improving the dashboard and will make it available soon.
There is also a fix that was published on the boards here: http://www.magentocommerce.com/boards/viewthread/5366/P15/#t19853
This fix is helps with the caching of the JavaScript.
Thanks
yoav
17MagKen |posted April 15 2008
Hi Yoav, I am with Clook in the UK. We have one of their vps hosting packages. I tried the fix you mentioned but it did not seem to make any difference, so I tried reinstalling again but with no improvements.
18MagKen |posted April 17 2008
Just been comparing load times of my existing store with a fresh magento installation using the Firebug plugin. Even though the 39 requests for my store took 2.67s and 16 requests for the magento page took 1.3s my page definitely refreshed much quicker in the browser window (Safari and Firefox).
19mascker from Portugal|posted April 17 2008
Hi,
Don’t know if this makes sense, but as far as I know, the system loads every time the etc/ folders and parses the XML files.
This is necessary for process new modules.
Here is the idea. Why not cache all XML files for 30 days, and when we want to add a new module we go to the admin section and we empty the cache, forcing Magento to process the XML files again.
Regards
20webfxmasta |posted April 25 2008
After installing XCache or APC, do I have to modify the magento source to utilize apc_compile_file or xcache_get/set? Or is the opcode caching automatically working after installation (and apache restart)?
21jan212 |posted May 4 2008
@magentoteam - You are talking about Xcache in this article and say that you’re only supporting APC atm in the wiki installation guide… What’s correct?
22oshipper from New York, NY|posted May 18 2008
Any news on if new versions have addressed these concerns?
23jayhlee |posted July 2 2008
When is the new version of Magento slated to come out?
24i960 from Bakersfield, CA|posted July 2 2008
@jayhlii - They mentioned this in their latest blog on 1.1: “Magento 1.1 stable release is scheduled for mid to late July and more information will be made available shortly. “
25SimpleHelix.com from Huntsville, AL|posted July 19 2008
sometimes, choosing the right web host makes all the difference.
take a look at http://www.magentify.com for instance, speed is really not an issue anymore.
26nicksh |posted July 26 2008
I’ve considered moving from osCommerce to Magento, but performance has been a real concern. The eCommerce websites that I look after are quite heavily trafficed (i.e. 50-100 users online at all times). osCommerce performs very well, but I wonder how well Magento would perform under these loads and it’s running in a shared hosting environment. Can the Magento team publish some sort of benchmarks and recommendations with different performance test scenarios? Obviously, its best to have a dedicated server of a VPS, but the cost is quite prohibitive, and I see this as the main hurdle for takeup of magento. It would be very good to have some sort of guidelines for the hosting environment. Even having detailed benchmarking, would really help to determine the required setup on a case by case basis.
27viovao from Denmark|posted August 7 2008
It helped! Thanks great job!
28AldoLinares |posted August 14 2008
@SimpleHelix.com
No Offense but you do commercials on every post about speed, to much just look bad…
29Ata Asgari from Dubai, UAE|posted August 17 2008
Wow! I’ve achieved a great performance on my Intel Atom (1.6Ghz + 2GB Ram + Sata HDD) test server running CentOS 5.0, Apache 2.2.8, Mysql 5.0.58 and Xcache 1.2.2.
Comparing to default setting for apache and mysql it loads 8x faster!
What I’ve changed :
- Apache Config
- Mysql Config
- Installed Xcache
- Moved var directory to Ramdisk using tmpfs
- Moved /tmp directory to Ramdisk using tmpfs
Thanks!
30webjay |posted August 20 2008
Wayback I hacked a lot in Postfix and wonder if some of these tips could be used?
http://web.archive.org/web/20070308153119/http://www.arschkrebs.de/postfix/postfix_ext3.shtml
31wojtekatbyte from Amsterdam, NL|posted September 9 2008
"Note: One current limitation in Magento is that there is no official method to propagate catalog images to each web node. If you’re not comfortable setting up some sort of facility to do this, then you may want to hire a professional or try a different method of scaling. “
Does this mean that the this is the only read/write directory in Magento installation?
Suppose I have a webcluster and magento is installed on every server’s localdisk.
Is it then enough to rsync the media/ directory?
Thanks!
32YoavKutner |posted September 10 2008
@wojtekatbyte - First thing is that you can use rsync to sync the media folder (no other folder has files uploaded to it by default). The second way and something that is built into Magento is to specify one host that will serve images. This can be done by configuring the ‘Base Media URL’ both in the secure and unsecure web configurations. this will allow you to serve images from one (image) server with out the need to sync any folders.
Thanks
yoav
33azolia |posted September 25 2008
Hi,
As many people, after having been wondered by Magento look and features, I’d been frustrated by the loading time.
I then have decided to make some server tuning to improve page loading performances.
I’ve first installed xcache, used tmpfs for cached page (mysql query cache was already in process).
Loading time, for all the demo store pages, has been clearly cut .
Having full access to apache configuration files, I’ve removed .htaccess files rules from directories to virtualhost configuration directives.
This change has improved loading speed as much as xcache installation. Further more, it has divided loading time by 2 for the smallest server I’ve used for test.
Francois
http://www.azolia.fr
34magento-man |posted November 5 2008
i think there are still some problems with overloads! I was trying to adjust a test free TemplateMonster magento theme which they are releasing soon (i hope so) http://www.templatemonster.com/magento-themes.php and my server crushed, the only thing I was able to do is to adjust it manually I dont know who’s fault is it TM or Magento but ...
35jim_keller |posted December 11 2008
Might it be reasonable to start implementing stored procedures for some of the more intensive queries? mySQL 5 is widely used now, and for backward compatibility, the object factories could load the appropriate code based on the mySQL version
36henriv |posted March 24 2009
I have installed Magento on Linux / Windows and done all the optimizations that are laid out here but still the CPU for apache or iis spikes at 100% and pages only complete in around 30s. Any indicators to how to solve this?
37Ross from Scarborough, North Yorkshire, UK|posted March 24 2009
@henriv - that doesn’t sound right, I think there must be a problem with your system. If you are interested in getting help, probably best to ask in the forums
38henriv |posted March 24 2009
Ross thanks. I posted a comment on the forum http://www.magentocommerce.com/boards/viewthread/36459/ to get some more help. The different OSs were tested on different machines using basic configuration.
39dr.evil |posted April 3 2009
Shouldn’t key_buffer be key_buffer_size. key_buffer does not exist in MySql Reference Manual. See, http://dev.mysql.com/doc/refman/5.0/en/server-system-variables.html#sysvar_key_buffer_size
40Euklid from Denmark|posted 1 month ago
Adding memcache & varnished to the LAMP Stack indeed boost performance bigtime...!
41jarnail123 |posted 1 week ago
Hi Everybody,
i am new to magento, i have problem in magento 1.3. i installed it on my localhost.i installed succsfully. but when i click on login or register or any other link it shows cookies not enable error. although my browser cookies are enable.
Please guide me how i can resolve this problem
Thanks In Avdance
Amit
42Phil Freo from Jacksonville, FL|posted 1 day ago
Magento team:
It’d be really helpful if there was an official wiki page or set of pages about Magento optimization. Right now I’ve discovered a lot of useful tips but they’ve all been scattered throughout multiple forums threads. It seems like there are some basic things that aren’t documented online anywhere officially.
For example, how to configure Magento to take advantage of APC, if installed. I found a solution in the forum about editing local.xml but I still do not see this documented online anywhere outside of the forums. Having a trusted and compiled resource for stuff like this would be extremely useful.