Ok. Figuring out how to get this puppy purring over the last few days.
Thought I would post a ‘How To Speed up Magento’ Guide based on what I have found out. And maybe others can chip in afterwards with their top tips, and we can turn this into a one-stop thread to getting Magento installs working nice and fast
1/ Install Fooman Speedster
This one made a massive difference to page loads for the first site I am testing.
Looking at benchmarks and folks comments, it seems to be a toss of coin which one to go for.
I will probably go for APC on balance.
Will be asking my host to install this.
Ok. My host installed it. And further to Discovery posting below, thought I would add that in here to the main body. Discovery pointed out that thread here had extra tips once APC had been installed: http://www.magentocommerce.com/boards/viewthread/9882/
Essentially, after APC has been installed, you locate the local.xml file in the install. (Mine was here - /home/yoursitename/public_html/app/etc/local.xml
I have to say that I actually have installed APC, and then tried this, but that it actually slowed down my server. Response times went up from about 2 seconds to 3 seconds.
Your mileage may however vary, so give it a go
Edit to My Edit.... Turns out I didn’t have APC installed properly. Thought it was odd that I had no speed improvements when everyone else seemed to, so I went hunting for the reason.
Once you have a running server, you should copy the apc.php script that comes with the extension to somewhere in your docroot and load it up in your browser. It provides you with a detailed look at what is happening in your cache. If you have GD enabled in PHP, it will even have pretty graphs.
then it becomes clearer whether it is working correctly)
4/ “Make sure your Apache configuration has KeepAlives enabled. KeepAlives are a trick where multiple HTTP requests can be funneled through a single TCP connection. Since the setup of each TCP connection incurs additional time, this can significantly reduce the time it takes to download all the files (HTML, JavaScript, images) for a website.”
No idea what this is. And there is no further mention of it anywhere. So I will ask my host to enable it. Would be good to have some more info on this, if anyone knows what it is?
5/"Modify the configuration for your MySQL server to take better advantage of your server’s RAM. Most Linux distributions provide a conservative MySQL package out of the box to ensure it will run on a wide array of hardware configurations. If you have ample RAM (eg, 1gb or more), then you may want to try tweaking the configuration. An example my.cnf is below, though you will want to consult the MySQL documentation for a complete list of configuration directives and recommended settings.”
They have a list of settings there for my.cnf as follows:
Again, get your host to do it, (which is what I will be doing ) or figure it out.
6/ “Use a memory-based filesystem for Magento’s var directory. Magento makes extensive use of file-based storage for caching and session storage. The slowest component in a server is the hard drive, so if you use a memory-based filesystem such as tmpfs, you can save all those extra disk IO cycles by storing these temporary files in memory instead of storing them on your slow hard drive.”
Ok. Rather unhelpfully, there is no mention anywhere of how to do this.
So, if someone could post a step by step on this, then that would be useful.
a/ First off, I made some changes to your system, I increased the APC.shm.size to 128 This will allow more data to be cached by apc, in theory it’ll store more data.
(This sounds like it is linked to APC above - So maybe install APC first before doing this )
(Continued in next post because I ran out of characters… )
“Hi,
As many people, after having been wondered by Magento look and features, I’d been frustrated by the loading time.
I then have decided to make some server tuning to improve page loading performances.
I’ve first installed xcache, used tmpfs for cached page (mysql query cache was already in process).
Loading time, for all the demo store pages, has been clearly cut .
Having full access to apache configuration files, I’ve removed .htaccess files rules from directories to virtualhost configuration directives.
This change has improved loading speed as much as xcache installation. Further more, it has divided loading time by 2 for the smallest server I’ve used for test.
This comes under the ‘get more servers’, ‘better equipment’ banner; which is of course the obvious way to improve speed
And also this one…
11/ Optimizing MySQL Database
After all products and categories have been imported into the MySQL Database then go into webhost manager and select PhpMyAdmin at the bottom. (I have cpanel on my panel
Select the relevant database and first repair all the tables, and then optimise them.
Goto Magento Admin -> System -> Cache Management
- Tick the following items:
-- Configuration
-- Layouts
-- Block HTML output
-- Translations
-Untick the following items: (if ticked they seem to have a negative impact atm)
-- Collections Data
-- EAV types and attributes
-- Web Services Configuration
Goto Magento Admin -> CMS -> Manage Pages -> Home -> Custom Design
- The “Layout Update XML” box is pre-populated with functions and calls, all of which can be removed without any apparent problems, according to a number of forums comments.
Introduce a new Apache AddHandler to compress CSS files on the fly. I came across this technique eleswhere on the Internet, but my site pages are now flying as a result.
- Create a file called “gzip-css.php” and add/save the following into it:
- Locate your template CSS directory and upload both files that you created to it.
cd skin/frontend/default/YOUR_TEMPLATE/css/
PS I actually tried this (the post above has someone saying that there style sheets didn’t work afterwards), and it worked fine for me.
One suggested change for those it didn’t work for in that thread was to change:
AddHandler application/x-httpd-php .css to AddHandler application/x-httpd-php5 .css
However, as I said, the original version worked fine for me.
15/ Enable GZip Compression in the .htaccess (Part 2)
By deafult the .htaccess has this:
############################################ ## enable apache served files compression ## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter #SetOutputFilter DEFLATE
# Netscape 4.x has some problems... #BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems #BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine #BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # Don't compress images #SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content #Header append Vary User-Agent env=!dont-vary
</IfModule>
Change to:
############################################ ## enable apache served files compression ## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter SetOutputFilter DEFLATE
# Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # Don't compress images SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content Header append Vary User-Agent env=!dont-vary
Most guides I have found on the net as to how to do this give the impression that you need Apache2? Not sure. I am checking with my webhost, and will update this when I know.
To the usr/local/apache/conf/httpd.conf file. and then re-compiling Apche (you do this in WHM - If you have a cpanel install - on the far left where it says ‘Restart services’ - You just click on HTTP Server (Apache)) - But mine failed. So I had to remove the line, upload it again (ie the same as before) and it worked fine.
The line didn’t seem to be in the httpd.conf file anywhere though. Anyhow, will update when I know
Update: My host has got back to me and mod_deflate does indeed require Apache2 to run.
Beyond my skills to install this, but my hosts are excellent, and have made no problem of doing it for me.
--- Hmmm, Just looking at load times. They were previously 1.5 to 2 seconds; and after changing to apache2 they are 1.5 to 3 seconds. I reckon its added at least half a second (and often a second) to load times. So, on my server at least, this may be a duffer and I will be rolling back to Apache 1.
Ok. Decided to try Apache 2.2.
So, I have now tried Apache 1.3 (no mod drflate), Apache 2.0.63 (with and without mod_deflate) and 2.2.1.2 without mod_deflate.
Honestly, I would have to say that 1.3 without mod_deflate was the quickest. Apache 2.0.63 with mod_deflate the slowest.
I have settled on Apache 2.2.1.2 without mod_deflate, because whilst I think it is actually marginally slower then 1.3 (as measured by yslow - http://developer.yahoo.com/yslow/) it is at least being updated with new features. And the speed difference is not large in my view.
This has a tip in it to add the following to your php.ini file (mind is found here - /usr/local/lib if you are trying to locate it :-))
realpath_cache_size=1M (careful, this is per apache process) realpath_cache_ttl=86400 (ok for production site)
I tried this, and I have to say that the site definitely feels faster. The yslow numbers are not much different, but I would say that it because it doesn’t seem to have speeded up the loading up of the images on the page (from what I can see). The actual html page and text are definitely appearing faster.
Note: Regarding the replace_recursive.php file that he also discuesses in that thread… I had a look at my own install (Located here - /home/cpanelsitename/public_html/lib/Zend), and it seems that it is already in there, so I assume that this is now part of the core magento install
I was wondering if somebody know how to put the files data.php and exception.php in the cache which would increase the performance a lot.
Not sure whether or not this is a clue to someone that could increase performance?
Personally, I have no idea what he is taking about, so you will have to make up your own mind :-) If you do know. And it makes sense, then post how to do it below.
18/ Changes to MySQL Configuration
Ok. Right. My server flipped out this morning and had some speed problems after Apache crashed, so I had to get in touch with my host (who have been excellent) about speeding the site back up.
Two things that they recommended: (This is my server guys words)
“changing the temporary directory for eaccelerator to /home/eaccelerator because it looked like it may have been having space issues on /tmp.”
And the following changes to the MySql Configuration:
“Increase max_heap_table_size and tmp_table_size (I would probably try doubling each to 32M and 256M, respectively) Doubling the table_cache to 2048 Tables Enabling log-queries-not-using-indexes so we can see if there are performance gains to be had there and increasing the query_cache size to 256M
This may decrease the time spent waiting for the mysql requests, but at the cost of more memory.”
19/ Change from Apache to Litespeed Web Servers
Several posts on here make reference to changing from Apache to Litespeed as a way of improving the speed of Magento.
# To enable one or both first go to your admin panel and navigate to System->Cache Management. Under Catalog click on the Rebuild button next to Rebuild Flat Catalog Category or Product.
# Navigate to System->Configuration click on Catalog and select the Frontend tab. Choose Yes next to the appropriate selection of either Use Flat Catalog Category or Product.
Ok. There you go guys.
20 Ways that I have found so far to potentially speed up a Magento install.
Please post any new ways that you find below so that we can turn this into a useful repository of tips for everyone
There is more power in sharing, so share any speed up tips you have here
1) That extension breaks things, besides it just compresses files differently to how they would be compressed anyhows. You can delete the comments from your js and css to optimise it all by yourself and let the computer compress/cache as per normal.
4) You probably have keepalives already. Put a file with <?php phpinfo(); ?> in it, save to /var/www/html/phpinfo.php then call http://www.host.com/phpinfo.php and see what you get. Keepalives will probably be in there somewhere.
When you have tmpfs working you can do du -sm /var/www/html/var/* to see what space is actually used. If you need to cut it down then you can by re-doing the tmpfs drive mount to be - say - 32M. Or you can set a drive up on tmpfs and check it later with df.
Also remember to add to /ect/fstab:
That conveniently mounts the drive at boot, greedy 64Mb size, so you don’t have to forget
9) Putting the htaccess stuff in the main httpd.conf is a huge tip - thanks for that. Mine is vastly quicker now, My demo store furniture page loads in less than two seconds, Zolof T-shirt in 3.3. I was aiming for 4 seconds, I can live with 3…
With the htaccess out the way and the directives in a .conf include file you get a couple of warnings - not everything in there was that good and since it gets loaded every time you have to wonder. A typical message: [warn] module php5_module is already loaded, skipping - it is better if that gets warned of once, not every time a file is accessed…
A big tip is to not have a hosting company… A shared server setup has the db fs and www servers on different machines that work no better than blu-tak and string.
Go for a virtual server at the minimum, something you can properly log onto. Never bother with anything more than a blog on a shared hosting setup. The pain is not worth it. Same goes for anything Windows… Also you should be command prompt all the way, not CPanel.
Despite the 4-5 hours compiling-searching fro rpms-installing and testing my SOAP API is only running 20% quicker than it was earlier.
9) Putting the htaccess stuff in the main httpd.conf is a huge tip - thanks for that. Mine is vastly quicker now, My demo store furniture page loads in less than two seconds, Zolof T-shirt in 3.3. I was aiming for 4 seconds, I can live with 3…
With the htaccess out the way and the directives in a .conf include file you get a couple of warnings - not everything in there was that good and since it gets loaded every time you have to wonder. A typical message: [warn] module php5_module is already loaded, skipping - it is better if that gets warned of once, not every time a file is accessed…
Discovery, (or anyone else who knows) - Can you explain exactly what you did with this as a step by step?
I know that I put the tip in
That’s because I could clearly see that it had increased someones performance with Magento. But I still didn’t really understand what Francoise had actually done, and I didn’t find mention of it anywhere else on here.
Now that two people have tried it and got such dramatic results, it REALLY makes me want to give it a go
As ‘discovery’ has experimented it, apache server is faster when .htaccess files are disallowed.
Consequently, Magento shop is responding faster too.
A french user, ryu007, wrote me months ago, that he has successfully speeded up his magento shop doing this.
Here is the explanation (see Apache tutorial) :
In general, you should never use .htaccess files unless you don’t have access to the main server configuration file. There is, for example, a prevailing misconception that user authentication should always be done in .htaccess files. This is simply not the case. You can put user authentication configurations in the main server configuration, and this is, in fact, the preferred way to do things.
.htaccess files should be used in a case where the content providers need to make configuration changes to the server on a per-directory basis, but do not have root access on the server system. In the event that the server administrator is not willing to make frequent configuration changes, it might be desirable to permit individual users to make these changes in .htaccess files for themselves. This is particularly true, for example, in cases where ISPs are hosting multiple user sites on a single machine, and want their users to be able to alter their configuration.
However, in general, use of .htaccess files should be avoided when possible. Any configuration that you would consider putting in a .htaccess file, can just as effectively be made in a <Directory> section in your main server configuration file.
There are two main reasons to avoid the use of .htaccess files.
The first of these is performance. When AllowOverride is set to allow the use of .htaccess files, Apache will look in every directory for .htaccess files. Thus, permitting .htaccess files causes a performance hit, whether or not you actually even use them! Also, the .htaccess file is loaded every time a document is requested.
Further note that Apache must look for .htaccess files in all higher-level directories, in order to have a full complement of directives that it must apply.
Most of Web applications sources available for download are including .htaccess files. This way, they can be hosted everywhere (if there is no special directive in those files).
If you’ve bought or hired a dedicated server, you should rewrite .htaccess directives in a main apache configuration file.
Here is an example of what could be VirtualHost directives for a Magento shop :
<Directory /var/www/magento> Order allow,deny Allow from all DirectoryIndex index.php AllowOverride None Options +FollowSymLinks AddDefaultCharset Off
php_value memory_limit 64M php_value max_execution_time 300 php_flag magic_quotes_gpc off php_flag session.auto_start off php_flag suhosin.session.cryptua off php_flag zend.ze1_compatibility_mode Off php_flag zlib.output_compression on
<Directory /var/www/magento/var > <Files config.xml> Deny from all </Files> </Directory>
</VirtualHost>
The limitation is that you should be apache administrator (root on a dedicated server) to do that.
It’s impossible if you are running Magento on a shared hosting server.
Removing .htaccess file rules from directories and putting them in httpd.conf
At Your own risk…
For the brave…
Oh Whatever. Just know that servers can, and do explode when you mess with them
In all seriousness, backup httpd.conf etc BEFORE you mess with it. In that way its easy to revert back if you mess this up (as I did a couple of times before figuring out what worked)
1/ Find the location of your httpd.conf file
If you look around the web then most places say it lives here:
/etc/httpd/conf/httpd.conf
However, in my own servers case it lives here:
/usr/local/apache/conf/http.conf
So you will have to hunt around and figure out where it is.
2/ Once you have that then download it to your computer and open up in notepad (I just did this using ftp, but there is probably a better way to do it with ssh)
3/ If you scroll through it you will see that it contains all the details of your different sites on your server, as well as different stuff that the script is automatically calling at the beginning.
You will ‘know’ which ones are which, because as you scroll through you will see something along the lines of:
To put this tip into operation scroll down to the site where the magento install is:
You will see two lines below it:
# To customize this VirtualHost use an include file at the following location # Include "/usr/local/apache/conf/userdata/std/2/yourcpanelname/mydomain.com/*.conf"
(Naturally the exact locations are going to depend on your installations, so the exact path will probably be different in your httpd.conf - But you get the idea)
Change that to: (ie uncomment it by removing the # and create a name for the new .conf we are going to create)
# To customize this VirtualHost use an include file at the following location Include "/usr/local/apache/conf/userdata/std/2/yourcpanelname/mydomain.com/myhtaccess.conf"
4/ Save that file (Note - make sure to save it as httpd.conf and not anything dumb like .txt)
5/ Open up a blank new notepad and paste in the code that Francois rather handily provided as a starting point (you are going to want to put in stuff from your own .htaccess).
<Directory /var/www/magento> Order allow,deny Allow from all DirectoryIndex index.php AllowOverride None Options +FollowSymLinks AddDefaultCharset Off
php_value memory_limit 64M php_value max_execution_time 300 php_flag magic_quotes_gpc off php_flag session.auto_start off php_flag suhosin.session.cryptua off php_flag zend.ze1_compatibility_mode Off php_flag zlib.output_compression on
Here is my original .htaccess so you can see what this is about.
############################################ ## uncomment these lines for CGI mode ## make sure to specify the correct cgi php binary file name ## it might be /cgi-bin/php-cgi
############################################ ## disable magic quotes for php request vars
php_flag magic_quotes_gpc off
############################################ ## disable automatic session start ## before autoload was initialized
php_flag session.auto_start off
############################################ ## enable resulting html compression
php_flag zlib.output_compression on
########################################### # disable user agent verification to not break multiple image upload
php_flag suhosin.session.cryptua off
########################################### # turn off compatibility with PHP4 when dealing with objects
php_flag zend.ze1_compatibility_mode Off
</IfModule>
<IfModule mod_security.c> ########################################### # disable POST processing to not break multiple image upload
SecFilterEngine Off SecFilterScanPOST Off </IfModule>
<IfModule mod_deflate.c>
############################################ ## enable apache served files compression ## http://developer.yahoo.com/performance/rules.html#gzip
# Insert filter SetOutputFilter DEFLATE
# Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # Don't compress images SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content Header append Vary User-Agent env=!dont-vary
</IfModule>
<IfModule mod_ssl.c>
############################################ ## make HTTPS env vars available for CGI mode
############################################ ## rewrite everything else to index.php
RewriteRule .* index.php [L]
</IfModule>
############################################ ## Prevent character encoding issues from server overrides ## If you still have problems, use the second line instead
############################################ ## By default allow all access
Order allow,deny Allow from all
############################################ ## If running in cluster environment, uncomment this ## http://developer.yahoo.com/performance/rules.html#etags
#FileETag none
Ok. So I am now going to transcibe those elements in my .htaccess (that include the changes above eg Fooman Speedster etc) and using Francois structure as a template.
( Directory will be the same as DocumentRoot eg /home/yourcpanelname/public_html)
I end up with the following:
<Directory /home/mydomain/public_html> Order allow,deny Allow from all DirectoryIndex index.php AllowOverride None Options +FollowSymLinks AddDefaultCharset Off
<IfModule mod_php5.c>
php_value memory_limit 128M php_value max_execution_time 18000 php_flag magic_quotes_gpc off php_flag session.auto_start off php_flag suhosin.session.cryptua off php_flag zend.ze1_compatibility_mode Off php_flag zlib.output_compression on
6/ Save that as myhtaccess.conf (naturally use your own domain path, and you would also have to install fooman speedster if the - RewriteRule ^(index.php/)?minify/([^/]+)(/.*.(js|css))$ lib/minify/m.php?f=$3&d;=$2 - line in the code was going to be of any use to you)
(you can call this whatever you like, but be sure to change it in the httpd.conf)
7/ The httpd.conf handily tells you exactly where you need to upload the new myhtaccess.conf file to:
Problem: User wants to add something to a virtualhost.
Solution: It is possible to create a custom virtualhost template and have it used
instead of the default virtualhost templates. You’d need to add a line to the
virtualhost datastore telling it the location of the template. Usually minor
configuration changes can be most simply made via the user’s .htaccess files or
included into httpd.conf under /usr/local/apache/conf/userdata. The
include directory structure is not created automatically, but use the following file
structure:
/usr/local/apache/conf/userdata/(ssl|std)/(1|2)/<user>
/<domain>/<something>.conf - Individual VirtualHost
/usr/local/apache/conf/userdata/<something>.conf - All VirtualHost
containers
/usr/local/apache/conf/userdata/[ssl or
std]/<something>.conf - All VirtualHost containers for SSL or
standard VirtualHosts
/usr/local/apache/conf/userdata/[ssl or std]/[1 or
2]/<something>.conf - All VirtualHost containers for SSL or
standard VirtualHosts with version specific settings
/usr/local/apache/conf/userdata/[ssl or std]/[1 or
2]/<user>/<something>.conf - All of a users VirtualHost
containers for SSL or standard VirtualHosts with version specific settings
/usr/local/apache/conf/userdata/[ssl or std]/[1 or
2]/<user>/<domain>/<something>.conf - Individual VirtualHost
container for SSL or standard VirtualHosts with version specific settings
Includes are located as the final directive in the VirtualHost container. Default
values can usually be overridden via the include files. Include files beginning with
“cp_” are reserved for cPanel settings and can and will likely be automatically
rewritten. After adding any new include files, the required VirtualHost include
statements can be ensured by running /scripts/ensure_vhost_includes
with the proper arguments.
Ok. So basically, I just created the rest of the structure in my ftp client, and set each of the new folders to 777. I also set the myhtaccess.conf to 777
8/ I then uploaded the altered httpd.conf to /usr/local/apache/conf/http.conf (which is where it was on my server)
(Remember I have also uploaded /usr/local/apache/conf/userdata/std/2/yourcpanelname/mydomain.com/myhtaccess.conf)
Note: I did also try just pasteing the contents of myhtaccess.conf directly into the httpd.conf; to see if it was faster, but in my tests it made no difference, so I left it using the include setup, as that seemed to be neater)
9/ Apache then needs to be restarted for the changes to take effect.
On a cpanel server you do this by logging into WHM. On the left is a section called ‘Restart Services’. If you click on ‘HTTP Server (Apache)’ then that will restart the Apache service.
10/ You can now delete your original .htaccess (assuming everything worked)
Conclusion:
This wasn’t the magical speed transformation for me, that it has been for others.
I would say 2/10s of a second faster.
My pages load in the 1.5 - 3 second range. Most around 2. This probably dropped the average to about 2 seconds.
Magento is just plain slower then solutions like creloaded, which in my tests often load in about 1 second.
Magento is useable, but really could do with being faster
When you consider the amount of speed improvements I have tested out it seems that about 1.5 seconds is about as quick as it gets on Magento. And even that is uncommon. 2 is more normal. Whilst oscommerce solutions are considerably faster.
Hopefully the Magento guys realise that bells and whistles are not enough. And that they need to find ways of improving the core code of this to speed it up, as even with all these tweaks, it is still considerably slower then other cart solutions.
Thought I would bump this one up to the top, as I have added another 3 methods I have found into the guide above (making 18 so far that I have found) plus a couple of others from Discovery, and a good clarifying post from Francois.
Feel free to add new ones, or clarifications to the others
Turns out I didn’t have APC installed properly. Thought it was odd that I had no speed improvements when everyone else seemed to, so I went hunting for the reason.
