Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 4
Tutorial - Increase Magento’s perfomances without [edit] a lot of [/edit] core changes
 
asrenzo
Member
 
Avatar
Total Posts:  42
Joined:  2008-02-25
France
 

Here’s a simple tutorial on how to improve Magento’s performances without changing core files.
I’m sorry this tutorial won’t help users hosting their store on a shared environment.

Installation instructions will continue my fedora / centos install guide

1 - Choose your hardware
You should try to use hardware RAID 1/5 setup in order to increase I/O rates

2 - Split the load
You should try to run apache/lighttpd/ngnix and mysql server on different servers in order to split the load of each

3 - Use php-pecl-apc extension
This module will provide some code cache of php scripts.

Installation for fedora / centos

yum install php-pecl-apc

Edit module configuration file and enable this module extension by commenting out this line

vi /etc/php.d/apc.ini
extension apc.so

To fine tune apc performances, you may have to change other values in this default configuration file. See documentaion of APC for full configuration options.

4 - Memcache your sessions
First install a memcache server

yum install memcached

Configure memcached

vi /etc/sysconfig/memcached
And check for these lines
PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""

To add more cache size, change the CACHESIZE option

Launch memcached server

service memcached start

Add php memcached sessions extensions

yum install php-pecl-memcache

Check extension configuration matches your memcached setup

Enable memcache extension module
extension
=memcache.so

Options for the memcache module

Whether to transparently failover to other servers on errors
;memcache.allow_failover=1
Defines how many servers to try when setting and getting data.
;
memcache.max_failover_attempts=20
Data will be transferred in chunks of this size
;memcache.chunk_size=8192
The default TCP port number to use when connecting to the memcached server
;memcache.default_port=11211
Hash function {crc32fnv}
;memcache.hash_function=crc32
Hash strategy {standardconsistent}
;memcache.hash_strategy=standard

Options to use the memcache session handler

; Use memcache as a session handler
session
.save_handler=memcache
Defines a comma separated of server urls to use for session storage
session
.save_path="tcp://localhost:11211?persistent=1&weight;=1&timeout;=1&retry;_interval=15"

5 - Use a tmpfs filesystem for magento/var/cache
Supposing your magento leaves in /var/www/html/magento, we need to :
Stop apache

service httpd stop
Clean cache
cd /var/www/html/magento/var/cache/
rm -rf mage*
Mount a tmps cache
mount tmpfs /var/www/html/magento/var/cache -t tmpfs -o size=64m
For more space change 64m by any value (take care of your system RAM)

Restart apache

service httpd restart

6 - But not fully tested now is use mysql query cache
This need to be tested. If someone has any clue, please tell me.

7 - Some results
a) Basic setup no apc - memcached - tmpfs

ab -n 100 -c 5 http://magento.mydomain.com/

Server Software:        Apache/2.2.8
Server Port
:            80
...
Requests per second:    5.42 [#/sec] (mean)

b) APC setup no memcached - tmpfs

ab -n 100 -c 5 http://magento.mydomain.com/
Server Software:        Apache/2.2.8
Server Hostname
:        magento.yves-rocher.com
Server Port
:            80
...
Requests per second:    10.93 [#/sec] (mean)

c) APC - memcached setup no tmpfs

ab -n 100 -c 5 http://magento.mydomain.com/
Server Software:        Apache/2.2.8
Server Hostname
:        magento.yves-rocher.com
Server Port
:            80
...
Requests per second:    15.57 [#/sec] (mean)

d) Full setup

ab -n 100 -c 5 http://magento.mydomain.com/
Server Software:        Apache/2.2.8
Server Hostname
:        magento.yves-rocher.com
Server Port
:            80
...
Requests per second:    17.85 [#/sec] (mean)
 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial Web Host
Guru
 
Avatar
Total Posts:  364
Joined:  2007-11-08
Phoenix, AZ
 

Thank you for this excellent post.  I am certain a lot of people will find this handy.

Something worth mentioning is there is a known bug causing incompatibility with APC Cache and Zend Optimizer.  You can not run these two at the same time without seg faulting Apache.  In a shared hosting environment the ZO is likely going to have to take priority over APC Cache due to that shared hosting ‘configure to the lowest common denominator’ mentality.

However, from our testing we found no advantage to using APC over eAccelerator Cache, which IS fully compatible with ZO.  Many cases showed eAccelerator outperforming APC by 10% - 30%. 

Further, APC code optimizer has some known bugs of it’s own and is considered ‘experimental’, which is equivalent to ‘not ready for production’.  eAccellerator has the code optimizer ready to go, now.. It’s results are something else be be debated wink

One other thing that would be very handy and prove to enhance this Magento HowTo would be to implement the MySQL Query Cache configuration that I’ve posted at our blog - “Improving Magento Speed and Performance with MySQL Query Cache” . It’s a quick change that can provide massive improvement for most Magento installations.

I am intrigued by the memcachd use - I’ve not personally used this myself however I’ll be doing some research & experiments with this - thanks for the tip!

Great post and thanks again!

[edit] At first glance, memcached does appear to require code changes to place objects into the memcache.  Anyone more familiar with this that could provide additional information?

 
Magento Community Magento Community
Magento Community
Magento Community
 
asrenzo
Member
 
Avatar
Total Posts:  42
Joined:  2008-02-25
France
 

@Crucial Web Hosting

Thanks for your comments wink

I’ll try out other code cache systems you mentioned.

Concerning memcaching sessions, I can’t see any code rewriting need. In fact, I don’t want to cache anything else than sessions data.

Do you mean that “serialization” of data in memcached sessions could lead to data corruptions ?

The biggest advantage I see when using memcached session is that sessions become “networked”. Then you can easily create a cluster of frontends sharing the same mysql database and the same memcached server. Then sessions become available from any frontend and you can easily deal with load.

Please feel free to answer this post in order to share point of views and experience about system optimizations.

Regards,

Laurent

 
Magento Community Magento Community
Magento Community
Magento Community
 
adimagento
Guru
 
Avatar
Total Posts:  626
Joined:  2007-08-29
Los Angeles, CA
 

@asrenzo, it would be great if you could start a wiki with your post so the community can learn and contribute. Thank you for this great post.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial Web Host
Guru
 
Avatar
Total Posts:  364
Joined:  2007-11-08
Phoenix, AZ
 
asrenzo - 28 May 2008 10:46 AM

Concerning memcaching sessions, I can’t see any code rewriting need. In fact, I don’t want to cache anything else than sessions data.

I missed this part, my apologies. I am currently building an environment to do some testing with memcached.

Please feel free to answer this post in order to share point of views and experience about system optimizations.

You will get amazing results using mysql query_cache, this is a quick and painless configuration. 

We’ve also experienced far better disk results using iSCSI SAN storage.  This likely relates to low level ISCSI caching and similar disk accesses.

I am continuing to experiment with MPM’s in Apache as well and contrary to most of the advise I have seen here at Magento Prefork really seems like the way to go.  I hope to have a post on this in the next few days.

The problem is with some of these more exotic configurations is that strange, intermittent problems start arising, sometimes in non-related applications, and they are often times extremely difficult to troubleshoot.  I believe that a LOT of the problems reported on the boards are do to simple misconfiguration of servers.  Not so much bad configurations, just mixing technologies together that may or may not play very well together.  The APC Cache and Zend Optimizer incompatibility is a prime example.

Another issue is that a lot of times we see a configuration work VERY well - when one person is on the site.  However, benchmark the configuration with 100 concurrent visitors over 15 minutes and bad bad things start to happen.  smile

My only real advise - Test, test and retest.  YMMV smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
asrenzo
Member
 
Avatar
Total Posts:  42
Joined:  2008-02-25
France
 

@adimagento

For sure, I’ll be glad to add a wiki page about this topic. Am I allowed to do so with my current credentials ? Is there anything to do before ?

@Crucial Web Hosting

I’ll try the mysql query cache. My default configuration is to add these lines in mysqld definition in my.cnf

query-cache-type 1
query
-cache-size 20M

Is there anything else I should know ?

 
Magento Community Magento Community
Magento Community
Magento Community
 
Crucial Web Host
Guru
 
Avatar
Total Posts:  364
Joined:  2007-11-08
Phoenix, AZ
 

You don’t need any special permissions to edit the wiki, it’s there for all to add to and help grow.

Those cache settings are fine - It really is dependent on the use of your cache and how much memory you have available.  Though, I really dont see much need to get this too big with Magento - eventually, as with all caches you will hit the law of diminishing returns as the cache grows in size.  I’ve found 64MB to 128MB work very well for our setup.  This is dependent on a quite a few factors however.

Check out these paramenters too;

query_cache_min_res_unit
query_cache_limit

 
Magento Community Magento Community
Magento Community
Magento Community
 
asrenzo
Member
 
Avatar
Total Posts:  42
Joined:  2008-02-25
France
 

Preparing my wiki content, I realized I had forgotten something rather important in the memcached process.

In fact, I had to change some Magento core files to be able to use memcached sessions. These files are :

app/code/core/Mage/Core/Model/App.php
app
/code/core/Mage/Core/Model/Mysql4/Session.php

These files contains some code that forces session handler to be used

session_save_path(Mage::getBaseDir('session'));

and

session_module_name('files');

I can’t see any reason to force module and save_path expect being sure that a fresh install would run without problem.

@Moshe, if you are online:

- Is there any configuration option to set to bypass these redeclarations ? (I don’t think so)
- Is this a feature or a bug or a bugture ?
- Do we need to implement these class in a custom module to change default behaviour ?

Regards,

Laurent

 
Magento Community Magento Community
Magento Community
Magento Community
 
Michael_1
Enthusiast
 
Total Posts:  826
Joined:  2007-08-31
 

Hi Laurent,

There are no configuration options in the released version. In trunk we already have an option to save session data in the database (that might be used on a cluster setup). We might also add apc and memcached session storage before 1.1

Btw, you can try memcached or apc cache backends instead of tmpfs.

Thank you,
Michael.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

Hello folks,

I am getting poor and loosy

ab2 -n 100 -c 5 http://saintgeorges24.de/
Requests per second:    5.08 [#/sec] (mean)

with installed/running memcache(d), apc, tmpfs, mysql query cache

HW setup:
AMD Opteron DualCore 1212he (2,0Ghz)
2GB Ram
---
Basic SW setup
OpenSuse 10.2
PHP 5.2.5
mysql 5.0.26
Apache 2
Plesk 8.3.0
-> phpinfo @ [url=http://www.saintgeorges24.de/info.php]http://www.saintgeorges24.de/info.php[/url]
---
Magento Version
1.0.19870
---

---
Tmpfs

#df
returns
tmpfs                   131072     11488    119584   9% /srv/www/vhosts/saintgeorges24.de/httpdocs/magento/var
(Yes, i mounted the whole var directory as tmpfs)
---
APC
http://www.saintgeorges24.de/apc.php
---
Mysql Query Cache
mysqlSHOW STATUS LIKE 'Qcache%';
+-------------------------+-----------+
Variable_name           Value     |
+-------------------------+-----------+
Qcache_free_blocks      1         |
Qcache_free_memory      133631808 |
Qcache_hits             1035      |
Qcache_inserts          349       |
Qcache_lowmem_prunes    0         |
Qcache_not_cached       174       |
Qcache_queries_in_cache 349       |
Qcache_total_blocks     720       |
+-------------------------+-----------+
8 rows in set (0.00 sec)
---

WTF did i/ am i doing wrong? Maybe asrenzo could describe his modifications on session handling in

app/code/core/Mage/Core/Model/App.php
app
/code/core/Mage/Core/Model/Mysql4/Session.php
 
Magento Community Magento Community
Magento Community
Magento Community
 
Greven
Member
 
Total Posts:  36
Joined:  2007-08-31
Florence, AL
 

Does anyone have a Ngnix config file to make it work right to emulate the .htaccess file like Apache does for the mod_rewrite.  That is the biggest turn off for me and NgniX, the no .htaccess support.  I was able to make Lighttpd work but I think it is breaking my search, so I want to give NgniX a run.  But again, I am not sure how to do the mod_rewrite for it…

 
Magento Community Magento Community
Magento Community
Magento Community
 
Greven
Member
 
Total Posts:  36
Joined:  2007-08-31
Florence, AL
 

Much like Jan212, I’m not getting the large bump in speed like you are using APC and MEMCACHE.  I got/get better numers with XCache then I do APC + Memcache together.  With both of these I get about 2.93 RPS, with Xcache alone I get 5.93.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

@Greven - that’s not much smile, but good to know that is reproducable…

 
Magento Community Magento Community
Magento Community
Magento Community
 
Greven
Member
 
Total Posts:  36
Joined:  2007-08-31
Florence, AL
 

Yeah.  I’m on a 256 Slice at Slicehost, and I’m thinking about testing it on a 512 slice.  My current setup is Ligttpd + XCache + Fast PHP and I’m only getting 5 to 6 RPS......... down right horrible.

 
Magento Community Magento Community
Magento Community
Magento Community
 
jan212
Guru
 
Avatar
Total Posts:  407
Joined:  2008-01-03
 

that’s an optimal setup with crappy performance… i am sure we’ll find the bottlenecks the upcoming weeks… salsapepp is also working and testing around.

 
Magento Community Magento Community
Magento Community
Magento Community
 
bowers2000
Member
 
Total Posts:  31
Joined:  2008-05-21
 

Has anyone found the optimal values for these?

query_cache_limit = ???
query_cache_size = 64-128K
query_cache_type = 1
query_cache_min_res_unit = ???K

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 4