Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Magento 1.0 Performance Issues on high traffic shops
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Hi everyone,

We have been developping a Magento based shop for one of our main customers. We put it online last week and we were forced to remove it because of performance issues.

The server
- Processor : Xeon 3000@2.66Ghz
- Ram : 4Gb
- Hard disks : Raid0 @ 10K rpm

The shop
- about 15,000 products
- 15,000 unique visitors per day (and 30,000 on peaks)

Configuration
- you can have a look at the phpinfo() page here : PhpInfo
- Apache/2.2.3 (Debian) PHP/5.2.0-8+etch11
- APC

Benchmark

For only 30 concurrent requests on the home page we get around 10 seconds results and we reach 30 seconds on quick search:

ns27118:~# ab -c 30 -n 30 http://123.123.123.123/index.php

Document Path:          /index.php
Document Length
:        28107 bytes

Concurrency Level
:      30
Time taken 
for tests:   10.256282 seconds
Complete requests
:      30
Failed requests
:        0
Write errors
:           0
Total transferred
:      859470 bytes
HTML transferred
:       843210 bytes
Requests per second
:    2.93 [#/sec] (mean)
Time per request:       10256.282 [ms] (mean)
Time per request:       341.876 [ms] (meanacross all concurrent requests)
Transfer rate:          81.80 [Kbytes/sec] received

Connection Times 
(ms)
              
min  mean[+/-sd] median   max
Connect
:        8    8   0.7      9       9
Processing
:  9181 9885 270.5   9910   10248
Waiting
:     8944 9687 346.9   9795   10135
Total
:       9190 9894 270.6   9918   10256

So I now have two questions :

- Browsing the forums, I’ve seen that Magento 1.1 has really a performance increase. Will it make my site browseable and is this really complicated to convert a 1.0 setup to a 1.1 one ? (We have made some specific developments) ?
- Will it be really better if we setup a new server (same one) hosting only the database ?

Thanks for your help !

 
Magento Community Magento Community
Magento Community
Magento Community
 
J_T_
Mentor
 
Avatar
Total Posts:  1961
Joined:  2008-08-07
London-ish, UK
 

If you made your modifications as stand-alone Modules, it’ll be easy to upgrade. If you hardcoded them into the core files, you have it set up to cause you a headache every time you upgrade in the future.

You will have to upgrade, especially with those volumes, as < 1.1.4 contains a flaw which can lead to your catalogue to be deleted. As Magento is fairly young, you will need consistent upgrading.

I’d suggest you wrap your core modifications into real Modules so you more easily upgrade as the code is stand-alone.

Your server seems fine in principle but I’m not surprised it slows at 15-30K visitors a day. Upgrading will speed it up but only so much. It won’t make it load in 2 seconds alone.

Read and write queries are separated by Magento so you could set up a DB cluster and split them that way.

But with 15K visitors and an assumed conversion of just half a percent, you should be getting at least 75 orders a day. I don’t know what you sell but I’d imagine that could pay for 2 or 3 servers or for hiring in expert in squeezing the most out of mission critical high volume servers.

 
Magento Community Magento Community
Magento Community
Magento Community
 
marc_j
Member
 
Total Posts:  37
Joined:  2008-08-19
 

Our benchmarks on 1.1 and up have been incredibly positive. We are very happy with Magento’s performance and believe it stands up strong.

Something doesn’t sound right about your testing (we got much better results on 1.0 than what you have). In any case, give it a shot and post back the results on 1.1 - we’d love to all see it.

Splitting the Web/DB to two servers is very recommended. In high traffic sites - this is almost the standard (at least two servers - clustering even more).

MJ

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Hi J.T. and Marc and thank you for your answers.

My client doesn’t want to sell on his website. I know it seems odd but that’s the case. So every dollar spent on hosting counts (note that we’re currently billed 1500 euros for one month).

To be honest, we hired a company to develop this website and I’ve no idea if they really made it the right manner (in modules instead of core files). I’m going to test an upgrade as soon as possible (in one or two hours) and I’ll post my results here.

I think we will order a second (or more) server to host the DB asap. Seems the right thing to do.

Thanks again and I’ll keep you informed as soon as I’ve got results.

 
Magento Community Magento Community
Magento Community
Magento Community
 
J_T_
Mentor
 
Avatar
Total Posts:  1961
Joined:  2008-08-07
London-ish, UK
 

For God’s sake, take a back-up first before upgrading! smile

1500 Euro’s for that server seems high, I pay 700 or so Euro’s for a better one but with 100GB a month bandwidth (you probably buy a lot more).

Perhaps instead of jumping in a new 12 month+ contract for a dedicated server, you can first try 2 or 3 extra VPS deals (should cost maybe 100 E a month). You can then build a MySQL cluster on them and see if it significantly improves. If so, you can then decide on an expensive dedicated server.

As marc_j said though, there may be more to it. I’m not the best sysadmin so I don’t exactly know what to suggest but somewhere in the LAMP stack you may be able to squeeze more power. You have APC which is a good move but maybe your MySQL setup is dead-slow because of some config issue.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Our hosting provider has been tunning the whole LAMP server for 3 days from now (that’s where the 1500/month euros go, our platform is completely maintained by the provider).

I’ve just tested to update via pear but I’m now having a mysql issue on upgrade:

[EDIT] Ok, a table wasn’t in InnoDB format.

PDOException Object
(
    
[message:protected] => SQLSTATE[HY000]General error1005 Can't create table './xxxx/#sql-5f64_30.frm' (errno: 150)
    
[string:private] => 
    
[code:protected] => HY000
    [file
:protected] => /var/www/xxx/lib/Varien/Db/Adapter/Pdo/Mysql.php
    [line
:protected] => 156
    [trace
:private] => Array
        (
            
[0] => Array
                (
                    
[file] => /var/www/xxx/lib/Varien/Db/Adapter/Pdo/Mysql.php
                    [line] 
=> 156
                    [function] 
=> query
                    [class] 
=> PDO
                    [type] 
=> ->
                    
[args] => Array
                        (
                            
[0] => ALTER TABLE `sales_flat_quote_itemADD CONSTRAINT `FK_SALES_QUOTE_ITEM_CATALOG_PRODUCT_ENTITY`FOREIGN KEY (`product_id`) REFERENCES `catalog_product_entity` (`entity_id`) ON DELETE CASCADE ON UPDATE CASCADE
                        
)

                )

Strange MySQL error. I’m enquiring about it.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

So I’ve updated Magento to 1.1.6.

First, we can see a performance increase.

We’re getting these results

Connection Times (ms)
              
min  mean[+/-sd] median   max
Connect
:        4    4   0.0      4       4
Processing
:  2850 2857  10.6   2865    2865
Waiting
:     2667 2689  31.8   2712    2712
Total
:       2854 2861  10.6   2869    2869

Instead of these :

Connection Times (ms)
              
min  mean[+/-sd] median   max
Connect
:        4    4   0.0      4       4
Processing
:  7277 7283   9.2   7290    7290
Waiting
:     6866 6869   4.2   6872    6872
Total
:       7281 7287   9.2   7294    7294

So it’s like the site is 3 times faster.

But the theme has to be recoded. Many parts of the layout don’t work as expected. I’ll get in touch with the company that developped this site to estimate the work that has to be done.

Thanks you all for your help and I’ll continue to post everytime I get some updates on my quest to increase Magento perfomances smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
RoyRubin
Enthusiast
 
Avatar
Total Posts:  968
Joined:  2007-08-07
Los Angeles, CA
 

@Guillaume Moigneu - Make sure you have your company review this - http://www.magentocommerce.com/wiki/upgrading_magento_themes_for_the_latest_releases

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Thanks Roy, I’ll have a look at it.

Strange thing, all my categories and products have been desactivated with the update and reactivating them in the admin panel is taking a while.

 
Magento Community Magento Community
Magento Community
Magento Community
 
J_T_
Mentor
 
Avatar
Total Posts:  1961
Joined:  2008-08-07
London-ish, UK
 

Will be interesting to see if the performance increase holds up after re-enabling them all.

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Hello,

I’ve made some new tests on the EAV cache :

Without EAV cache (average response : 4.47secs) :

Lifting the server siege...      done.
Transactions:                      96 hits
Availability
:                 100.00 %
Elapsed time:                  34.09 secs
Data transferred
:               2.57 MB
Response time
:                  4.47 secs
Transaction rate
:               2.82 trans/sec
Throughput
:                     0.08 MB/sec
Concurrency
:                   12.59
Successful transactions
:          96
Failed transactions
:               0
Longest transaction
:            5.06
Shortest transaction
:           3.68

With EAV cache (average response : 1.61secs) :

Lifting the server siege...      done.
Transactions:                     395 hits
Availability
:                 100.00 %
Elapsed time:                  57.09 secs
Data transferred
:              10.32 MB
Response time
:                  1.61 secs
Transaction rate
:               6.92 trans/sec
Throughput
:                     0.18 MB/sec
Concurrency
:                   11.17
Successful transactions
:         395
Failed transactions
:               0
Longest transaction
:            4.34
Shortest transaction
:           0.01

BUT, I now have some strange errors sometimes :

entity_type non valide:  random attribute (name, catalog_category, ...)

You can find a screenshot of the report and my test log as attachments. If you have any idea ...

Thanks.

Image Attachments
error.JPG
File Attachments
siege.log  (File Size: 264KB - Downloads: 78)
 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

I’ve browsed the forums and the only answer you get is “Keep the EAV cache disabled”. Magento Team, we need a solution on this problem ! smile

Bug tracking issue : http://www.magentocommerce.com/bug-tracking/issue?issue=1788

 
Magento Community Magento Community
Magento Community
Magento Community
 
Nls_io
Jr. Member
 
Avatar
Total Posts:  24
Joined:  2008-02-13
Paris, France
 

Our customer site is up and running at full speed !

All pageloads are now under 2 seconds on Magento 1.0.
Many tweaks have been applied in the database and the infrastructure has been boosted. 1 load balancer on two Xeon 4Gb of Ram web servers that connect to a Xeon 8Gb of Ram database server.

I’ll write up asap a sheet listing all tweaks and configurations that have been applied.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top