Using Magento on Amazon EC2

Last modified by filpt on Wed, June 23, 2010 15:39
Source|Old Revisions  

This is an old revision of the document!


Introduction

This wiki page is used to share some information about using Magento on Amazon’s EC2 cloud hosting environment.

We have tested with m1.small (1.7GB RAM) and m1.large (7.5GB RAM) instances first with Apache and later also with the not so widely used (only 1-4% market share) Nginx webserver.

You can launch your own instance with our (m1.small) Magento optimized Amazon Machine Image (AMI) has been registered in the US region and can be launched e.g. with the AWS Management Console or with the EC2 API command line utilities. You can easily find the AMI by typing magento into the management console’s search bar.

We’ve also created a Magento optimized m1.large AMI in the EU region which is also registered and publicly available, as well as an m1.small AMI using Nginx instead of Apache. Ssee below for more info on Nginx and the exact AMI names.

System and Package Info

I used the Virtualmin GPL Debian Etch AMI as a basis for the m1.small image and launched it from AWS Management Console.

The AMI is based on the Linux kernel 2.6.16-xenU and has (among others) the following packages preinstalled

  • Apache 2.2.3
  • PHP 5.2.0-8+etch13
  • MySQL 5.0.32

I had to install a couple of PHP related packages that are required by Magento in order to walk through the installation wizard and complete the installation successfully

  • php5-gd
  • php5-curl
  • php5-mcrypt
  • php5-mysql

Optimizing Performance with Apache

The default PHP memory_limit was set to 16M which triggered memory errors on the product listing page, so I upped this to 512MB which solved the problem. I purposely set it to such a large value, because this whole set up is directed towards running a single Magento store on one EC2 instance, but after I found out that reducing the value to 64MB would yield the same result, I preferred to keep it that way, i.e. on 64MB.

I continued by following the do it yourself performance enhancements outlined in Performance is Key! - Notes on Magento’s Performance

1. MySQL Configuration

Modifying the configuration of MySQL server to take better advantage of the server’s RAM.

Most Linux distributions provide a conservative MySQL package out of the box to ensure it will run on a wide array of hardware configurations. If you have ample RAM (eg, 1gb or more), then you may want to try tweaking the configuration. An example my.cnf is below, though you will want to consult the MySQL documentation for a complete list of configuration directives and recommended settings.

I double checked all query_cache_ variables and upped query_cache_limit from 1 to 16MB. Result: no further improvements. Reset value to 1MB

I also checked have_query_cache and query_cache_size variable values to make sure query caching is really enabled (see MySQL docs: http://dev.mysql.com/doc/refman/5.0/en/query-cache-configuration.html)

  • Parse time home page: 1.1-1.5s
  • Parse time product listing: 1.4-1.6s
  • Parse time product detail: 1.6-2s
  • Parse time add item to cart: 2.7-2.9s

Although others have reported huge performance improvements after tweaking MySQL config, the demo store with only a couple of products, does not seem to make a big difference. This might be different with stores that have more than 1000 or 10’000 products and many product attributes.

Although I did not do any precise benchmarking so far, the performance improvement was based on the Magento profiles parse times was not more than 100ms (mili seconds).

Finally, I ran the MySQL Performance Tuning Primer Script and got a couple of warnings, but I think that the configuration is still valid, because there has not been a lot of traffic so far. 48 hours have not yet passed, but I think the results are already representative:

http://ec2-174-129-235-96.compute-1.amazonaws.com/apache2-default/mysql-tuning-primer

2. Apache KeepAlives

Making sure the Apache configuration has KeepAlives enabled.

KeepAlives are a trick where multiple HTTP requests can be funneled through a single TCP connection. Since the setup of each TCP connection incurs additional time, this can significantly reduce the time it takes to download all the files (HTML, JavaScript, images) for a website.

  • Has already been enabled in the AMI used as a basis for this setup

3. PHP Opcode Cache

This can deliver significant improvements to PHP‘s responsiveness by caching PHP code in an intermediate bytecode format, which saves the interpreter from recompiling the PHP code for each and every request.

I installed PHP opcode cache XCache v1.2.2 as a Debian package via etch-backports.

  • Parse time home page: 1.0s
  • Parse time product listing: 1.2-1.5s
  • Parse time product detail: 1.3-1.6s
  • Parse time add item to cart: 0.9-2.3s

Seems that XCache has more effect on the current demo store than MySQL query caching optimization. Again we have to consider the fact that the demo store only has very little products in it!

After optimizing MySQL configuration and installing XCache on the m1.large instance, I got the following parse times:

  • Parse time home page: 0.2-0.3s
  • Parse time product listing: 0.4-0.6s
  • Parse time product detail: 0.6-0.8s
  • Parse time add item to cart: 0.6-1.2s

4. Memory-based Filesystem

(this has not yet been implemented on the EC2 instance as of now!)

Use a memory-based filesystem for Magento’s var directory. Magento makes extensive use of file-based storage for caching and session storage. The slowest component in a server is the hard drive, so if you use a memory-based filesystem such as tmpfs, you can save all those extra disk IO cycles by storing these temporary files in memory instead of storing them on your slow hard drive.

Public and Free Magento AMIs

  • magento-etch-virtualmin-gpl-3.63 (Apache, US region m1.small)
  • debian-4.0-etch-64-magento-2009-03-10 (Apache, EU region m1.large)
  • debian-4.0-etch-32-magento-nginx-2009-03-15 (Nginx, EU region m1.small)

Important note: When you launch your own instance, you have to make two small modifications in order for the demo store to run:

  1. Change the first two entries in the core_config_data table of the MySQL database to reflect the new URL that you have been assigned
  2. Clear the file cache under /var/www/apache2-default/magento/var/cache/*

Now the stylesheet will be read correctly and the demo store should be available at

http://your-ec2-domain-123.compute-1.amazonaws.com/apache2-default/magento/

Note for Nginx AMI: When you launch your own instance with the magento-nginx AMI, you have to manually start Varnish (HTTP accelerator) with varnishd -f /usr/local/etc/varnish/default.vcl. In production use, you might want to add a start script to make this happen automatically.

Launching an EC2 Magento Demo Store

After launching a new instance you can access the store in the browser by appending /apache2-default/magento/ to your instance URL. In case you want to login to the admin control panel, please use the username admin and password 4KKEzgn9zZ for the US m1.small instance and admin/magento for the EU m1.large instance.

Benchmarking with ApacheBench

I’ve been testing with 1000 requests and a concurrency of 10 on the m1.large instance, first without any performance optimization and then with MySQL query cache optimization and XCache installed:

I have no experience with the ab (ApacheBench) utility and wonder why 960 out of 1000 requests are failing. Maybe someone with more experience can shed some light on this in order to get some representative benchmarking results.

I’ve also installed osCommerce v3.0 Alpha 5 on an m1.small instance and got ApacheBench reports where all requests were successful, so it seems that either ApacheBench has problems with resource hungry scripts or there’s some other problem I’m not aware of.

Benchmarking with Pingdom

I’m not yet sure what to think about Pingdom, because the total page load times indicated often are a lot longer than how pages load on my machine, but it seems still to be a good indicator for overall performance as parse time is not everything!

m1.large instances have very fast parse times, but the total page load time as measured for the store home page by Pingdom is over 6 seconds, on an m1.small instance evenaround 10 seconds, let alone the product detail or product listing page.

After testing and tweaking with Apache Prefork (mod_php), we also did some testing and tweaking with Nginx, an open-source, high-performance HTTP server and as a result got total page load times of 4.5 seconds (see archived Pingdom test) on an m1.small instance even though the parse times themselves were between 0.8 and 1.0 seconds! This was only with php_fastcgi and no other tweaks.

We have not yet tested Apache together with php_fastcgi, but it seems that Nginx is the way to go if you’re after high performance Magento hosting.

Optimizing Performance with Nginx

After having set up Nginx and php_fastcgi under Debian Etch we added some tweaks to make it even faster based on the following resources:

Following Jauder Ho’s recommendations we also installed

After that, total page load time of the store home page according to Pingdom was between under 2.4 seconds (see this archived Pindom test!) and under 3.2 seconds. Total page load time both for product listing and product detail page was also below 3.2 seconds! This is really awesome for an m1.small instance!

Further PHP and MySQL configuration optimization has not been done yet, but from our previous experience under Apache, it seems that those are only worth it if you either have a lot of products in your store or you have a lot of traffic, so we will leave those for later.

Data Persistance with EBS (Elastic Block Storage)

The following is a hack and work in progress, it is functional, but it can an should be improved. I’m not responsible for any data loss, use a your own risk XD

The following examples are tailored to the debian-4.0-etch-32-magento-nginx-2009-03-15 AMI used as a development host. If you prefere apache or any other setup you need to adjust the scripts.

If you want data to persist between instances and don’t want to build your own AMI (which would be impractical), you need to create a EBS Volume.

You can do so using the AWS Management Console under the “Volumes” tab. When creating the volume you can specify the size (from 1GB to several TB). Pay attenton to the “Availability Zone”, you will only be able to attach the volume to instances within the same zone (e.g. eu-west-a1 or eu-west-b1).

Once it’s created, select the volume, click the “Attach” button and select your running instance. If your instance doesn’t appear in the dropdown it’s probably running in a differen availability zone, or hasn’t booted yet. You need to remember to which device node you attach the volume (e.g. /dev/sdf).

Now you need to initialize your volume so it can be used. Ssh into your instance. Before you can use the volume you need to format it (substitute your device node: mke2fs -j /dev/sdXX).

  • Create a base mountpoint (mkdir /mnt/store1)
  • Stop the services
    • /etc/init.d/nginx stop; /etc/init.d/mysql stop
  • Copy the directories you want to persist between instances to the EBS volume (adjust the list as you need)
    • cp -a /etc/php5 /mnt/store1/php5.conf
    • cp -a /var/lib/mysql /mnt/store1/mysql
    • cp -a /etc/mysql /mnt/store1/mysql.conf
    • mkdir -p /mnt/store1/nginx /usr/local/nginx/vhosts
    • cp -a /usr/local/nginx/conf /mnt/store1/nginx/conf
    • cp -a /usr/local/nginx/vhosts /mnt/store1/nginx/vhosts
    • cp -a /usr/local/nginx/html /mnt/store1/nginx/html
  • Create the mount scripts
    • mkdir -p /mnt/store1/scripts
    • Contens of /mnt/store1/scripts/mount.sh
  1. #!/bin/bash
  2. STORE=$(dirname $0)/..
  3.  
  4. echo "stopping services..."
  5. /etc/init.d/nginx stop
  6. /etc/init.d/mysql stop
  7.  
  8. echo "mounting directories..."
  9. mount -obind $STORE/nginx/conf/ /usr/local/nginx/conf/
  10. mkdir -p /usr/local/nginx/vhosts/
  11. mount -obind $STORE/nginx/vhosts/ /usr/local/nginx/vhosts/
  12. mount -obind $STORE/nginx/html/ /usr/local/nginx/html/
  13. mount -obind $STORE/php5.conf/ /etc/php5/
  14. mount -obind $STORE/mysql.conf/ /etc/mysql/
  15. mount -obind $STORE/mysql/ /var/lib/mysql/
  16.  
  17. echo "starting services..."
  18. /etc/init.d/mysql start
  19. /etc/init.d/nginx start
  20.  
  21. echo done
  • Contens of /mnt/store1/scripts/umount.sh
  1. #!/bin/bash
  2. echo "stopping services..."
  3. /etc/init.d/nginx stop
  4. /etc/init.d/mysql stop
  5.  
  6. echo "unmounting directories..."
  7. umount /usr/local/nginx/conf/
  8. umount /usr/local/nginx/vhosts/
  9. umount /usr/local/nginx/html/
  10. umount /etc/php5/
  11. umount /etc/mysql/
  12. umount /var/lib/mysql/
  13.  
  14. echo "starting services..."
  15. /etc/init.d/mysql start
  16. /etc/init.d/nginx start
  17.  
  18. echo done
  • Make the scripts executable (chmod 0744 /mnst/store1/scripts/*.sh)
  • Run the mount script /mnt/store1/scripts/mount.sh

And thats it. Now, whenever you boot an instance...

  1. Attach the volume in the AWS Management Console
  2. ssh in
  3. mkdir /mnt/store1
  4. mount /dev/sdf mnt/store1
  5. /mn/store1/scripts/mount.sh

Before terminating the instance, run the unmount script to make sure all databases are closed correctly.

TODO for this wiki block:

  • nginx vhost configuration
  • Elastic IP configuration

EC2 Pricing and FAQs

Performance Improvements in Next Magento Release

Performance is a key focus of the Magento core team for 2009. Version 1.3 due out soon should see some performance improvements using a flat catalog database

Todos

  • Add more products to the demo store to see how performance is with e.g. 1000 or 10000 products on both m1.small and m1.large instance

Feedback

Please send me your feedback and suggestions either via my forums profile or follow me on Twitter.




 

Magento 2 GitHub Repository

Magento Job Board - Some sort of tag line goes here

Latest Posts| View all Jobs