Benchmarking Magento Enterprise Edition 1.13

Customer Needs and Expectations

The focus of the Magento Enterprise Edition 1.13 release is performance and scalability. The benchmarking results presented here demonstrate that we have addressed the following concerns:

  1. The eCommerce landscape is changing, and merchants must provide customers with an online shopping experience that meets their performance expectations
  2. As merchants grow their businesses and increasingly larger enterprise merchants are adopting Magento, Magento Enterprise Edition must scale to handle increased traffic volume and larger catalogs

Magento's Plan

We identified the following areas for enhancement in Magento Enterprise Edition 1.13:

  • Improve indexing for catalogs of all sizes
  • Make re-indexing operations invisible to the shopper by executing them in the background
  • Improve full page caching support
  • Reduce page load times for key shopping flows (checkout)
  • Enable merchants to serve heavier traffic volume without need to purchase additional hardware

What Magento Accomplished

Magento Enterprise Edition 1.13 delivers the following improvements over Magento Enterprise Edition 1.12:

  • Incremental re-indexing has been introduced with Magento Enterprise Edition 1.13, and scenarios where full-re-indexing was required have been limited. This means that operations that previously took hours can now be completed in minutes
  • 53% improvement in completion times for a full re-index with a 500,000 SKU catalog
  • 35% improvement in “Place Order” performance
  • 65% improvement in page load times across shopper flow pages that were tested.
  • 33% improvement in orders per day that can be processed on the same hardware configuration as Magento Enterprise Edition 1.12
  • 31% improvement in page views per day supported on the same hardware configuration on Magento Enterprise Edition 1.12

Multi-Node Deployment Topology

Before we introduce you to the benchmarks, an overview of our multi-node benchmarking facility is in order. We started with a basic multi-node cluster with load balancer and caching and separate DB node, and installed Apache. This is a familiar hardware configuration for the Magento developer community. Once provisioned, we installed both Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12 on the cluster to compare their performance.

This is a representation of the multi-node Magento Enterprise Edition 1.12/1.13 installation used for performance testing. It consists of four physical nodes, three of which are virtualized.

We used HP ProLiant SL230s Gen8 servers with 8x16GB DDR3 and 2x Intel Xeon E5‐2660 CPUs for our benchmarking.

We used a physical disk of a usable 1.2 TB managed by a RAID10 controller. The load balancer we used is nginx.

The cluster resides in our Las Vegas data center, connected to the Internet via a gigabit connection. We tested the cluster using the popular Gatling suite from our engineering offices in Austin, TX.

Software Components

Scenarios

This section presents the merchant scenarios we simulated for our testing. These scenarios are based on real-world experience and industry standards with respect to shopper flows and catalog sizes.

Shopper Flows

We used real-world, established eCommerce metrics to simulate shopper flows.

  • 94% of eCommerce shoppers visited a storefront but did not purchase any products
  • 65% of shoppers added items to their carts but abandoned them
  • Of the 6% who did purchase products (otherwise known as the conversion rate), half checked out as guests without signing in, and half signed into the storefront.

Catalog

The catalogs we simulated also reflect real-world, established eCommerce experience.

  • We simulated small and medium-sized companies with a 50,000-item catalog with 27 product categories.
  • We simulated large companies 500,000 items in the catalog with 2,000 categories.
  • As many eCommerce sites offer different types of products, we specified physical (simple) products, virtual products, and downloadable products, with 60% of the products in each catalog being physical.
  • We benchmarked each catalog on one Web site/storefront.

Target Merchant Profile

The simulated merchant profile we benchmarked against represents the profile of our enterprise customers. We established these metrics based on the day-to-day experience of large merchants operating a successful eCommerce business. Again, we used what we consider to be a typical hardware and software configuration.

  • 50K visitors / day
  • 1M Page views / day
  • 18000 orders / day
  • 3000 orders during peak 4 hours
  • 1000 concurrent users
  • Standard HW/SW configuration

Benchmarking Results

These benchmarks were generated during extensive testing of our multi-node configuration running Magento Enterprise Edition 1.13 described above. Our configuration was modeled after what a commercial hosting partner would put into production, and the results reflect accurate gains over Magento Enterprise Edition 1.12.

The duration of the test sessions is 72 seconds, which significantly stresses the multi-node configuration.

Incremental Re-indexing

In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index. Magento Enterprise Edition 1.13 introduces a new feature--incremental re-indexing. With incremental re-indexing, only those items that were changed or added will be re-indexed, reducing the processing time to a fraction of what was required before.

Take the example of a merchant with a catalog containing 500,000 products. In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index operation. In Magento Enterprise Edition 1.13, incremental re-indexing means the merchant will only re-index items that were changed. The test focused on measuring the improvements provided by the incremental re-indexing feature in Magento Enterprise Edition 1.13. The table below compares improvements to common admin actions, such as changing a product description, prices or inventory.

Full Re-indexing

As part of the benchmarking effort, we also measured the improvements in the full re-indexing feature on Magento Enterprise Edition 1.13, where indexing a 500,000-item catalog was 53% faster than Magento Enterprise Edition 1.12. Faster re-indexing means less load on the system and that changes to the catalog propagate faster to the storefront.

  • 500,000 products
  • 2,000 categories
  • One store
  • One catalog update during test run

Individual Re-indexers

The Magento Enterprise Edition 1.13 indexer component contains a number of individual indexers such as Product Flat Data and Product Price. When a Magento admin changes the price of a product, it is only necessary to execute the Product Price indexer for pricing changes to propagate to the front end. The completion times of these individual indexers were measured in the benchmark environment for Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12.

This section presents the results for full re-index completion times for the individual indexers in Magento Enterprise Edition 1.13 compared to Magento Enterprise Edition 1.12.

  • URL Rewrite failed to run in Magento Enterprise Edition 1.12 but takes only 0.15 sec in Magento Enterprise Edition 1.13
  • Catalog description: 500,000 products, 2000 categories, one storefront

Page Load Times

When Magento Enterprise Edition 1.12 and Magento Enterprise Edition 1.13 were compared, with both running on our multi-node benchmarking configuration, Magento Enterprise Edition 1.13 loaded pages 65% faster than Magento Enterprise Edition 1.12.

Guest checkout and registered user checkout are two flows that are crucial to storefront operation. This section presents the results of page load time measurements for these two flows.

Guest Checkout Flow Page Load Times

In the guest checkout flow pages, Magento Enterprise Edition 1.13 provides a substantial decrease in load times over its predecessor, most of the time more than twice as fast.

Registered Checkout Flow Page Load Times

The bar chart below presents the improvements in page load times for registered checkout flow.

Page Views and Orders

In addition to page load times, the benchmark also focused on measuring throughput improvements, particularly page views per day and orders per day.

During our testing, which simulated a storefront running at peak hours, EE 1.13 executed 33% more orders and 31% more page views than Magento Enterprise Edition 1.12 on the multi-node benchmarking configuration. Notably, Magento Enterprise Edition 1.13 served 47K pages during the test run (10 minutes).

  • 10 minute session time

Observations

What we noted during the benchmarking tests:

  • The MySQL instance did not show any significant signs of CPU or I/O load during the tests.
    • The CPU was under 10% and no queries exceeded a 2-second threshold.
  • Redis and Memcached instances did not exceed a CPU load of 10% during the tests.
  • Web nodes showed high levels of CPU utilization under high load.
  • We anticipate achieving stable scaling by adding additional web nodes to the cluster until the services themselves begin to degrade.

Conclusion

Magento Enterprise Edition 1.13 was engineered for performance--and clearly delivers on the goal as measured by important metrics.

  • Indexing is improved to enable faster operations without impacting shopping experience. Merchant administrators can add and update products as needed while ensuring product URLs, promotions, navigational menus, and product search tools are always up to date.
  • The checkout process is improved by reducing page load times for browsing and placing orders. Faster checkout can significantly improve your customers’ shopping experience and customer satisfaction, and potentially improve your conversion rate.
  • Faster page load times means Magento Enterprise Edition 1.13 can support more page views per day and more orders per day, potentially increasing your conversion rate.

In addition, we have focused on providing these benefits without the need to upgrade existing hardware, improving your return on assets and investment.

Appendix

Page Load Times

This benchmark report presented analysis of the results produced by the Gatling load generator tool. In this appendix, you can see the actual results reported by Gatling. The results were obtained under the following test setup, which was designed to push Magento to its limit.
Concurrent session (users)1000
Peak load duration10 minutes
Session duration72 seconds
CPU utilization95%

CPU utilization is high because session duration was set to 72 seconds for the test. Session duration was set to a low value to increase the active load on Magento. A typical e-commerce site will see session durations of five minutes or more. Preliminary experiments by the benchmark team have confirmed that using a five-minute session duration reduces the CPU utilization to 50-60%.
Magento EE v1.13
RequestsTotalOKKOMinMaxMeanStd Dev95th Pct99th PctReq/s
Global Information06216162087748012910173518695780806075
product page0285732857308093907099962670492034
Click add to cart1104581045804407170186811394010498013
Click add to cart Redirect 121045810458096012910381622398050996013
click checkout334863486011001080034591965713089704
Registered Checkout:Login and Pick Address4336633660200738011551482487059004
Registered Checkout:Login and Pick Address Redirect 153366336607001144026051662564077104
Guest Checkout start612012003602490908487186024200
Registered:Pick Billing Address 17120120039042401457974351042100
Registered:Pick Billing Address 2-18120120041036901218685246030700
Registered:Pick Billing Address 2-29120120040036001201714270035100
Registered:Go to pick Shipping Method10120120034029001072633233027000
Guest:Pick Billing Address 111120120061046201688941358044400
Guest:Pick Billing Address 212120120038035301049589205027500
Guest:Go to pick Shipping Method1312012003002080796401153019600
Set Shipping Method (Flatrate) 1142402400510483017801066393047200
Set Shipping Method (Flatrate): Goto Payment1524024003003150905516194023000
Set Payment Method (Check/MO) 116240240067052901787987366049200
Set Payment Method (Check/MO) 21724024003002760904518201024400
Set Payment Method (Check/MO) 3182202200370818026961725581075900
/checkout/../success/1924024002502830931583197025300
/checkout/../success/ Redirect 120740741030835024731620559073100
Magento EE v1.12
RequestsTotalOKKOMinMaxMeanStd Dev95th Pct99th PctReq/s
Global Information04804447887157404058049853290106501387058
product page021327212864140200404557282097401235026
Click add to cart183408324164012130421622177520900010
Click add to cart Redirect 12832483204401650070223449117801350010
click checkout327802779192012350631128299760109503
Guest Checkout start49090039036902103795321034100
Registered Checkout:Login and Pick Address5269026900180191203298498415470169403
Guest:Pick Billing Address 1690900600775043481605643074000
Registered Checkout:Login and Pick Address Redirect 1726902688240243605433335011780140603
Guest:Pick Billing Address 2890900520824034121423548068100
Guest:Go to pick Shipping Method99090031047802334888330044500
Registered:Pick Billing Address 110909003701009036572453866096000
Registered:Pick Billing Address 2-11190900540176704148274110220125100
Registered:Pick Billing Address 2-2129090053012640398324118290115900
Registered:Go to pick Shipping Method1390900360629025461401505059800
Set Shipping Method (Flatrate) 1141801800450764039261884655074500
Set Shipping Method (Flatrate): Goto Payment151801800380629024511256427048300
Set Payment Method (Check/MO) 1161801800570829039231996646073000
Set Payment Method (Check/MO) 2171801800330523023181238421048100
Set Payment Method (Check/MO) 31818018009404058012673795725910295400
/checkout/../success/19180180023033301437854287031500
/checkout/../success/ Redirect 120930937801047050312251792088800

Estimating the Number of Visitors

Gatling does not report the number of users that cycled through during the test. However, it is possible to estimate the number using the following formula:

Discussion:

RSS: New Article posts

Explore the Knowledge Base