yashgt: You have been thoroughly warned of the underlining limitations of Magento and what needs to be fixed in order to do multiple sites, Good Luck with your adventure and let me know how it goes.
P.S. Your DB server will be inadequate, even amazon’s “Cluster Compute Eight Extra Large Instance” will be too small, but I would at least start from there..
We have to agree with Ryan’s points completely, Magento will have a major problem with this. There are three underlying options:
1) Use Magento as is already chosen
2) Develop a custom solution
3) Use an alternative application
We can say from experience that Magento is the best in class for eCommerce unless you are prepared to spend many 10s to 100s of thousands of dollars on an Enterprise level application for your needs. As already chosen, Magento then breaks down in the the following three options for your specific installation, with distributed systems being the least risk in your scenario.
a) Business - change current core Magento functionality to behave as you require
b) Distributed - have a master Magento installation that then distributes to satellite installations
c) Technical - throw as much hardware as possible at it and hope it all works
With Magento you really want to change as little as possible otherwise you will spend your life developing code, never being able to upgrade, fixing problems on a continual basis and be at the mercy of the system 24x7, therefore let’s rule out a).
With b) you would have a master with the 100,000 product catalog and limited stores, propogating to slave Magento installations each with say 500-2000 websites (depending on your risk tolerance), however you will need fully automated propogation. This makes it scalable, independent if one system crashes and minimizes code changes however you would have to manage multiple instances.
With c), as Ryan states, you will hit limits in the dashboard and the database server (we presume you are using RDS), you can only have one write database so your product saves will become an issue. Adding stores/websites from single store starts to become almost exponential in the performance degradation. On simple servers we can batch process 50records per minute with a single store, 20records per minute with 2-3 stores and 2-3 records per minute with 8-10 stores (50,000 product database), with products only active in a 2-3 of the websites. These are with layered catalog indexing off as it produces the heaviest load. You can switch indexes to manual update however you will need to run the indexer regularly, if you are using RDS you will be charged per million IOs, you will be suprised how quickly this mounts up.
We have tested clustering, you can replace foreign keys with stored procedures however we found it unreliable and maintenance intensive when being used for Magento. Planning for a distributed (b) solution is you best approach as you can test a real live scenario with two instances (master and slave), then launch new instances as necessary. If the venture becomes successful you can always join them together again should a solution be found to the various issues that will arise along the way. You are putting all your eggs in one basket with c), Magento and AWS can become unresponsive for no particular reason, you will be in an all or nothing scenario with 24x7 maintenance.
With option a) you have 80% chance of failure, option b) you have 80% chance of success, option c) 50/50 chance of success/failure (80% of the time being midly unsuccessful, we hope you fall in the other 20% who become wildly successful). In your scenario we always go with option b) even if our clients do not like it, we only deal with successful projects using risk mitigation.
I tested a few different cache storage engines and Redis was the only one able to handle our server’s load (it was faster when used with enterprise’s full page cache then varnish was for us). But also the cache server has to be able to handle multiple web nodes connection to it and also be able to be clustered and backed up… (Redis does all this)
We found on a large traffic client that completely disabling Magento EE FPC and implementing an varnish extension we gained an quite effective setup using varnish, memcached, nginx, php-fpm.
currently looking into the redis features on multiple instance setups in contrast to running multiple memcached instance. - Any experience with that ?
In the example with 1 Master Magento and multiple magento “slaves” did we solve as a conseqeunce of how we address integration with backoffice ERP systems. We need to be able to scale integrations effecient, also address scalable flexibility of integrating to multple applications using same data and ended up implemting a “BizTalk” Application server for Magento. Where we can have an PIM Connector for product distribution. (Publishing Connector) and dynamically load one to many subscriber connectors.
Scale MySQL on Cloud is somewhat a paradox. - i mean large scale eCommerce solutions without any control on iops…
Crown is doing an advanced SOLR webinar next week with Magento where we will address how SOLR search indexing works, common bugs and fixes, and increasing SOLR speed. If you’ve got questions on SOLR, this is the place to be.
Check it out here:
http://info.magento.com/081512CrownWebinar_reg.html
Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.
Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension of http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html
We handle a number of Magento multi-store setups. Our solution help integrate Magento with other Magento stores, POS systems as well as ERP, Inventory and suppliers. While we’d have to learn more about your particular solution, you may be interested in at least learning more about how this might be a solution for you with multiple stores.
Using the nChannel connector for Magento you can quickly and easily integrate and connect Magento to a variety of POS, ERP and accounting systems. For existing Magento sites, once downloaded, our Magento connector starts copying your customer, item, sales and supplier data to the nchannel platform. Simply add additional connectors for the other channel systems you use to begin exchanging data between Magento and any other channel!
If you are new to Magento and haven’t populated your site yet, you can use nChannel to push item and inventory data to it from your POS, other eCommerce, ERP or accounting system.
You can go to a full page cache however we find multiple optimisations such as apc, memcache, php compiling, optimised index management, etc are more efficient, we have worked with terrabyte databases, as long as you install the db correctly with optimal settings millions of records are not a problem.
In relation to Magento, it is slow especially with thousands of products and above, we normally use AWS with Elasticache, RDS, Route53 and load balancing as you can upscale instantly and leave the infrastructure headache to Amazon. For high performance sites it is best to allocate a batch server to take the load off the web servers for index management, admin, etc.
Monty22, we are not sure what you are trying to achieve but copying someone elses post carte blanche and passing it off as your own is probably not the best way to build trust.
Back to the point, multisite performance is truely abysmal once you get passed 15-20 websites/stores as Ryan points out, up to 250stores you are talking enterprise class environments. It is possible as we are doing it, but you need a very rare understanding and relatively good budgets for the hardware as well as incredibly efficient delta product handling to minimise loads on the servers.
