Do you have problems with:
- Reloading page after saving a product/catalog, getting a blank page
- Price rules are not properly applied to new products
Do you use many catalog rules for a large amount of products/customer groups?
Do you have a large number of rows in the catalogrule_product table?
Keep on reading the following in you servers error log?
Fatal error: Allowed memory size of xxxxxxxxx bytes exhausted (tried to allocate 128 bytes) in lib/Zend/Db/Statement/Pdo.php on line 294
Came here after reading bug report?
Looking for a fix for all of these issues?
If you think these questions above apply to you, please read on.
First off my situation and the numbers that I’m using. I’m running a large 5000 product webshop with around 20 different customer groups. Each group can have more than one discount percentage (10 available). All these numbers give me nicely filled catalogrule_product (~115k rows) and catalogrule_product_price (~278k rows) tables. The price display works like a charm. Magento is version 1.3.0 with quite a few extensions, the most relevant is http://www.magentocommerce.com/extension/596/simple-configurable-products
Saving products error
When I’m adding or modifying a product I will nicely see the ‘Please wait’ while Magento is saving it, after waiting for quite a while for a page reload I would always end up with a blank page (with URL, e.g. index.php/admin/catalog_product/save/id/5011/key/*****/), without any error message. It had just died without giving any clue why. Happy as I was that it saved the product I clicked the back button to get my menu’s again and continued managing the shop leaving this error for what it was assuming it would fade with the next upgrade.
Price rules not applied for new products
Due to a quite exotic price display setup and some recent changes to it I have been testing many possible variations and during these tests I noticed that for some products it would not show the price it should have according to the price rules. After some digging I discovered that all of these products were added after launching the webshop. Inserting a few new products showed that the problem was persistent, had nothing to do with inventory management (turned off in my case) but with the price rules not being fully applied. Only after firstly adding a product, secondly manually opening and saving all catalog price rules (not making any changes) and finally applying all price rules the product would show the correct price, weird!
Time for some deep digging so I got my shovel, spade and a hand grenade, hey ho, hey ho....
Since I’m using quite some extensions and tweaks to the extensions my attention was first aimed at them. But after some hours I couldn’t find anything on the forum or in the source code. They were all working as they should, perfectly fine (thanks guys!).
Since I knew that saving the product went fine and refreshing the page failed I focussed on the part between those. When the system is done saving a product it will issue a ‘catalog_product_save_after’ event. This event is then picked up by the following core observers running the specified functions:
- catalog/product_flat_observer => catalogProductSaveAfter
- catalogindex/observer => processAfterSaveEvent
- cataloginventory/observer => saveInventoryData
- catalogrule/observer => applyAllRulesOnProduct
- catalogsearch/full_text_observer => refreshProductIndex
- googlebase/observer => saveProductItem
- googleoptimizer/observer => saveProductGoogleOptimizerScripts
The order in which they are run is unknown to me.
Since I found that manually reprocessing all the catalog rules helped in displaying the price properly I dug into the catalogrule observer. When saving a single product, applyAllRulesOnProduct is called and we end up calling these functions (in order of execution):
- applyAllRulesOnProduct @CatalogRule/Model/Observer.php
- applyToProduct @CatalogRule/Model/Rule.php
- applyToProduct @CatalogRule/Model/Mysql4/Rule.php
- applyAllRulesForDateRange @CatalogRule/Model/Mysql4/Rule.php
The last function never returns after dispatching the ‘catalogrule_after_apply’ event. Searching the source code shows there are three functions reacting to this event:
- catalogindex/observer => processPriceRuleApplication
- sales/observer => markQuotesRecollectOnCatalogRules
- weee/observer => updateDiscountPercents
The first two are run without any problems, the third crashes however....... BINGO!
It seems this function, located inWeee/Model/Mysql4/Tax.php, tries to do a fetchAll on my 115k catalogrule_product rows and runs out of memory along the way, making a dead stop without any error messages, apart from the apache log file (Allowed memory size of 128MB exhausted).
Due to the dead stop, a number of functions will not run completely or not at all. With certainty I can say that the price rules are only applied to the first customer group of the first website, not any others since this function issues the ‘catalogrule_safe_after’ event and then dies quietly. It could be (not checked) that catalog-, search- and inventory-indexes are probably not processed properly and any events coming from any of these functions will not be run either. Depending on your setup this might have big consequences.
Even if you would not have my problem that the system does not handle a product save completely there are some performance implications. This is due to the way the updateDiscountPercents works. First it empties out the entire ‘weee_discount’ table and then starts filling it again. It does this for ALL the products in ‘catalogrule_product’ and it happens for each customer group where price rules are applied. In my case it would run the entire delete/fill sequence if 115k rows about 20 times (now that I think of it I’m glad it actually doesn’t). When you only have a small number of ‘catalogrule_product’ rows the impact is not that large but when that number is a bit bigger it will eat away time and resources.
Furthermore, if you have your MySQL bin-log turned on, be prepared to see these files GROW as you edit your products.
I’m running Magento 1.3.0 and to make sure that this thing wasn’t fixed in any of the other releases since I’ve also checked 1.3.2. The code in the WEEE module is the same as under 1.3.0, including xml files. In addition, the called events haven’t changed either.
Your eyes are probably 16:10 squares by now, like mine, so I’ll put my possible workarounds in the next post below.
A bit further down I’ve added notes on applying my workaround to my production environment, I had a small issue with saving the first product after applying the workaround.
Edit: Added the exact fatal error message I keep on getting. (all the way at the top of this post)