Magento

Mike Smullin - 07 March 2009 02:14 PM

I was trying to update all my product paths to “Use categories path for product URLs”. I did this from System > Configuration > Catalog > Search Engine Optimizations but when I clicked “Save Config” it would time out. I have over 12,000 products in this installation. I got the same result trying from System > Cache Management > Catalog Rewrites : Refresh.

I noticed in a lot of places, the php function set_time_limit() is used to enable/disable script timeout. On functions known to possibly take a particularly long time to process, set_time_timit(0) is used to disable temporarily. In many other places, set_time_limit(120) is used to re-enable it again during run-time. This behavior overrides anything you attempt to specify in php.ini or .htaccess.

So, in the spirit of this legacy, here’s what worked for me. I have provided it in unified diff / patch format so you can see exactly the changes I made:

Index: app/code/core/Mage/Catalog/Model/Url.php
===================================================================
--- app/code/core/Mage/Catalog/Model/Url.php    (revision 2102)
+++ app/code/core/Mage/Catalog/Model/Url.php    (working copy)
@@ -142,6 +142,7 @@
      */
     public function refreshRewrites($storeId = null)
     {
+        @set_time_limit(0); // because this can sometimes take a while
         if (is_null($storeId)) {
             foreach ($this->getStores() as $store) {
                 $this->refreshRewrites($store->getId());

Now I can run the catalog rewrites without timing out. Hope this helps someone!

Oh my gosh, everyone should give Mike like $100 because that just fixed it for me.  I am so happy.  that was the only thing holding my site from going live.  1.5K products at the moment which will go up to 6K hopefully it works later.

ok, but is this affecting the layered navigation as well? Can anybody confirm if adding a set_time_limit will help with refresh the Layered navigation

Here is the code I started using if it helps. Posted by Chris McKee at:

http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-deux/

It has worked wonders for me and also cuts down on memory used, which is great for shared hosting. Takes awhile though but can be adjusted through the settings he mentions on his site.

I wanted to add to this post. I used this script, and it worked great for me on a shared host and latest version of Magento and over 6000 Products that I imported. One thing I changed though was on the line:

sleep(1); //Pause a second, for less use u_sleep(microseconds)

I changed it to:

sleep(0); //No Pause

Works Great! Thanks for the great Script!

http://www.christmaseverymonth.com

More than welcome; I’ve a feeling this will still be useful when/if they fix the bug for cron jobs; in fact I’ve found a few cases where its better/more suitable to code a custom script; thank god the Zend API doesn’t get changed as often as Magento is all I can say.

The original files (and subsequent updates) are

http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-deux/
and
http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error/

The first post (secondlink) has a bit more info on the issue itself. the second was more about fixing it and making it easier to implement (like clearing the flags automatically)

fermium1 - 16 April 2009 01:34 PM

I wanted to add to this post. I used this script, and it worked great for me on a shared host and latest version of Magento and over 6000 Products that I imported. One thing I changed though was on the line:

sleep(1); //Pause a second, for less use u_sleep(microseconds)

I changed it to:

sleep(0); //No Pause

Works Great! Thanks for the great Script!

http://www.christmaseverymonth.com

You can change the script as above or even using microseconds instead. The delay is in there for OLD servers, and SHARED servers to avoid creating too much load.

If its not a risk for you (or your running the script on a local test server) you can afford to turn it to zero.

The Actual script (magento bit that is) leaks memory like a sieve; which is part of why it crashes in the official method. Mine is intended to be used in chunks to avoid the memory leak hitting the limit, so this is worth watching.

The next release will have a self contained loop (a lot like the JavaScript one provided by someone above) to check the value and reset the script; hopefully so you can kick it off and leave it over night to do its work.

Man, that is great! Thank u very much Savoy Shooters!

Big thanks for this script mr. Chris McKeem, you’re a life safer! My biggest problem is that prices in my store aren’t displayed when the indices aren’t refreshed. Using the backend, I get an 500 error. With your script it’s all done within 3 minutes.

SeL - 13 November 2008 04:37 AM

- If script executions times out, you should increase max_execution_time value

Can’t Magento done Layered Nav Refresh in few steps, if here is a too small limit of exec time?

When i try to done refresh of “Layered Navigation Indices” it’s take some time, and then 500 error, my Apache wrote to log:
[Thu Jul 2 17:05:48 2009] [error] [client x.x.x.x] Premature end of script headers: /www/project/www/htdocs/products/index.php

I search forum, but none of advices help me. Is someone have similar problem?

@Paul: Very welcome.

Ogonkov: Try following this http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-deux/ The magneto Built in system “really” needs to do it in steps; but at theyre targetting enterpries solutions as well as ZenCart size users they dont seem too arsed.

Savoy Shooters - 28 March 2009 11:31 AM

Here is the code I started using if it helps. Posted by Chris McKee at:

http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-deux/

It has worked wonders for me and also cuts down on memory used, which is great for shared hosting. Takes awhile though but can be adjusted through the settings he mentions on his site.

<?php
//################################
//# Magento Layered Nav QuickFix-b
//################################
//# @author: Chris McKee
//# @licence: GNU, basically do what you like but
//#           leave my somewhere
//################################
ob_implicit_flush(true); //Saves having to flush manually
set_time_limit(0); //Set time limit to unlimited, though we shouldnt need to
ignore_user_abort();
 
require_once("app/Mage.php");
umask(0);
 
$total = 2000; //How many records do we have (I'll make this automatic next time)
$hm = 50; // How many records to process at once?
 
$countFile = "cnt";
$store = null;
$count = null;
 
$store = Mage::app()->getStore('1');
 
//Create a count file if we dont have one
if((file_exists($countFile))&&(filesize($countFile)>0)){
        $fh = fopen($countFile, 'r');
        $count = fread($fh,filesize($countFile)) + 1;
        fclose($fh);
}
else
{
    $fh = fopen($countFile, 'w') or die("can't open file");
    $count = 0;
    fwrite($fh, $count);
    fclose($fh);
}
 
//If we havent started counting yet clear cache.
if($count == 0)
{
    Mage::app()->cleanCache();
    //Clear the database flag from any previous "normal" indexes that failed
    $flag = Mage::getModel('catalogindex/catalog_index_flag')->loadSelf();
    if ($flag->getState() == Mage_CatalogIndex_Model_Catalog_Index_Flag::STATE_RUNNING) {
        $flag->delete();
    }
}
 
$max = $count+$hm;
 
for($i = $count; $i <= $max; $i++)
{
    Mage::getSingleton('catalogindex/indexer')->plainReindex($i, null, $store);
    echo("\n #$i#CURRENT MEMRY USAGE = ".memory_get_usage().";\n</br>");
    //Write where were up to
    $fh = fopen($countFile, 'w') or die("can't open file");
    fwrite($fh, $i);
    fclose($fh);
    sleep(1); //Pause a second, for less use u_sleep(microseconds)
}
 
?>

<script>
<!--

/*
Auto Refresh Page with Time script
By JavaScript Kit (javascriptkit.com)
Over 200+ free scripts here!
*/

//enter refresh time in "minutes:seconds" Minutes should range from 0 to inifinity. Seconds should range from 0 to 59
var limit="0:03"

if (document.images){
var parselimit=limit.split(":")
parselimit=parselimit[0]*60+parselimit[1]*1
}
function beginrefresh(){
if (!document.images)
return
if (parselimit==1)
[removed].reload()
else{ 
parselimit-=1
curmin=Math.floor(parselimit/60)
cursec=parselimit&#x60;
if (curmin!=0)
curtime=curmin+" minutes and "+cursec+" seconds left until page refresh!"
else
curtime=cursec+" seconds left until page refresh!"
window.status=curtime
setTimeout("beginrefresh()",1000)
}
}

window.onload=beginrefresh
//-->
</script>

I’m running this fix from Chirs McKee posted by Savoy Shooters and it is successfully rebuilding by cache including layered naviagtion.  Great!!

What i’m wondering at this point is if the script ever stops.  I’m up to interation 5200 and it is still going everytime I refresh the page.  I’m wondering how I can check if the script has done it’s job.  I am looking through the database tables guessing that the catalogindex_ tables should be increasing records if the script is still doing something, however, I’m not seeing any changes.

Can anyone shed any light?

once you hit your selected total of records its done; Any indexing after that will be really fast as the index method will check; see sod all and move to the next.
I’ll change the file in the next release as total will be auto-calculated and auto-terminating.

Ta

Chris

pcdevils - 09 July 2009 11:20 AM

@Paul: Very welcome.

Ogonkov: Try following this http://chrismckee.co.uk/magento-commerce-layered-navigation-cache-error-part-deux/ The magneto Built in system “really” needs to do it in steps; but at theyre targetting enterpries solutions as well as ZenCart size users they dont seem too arsed.

Thanks, looks like it’s a only way to work around it, i don’t get any support from dev. team.

I will try to use script above, and have no success.

1. I download script
2. Rename it to cm_cache.php
3. Change $total = 11332, $hm = 10
4. Run it
5. After about 5 minutes of run script is get 500 error, apache wrote:
[Fri Jul 17 12:59:33 2009] [error] [client x.x.x.x] Premature end of script headers: /www/project/www/htdocs/products/cm_cache.php

File cnt is have 0 inside.

max_execution_time 510

What i done wrong?