Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Hiding Layered Navigation from Search Engines
 
FirmWebDesign
Jr. Member
 
Avatar
Total Posts:  1
Joined:  2013-04-19
United Kingdom
 

Hi all,

Recently we were concerned about content duplicates in magento shop caused by layered navigation. It is known, that crawlers shouldn’t see content not visible for browsers with disabled cookies. After research we have used such code for preventing showing the layered navigation when cookies are turned off.
What do you think about this code? Maybe you have some suggestions? Is it a good approach to the problem?
You can see the layered nav on http://treasurion.com/gemstones/.

<?php
$stateHtml 
$this->getStateHtml();
$filtersHtml '';
foreach (
$_filters as $_filter{
    
if ( $_filter->getType() == 'catalog/layer_filter_price' ) continue;
    if (
$_filter->getType() == "catalog/layer_filter_category" && !Mage::getStoreConfig('shoppersettings/catalog/categories_filter') ) continue;
    if (
$_filter->getItemsCount()) {
        $filtersHtml 
.= '<dt>'.$this->__($_filter->getName()).'</dt><dd>'.$_filter->getHtml().'<a href="#" class="shopby_more">'.$this->__('more...').'</a></dd>';
    
}
}
?>

<div id="filters-no-follow" class="block block-layered-nav"></div>

<?php
function prepare_for_echo($string{
$no_br 
trim(preg_replace('/\s+/'' '$string));
$no_slashes str_replace('\'', '\\\''$no_br);
return 
$no_slashes;
}
?>

<script>
function 
please_enable_cookies() {
var document.getElementById('filters-no-follow');
f.innerHTML '<div class="no-cookies-error">Enable cookies to choose filters.</div>';
}

function please_load_filters() {
var document.getElementById('filters-no-follow');
f.innerHTML '<?php if ( !empty($filtersHtml) || !empty($stateHtml) ): ?>'
'\n<div class="">'
'\n    <div class="block-title">'
'\n        <strong><span><?php echo prepare_for_echo($this->__('Shop By')); ?></span></strong>'
'\n    </div>'
'\n    <div class="block-content">'
'\n        <?php echo prepare_for_echo($this->getStateHtml()); ?>'
'\n        <?php if ($this->canShowOptions()): ?>'
'\n        <p class="block-subtitle"><?php echo prepare_for_echo($this->__('Shopping Options')); ?></p>'
'\n        <dl id="narrow-by-list">'
'\n            <?php echo prepare_for_echo($filtersHtml); ?>'
'\n        </dl>'
'\n        <?php endif; ?>'
'\n    </div>'
'\n</div>'
'\n<?php endif; ?>';
}

function are_cookies_enabled()
{
    
var cookieEnabled = (navigator.cookieEnabled) ? true false;

    if (
typeof navigator.cookieEnabled == "undefined" && !cookieEnabled)
    

        [removed]
="testcookie";
        
cookieEnabled = ([removed].indexOf("testcookie") != -1) ? true false;
    
}
    
return (cookieEnabled);
}

if(are_cookies_enabled()) {
please_load_filters
();
else {
please_enable_cookies
();
}
</script>
 
Magento Community Magento Community
Magento Community
Magento Community
 
RDalmagro
Jr. Member
 
Total Posts:  13
Joined:  2012-06-22
 

Is this solution better than using meta robots?

<meta name="robots" content="noindex,follow">

Can you explain why’?

 
Magento Community Magento Community
Magento Community
Magento Community
 
inMotionGfx
Jr. Member
 
Total Posts:  7
Joined:  2013-01-12
 

@FirmWebDesign, I’m quite interested to know how your code above is working out for you? Do Google and Bing indeed ignore your entire layered navigation now?

If this has been successful for you, I’d like to try and implement it as well.

Thanks.

Brendon

 
Magento Community Magento Community
Magento Community
Magento Community
 
Saggy
Sr. Member
 
Total Posts:  133
Joined:  2009-02-03
 

This below post should help, it was pain for us as well but it has sorted it out for us now. It was hard work though smile

http://www.magentocommerce.com/boards/viewthread/841410

Best of luck

Cheers
S

 
Magento Community Magento Community
Magento Community
Magento Community
 
inMotionGfx
Jr. Member
 
Total Posts:  7
Joined:  2013-01-12
 

Thanks for updating us Saggy,

I actually ended up developing my own solution for this, instead of using an extension. If anyone is interested in how I did it, here’s the gist of it:

Note: I just took this directly from my personal notes I kept, so you may need to fiddle with it to get it to work with your own theme.

All I did was add a conditional statement around the layered navigation block of code in my layered navigation template file that queries the user agent. For me this file was located here:

PathToThemeTemplateFiles/priceslider/slider_layered_nav.phtml

Add to line 1:

<?php if (strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "googlebot") || strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "bingbot") || strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "slurp") || strstr(strtolower($_SERVER['HTTP_USER_AGENT']), "msn")): ?>

<?php 
else: ?>

Add to last line:

endif; ?>

The search engines covered by the above user agent queries are Google, Bing, Yahoo and MSN.

You can test if it is working correctly by going to “Fetch as Google” under “Crawl” in Google Webmaster tools.

Once Google has fetched one of your category pages, click on the “Success” link to view the code as Google sees it. You should now no longer see the layered navigation code, or anything else you have chosen to hide.

Before implementing this strategy, I had around 20 000 pages of “duplicate” content thanks to the layered navigation. I’m happy to report that within a period of about 6 months, my number of indexed pages is down to less than 800 (I know, small site), and Google is no longer reporting any content issues under “HTML Improvements”.

Also, I can confirm a significant improvement in our rankings and traffic over this period, which confirms for me the value of avoiding duplicate content, even though Google claim they can figure it out on their own… yeah, right…

Oh, I should also mention that I used the “URL Paramenters” feature to further tell Google how to handle various parameters used by Magneto… just in case… wink For all layered navigation and most of the sorting parameters, I set it to “No URL’s”. If you are receiving significant traffic from Google organic search to your layered pages, you may want to use a different setting. For pagination (p) I set it to “Let Google Decide”, and for “limit” I set it to “Value = all”, as Google sometimes prefers the full page, not the first page.

I spent months on and off trying to find a solution for this, so I hope this will help save someone the time and hassle I had to invest.

All the best!

Brendon

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top