Magento - Robots.txt - is it still needed with 1.1.4? What should it be? - Search Engine Optimization - eCommerce Software for Growth

Skip to site navigation »
Skip to main content »
Skip to footer content »

Login

Login:

Password:

Remember password
Show my name in the online users list

Forgot password?
Register
Job Board

Magento Forum

From setting up your store to managing your products, pages and promotions to generating detailed reports, the Magento User Guide empowers the user to utilize the platform for all of its vast capacity.
Available in eBook and Print formats – Download Now!!!

Robots.txt - is it still needed with 1.1.4? What should it be?

John34 Total Posts: 33 Joined: 2009-05-31 Ignore user	@tapebrothers - Maybe I’m old school, but I don’t even bother with the Magento built in sitemap thing. The better way to do it is to talk with Googlebot yourself (i.e. tell it what you want it to know directly from Google). Just go to http://www.google.com/webmasters/ and sign up for a Google account (it’s free and well worth it). You’ll then have access to Google’s “Web Master Tools”. Use it. From there you can tell Google directly where your sitemap(s) is/are located, find broken links and problems with your website, and do all sorts of other stuff. That’s the way to go in my opinion. Signature John
	Posted: June 17 2009 \| top \| # 16

blueben Total Posts: 214 Joined: 2008-10-09 Charleston, SC Ignore user	IMO /media should be indexed, especially for retailers of popular brands; I am certain image searches lead to conversions. Signature Benjamin Marks • Blue Acorn • eCommerce Consulting
	Posted: June 25 2009 \| top \| # 17

blueben Total Posts: 214 Joined: 2008-10-09 Charleston, SC Ignore user	Our standard robots.txt (not .htaccess as I stated previously): `User-agent: * Disallow: /app/ Disallow: /catalog/product_compare/ Disallow: /catalog/product/view/id/ Disallow: /checkout/ Disallow: /contacts/ Disallow: /customer/ Disallow: /downloader/ Disallow: /index.php/ Disallow: /js/ Disallow: /lib/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /skin/ Disallow: /var/ Disallow: /wishlist/` Signature Benjamin Marks • Blue Acorn • eCommerce Consulting
	Posted: June 29 2009 \| top \| # 18

oldboy Total Posts: 71 Joined: 2008-08-12 Ignore user	When searching my own site on google i found this page being indexed> /catalogsearch/ajax/suggest/ its i list of all searches made on my site. I this directory should also be added to robots.txt Signature Running version: 1.4.1.1
	Posted: August 1 2009 \| top \| # 19

professionalav Total Posts: 4 Joined: 2009-05-10 Ignore user	When I have the URL rewirte turned on, I can not show up any pages other than home page, when I click any catagory, I got a error mesage said “Sorry, the page you were looking for could not be found” Please help me to resolve this problem. Thank you very much.
	Posted: August 8 2009 \| top \| # 20

moomoo Total Posts: 92 Joined: 2009-07-10 Ignore user	Thanks blueben, I also put this line in just after the first line: `Sitemap: http://www.yoursite.com/sitemap.xml` Signature • Magento Warehouse Pick Lists and Packing Sheets Molecule clothing - quality cargo shorts and cargo pants direct from manufacturer
	Posted: August 18 2009 \| top \| # 21

Jubba Total Posts: 35 Joined: 2009-05-07 London Ignore user	blueben - 29 June 2009 05:55 AM Our standard .htaccess: `User-agent: * Disallow: /app/ Disallow: /catalog/product_compare/ Disallow: /catalog/product/view/id/ Disallow: /checkout/ Disallow: /contacts/ Disallow: /customer/ Disallow: /downloader/ Disallow: /index.php/ Disallow: /js/ Disallow: /lib/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /skin/ Disallow: /var/ Disallow: /wishlist/` Why in your standard .htaccess and not robots.txt? I have domain aliases set up on a VPS and I am wondering whether in the robots.txt I can have the address of all my sitemaps as seperate domains in the one domain root? e.g. Sitemap: http://www.site1.com/sitemap1.xml Sitemap: http://www.site2.com/sitemap2.xml Sitemap: http://www.site3.com/sitemap3.xml would this work? Signature Brain surgery is probably easier than this......no one said it would be easy! http://www.tapstore.com
	Posted: November 4 2009 \| top \| # 22

RosesMiami.com Total Posts: 37 Joined: 2008-10-22 Ignore user	this is a great topic, but i still have some concerns, why ht acces ans not robots.txt, i need to setup my robots.txt, it just dont come by default in 1.4, thanks Signature http://www.ThePaginasWeb.com http://www.CosasPeruanas.com
	Posted: May 25 2010 \| top \| # 23

Magento Community

Magento Community

Back to top

Where can I add a rel="nofollow" to "View as: Grid List" and "Sort By" etc

Same content in same language but different TLD

© Copyright 2010 Magento Inc.
Privacy Policy|Terms of Service Magento Community Count
275565 users|1247 users currently online|346204 forum posts