Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Page 1 of 2
Core_url_rewrite bug: Massive amount of duplicate url’s for each product generated on index
 
snowcountry
Jr. Member
 
Total Posts:  16
Joined:  2009-08-11
 

I have been searching for a while but i havent found any real cases of people having this problem: Each time i re-generate my core url rewrite. It generates a new url rewrite with a extra number for almost each product. This might cause big duplicate content issues in google since each product is callable by multiple urls with different canonical urls each time.

Furthermore, my core_url_rewrite explodes with over 600k records for +/- 5k products.

In the attatched screenshot an image of one of my products having many urls.

Anyone have a solution for this? I\\\’ve had this problem with one of my customers as well, there, a db with 100k products generated an index of 10m records within a few days. I have found This mention of the same problem but no real solution.

I\\\’m digging into app/code/core/mage/catalog/model/url.php now to see if i can find the problem and make a quick fix. Sure hope this is solved in 1.8!

edit: bug only exists here for products where we have 2 products having the same url-key.

Magento version 1.7.2

Edit2: there is already a bug-report regarding this problem. No fix yet tough. Anyone have a fix?
+-/ 10 stores

Image Attachments
url-rewrite.JPG
 
Magento Community Magento Community
Magento Community
Magento Community
 
snowcountry
Jr. Member
 
Total Posts:  16
Joined:  2009-08-11
 

For the people interested.

In the file app/code/core/Mage/Catalog/Model/Url.php on line 820 there is this code:

$fullPath $requestPath.$suffix;
            if (
$this->_deleteOldTargetPath($fullPath$idPath$storeId)) {
                
return $fullPath;
            
}

This seems to check if the current url (fullpath) for the product is already in the url’s and is equal to the ID path found based on this request path.

In the case of a 2nd product with the same url key, $requestPath.$suffix is equal to the first product, not the second. Therefore this should be changed to:

$fullPath $existingRequestPath.$suffix;
     if (
$this->_deleteOldTargetPath($fullPath$idPath$storeId)) {
        
return $fullPath;
      
}

I haven’t tested this very thoroughly but it seems to be working. Als re-generating all the urls for my problem products after deleting all their records works fine. Re-generating the index after that again doesn’t create more rewrites so it fixes the initial problem. Further effects still need some testing. I’ll create a small moduleto fix this problem upgrade-proof later.

 
Magento Community Magento Community
Magento Community
Magento Community
 
markf
Sr. Member
 
Total Posts:  145
Joined:  2007-09-20
 

Thanks very much for the fix, I thought I was going crazy because I keep noticing that URLs are changing and I’m getting 404 pages in google results. This is a HUGE bug and potentially disastrous in terms of SEO. Really hope this is addressed in 1.8.

I do have one question about this though, I have about 2300 products in my store yet in the URL rewrite management area, it shows there are over 7000 rewrites. Is this normal or should the rewires match (or be close to) the actual amount of products? Can others share their products to rewrites ratio so I can compare?  Thanks.

 
Magento Community Magento Community
Magento Community
Magento Community
 
kusibe
Jr. Member
 
Total Posts:  4
Joined:  2012-02-06
 

many thanks to snowcountry!!
your fix is working and the url rewrite bug is history now wink

@markf
that’s normal to have more url rewrites than products, when you have your products in different categories and/or you have different store_views (languages). you can check by searching all url rewrites for a given product:

Catalog > URL Rewrite Management

Then filter for a specific product by entering the id path e.g. “product/2421”

 
Magento Community Magento Community
Magento Community
Magento Community
 
FD CITY
Member
 
Total Posts:  38
Joined:  2012-09-25
 

Hi,

I believe I am also having this problem.  I am keen to try your fix but I am worried about truncating the table.  If I do this and run the core_url_rewrite from scratch then I\’m going to break external links to these products, correct?

I\’m a little bit wary of creating more of an issue than I have already!  However if this is true and the correct fix, this is a major bug in Magento and I\’m surprised that it doesnt seem to have been picked up sooner.

Craig

 
Magento Community Magento Community
Magento Community
Magento Community
 
markf
Sr. Member
 
Total Posts:  145
Joined:  2007-09-20
 

UPDATE:

After apply the fix a few days ago, the problem returned so the fix does NOT work. After a reindex, a whole bunch of my products URLS were changed again. This is a huge bug and is devastating for SEO. Every time google crawls and indexes your site, magento goes and switches the urls a week later.

 
Magento Community Magento Community
Magento Community
Magento Community
 
FD CITY
Member
 
Total Posts:  38
Joined:  2012-09-25
 

Ah, that’s bad news!

Is there no way to elevate this issue with their devs?

 
Magento Community Magento Community
Magento Community
Magento Community
 
summitecommerce
Jr. Member
 
Total Posts:  1
Joined:  2013-04-27
 
markf - 22 June 2013 02:25 PM

Thanks very much for the fix, I thought I was going crazy because I keep noticing that URLs are changing and I’m getting 404 pages in google results. This is a HUGE bug and potentially disastrous in terms of SEO. Really hope this is addressed in 1.8.

I do have one question about this though, I have about 2300 products in my store yet in the URL rewrite management area, it shows there are over 7000 rewrites. Is this normal or should the rewires match (or be close to) the actual amount of products? Can others share their products to rewrites ratio so I can compare?  Thanks.

Hi markf, bad news. I still have the same problem. However, it’s significantly less big, i have much less new url rewrites after a refresh. I just refreshed my index 3 times and i have no new rewrites (on about 40k total). So there seems to be a ‘leak’ somewhere else as well.. I’m diving into it later. Hope it’s fixed in 1.8 beacause it’s a huge bug..

(edit: this is snowcountry under another name because i couldn’t remember my passw.)

 
Magento Community Magento Community
Magento Community
Magento Community
 
chridster007
Jr. Member
 
Total Posts:  2
Joined:  2012-03-07
 

I’ve got this issue with Enterprise 1.12.0.2,

We re-index each morning and it generates an extra 3,000 +- urls each night (we are approaching 1 million urls)

I’ve been on with their support before it went down. They released a patch for me which is working no duplicate urls are created when I run the index… but....  when I truncate the core_url_rewrite table (which I have to do to get rid of the 700,000+ duplicate urls) then re-index it breaks leaving the table empty, and every product and category unreachable, so I have to restore the core_url_rewrite table.

Once I get a fix from Magento that completely works, I’ll update this post, and explain what I had to do.

Good luck all

 
Magento Community Magento Community
Magento Community
Magento Community
 
FD CITY
Member
 
Total Posts:  38
Joined:  2012-09-25
 

Hi,

Just checking in to see if there’s any development on this?

I’m using this extension to reduce the number of items indexed which means I can do a re-index in just a few minutes, which is fantastic: http://www.magentocommerce.com/magento-connect/dn-d-patch-index-url-1364.html

I’m still concerned that I have a bloated core_url_rewrite table though. 

Mostly I’m concerned that if I truncate the table, all pages indexed by google and inbound links are going to become 404 errors as the site is still young and there’s a lot of changes being made.

Any advice appreciated. smile

 
Magento Community Magento Community
Magento Community
Magento Community
 
Jahnni
Jr. Member
 
Total Posts:  9
Joined:  2010-05-26
 

hi there,
in app/code/core/Mage/Catalog/Model/Url.php around line 807 it tries to “Check if existing request past can be used”, but it only checks it if the product doesn’t have an url key (which is rare - when does it happen? for new products? if so we’ll never get through this code as long as a new product doesn’t have a rewrite yet (line 797)). That being said i suggest to change line 809 from

if ($product->getUrlKey() == '' && !empty($requestPath)
                && 
strpos($existingRequestPath$requestPath) === 0
            
)

to

if (!empty($requestPath)
                && 
strpos($existingRequestPath$requestPath) === 0
            
)

do you see any side effect ?
yoh
G

 
Magento Community Magento Community
Magento Community
Magento Community
 
paulcook
Jr. Member
 
Total Posts:  1
Joined:  2013-03-04
 

+1 for Jahnni\s solution instead of snowcountry’s—they both seem to work (for me at least), but Jahnni’s seems more in the spirit of the code, and is likely to be slightly more efficient too.

Note, too, that this fixes only the issue of duplicate product names creating additional rewrites. We also had an issue with a category showing the same behaviour: we had renamed a category A to B, and then created a new category called A again. Due to the first renaming, the URL key for A pointed to B, and the new A also displayed the behaviour of a new rewrite per reindex. In this case, however, we didn\’t solve with code—rather, we removed the original A->B rewrite and cleaned up everything, which is a better result for clean URLs.

A useful trick is to view the URL rewrites in the admin interface, sorted by the \"ID\" column descending. Reindex, and then reload the page, and see what new rewrites have been created—this flagged the above category issue for us.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ali1neil
Jr. Member
 
Total Posts:  2
Joined:  2011-05-18
 

just noticed my store had 17million records in core_url_rewrite, for a magento 1.8.0 install with 16k skus and 6 store views

a rather nasty bug indeed ..

any official updates on this ?

 
Magento Community Magento Community
Magento Community
Magento Community
 
freshfromi
Jr. Member
 
Total Posts:  8
Joined:  2012-02-22
 

I don’t know what happened with my store, but I was looking for a way to clean up the URL rewrite also. We have only about 500 or so SKU’s, but the rewrites were up over 6500. Not much when I read about so many other stores having 100,000+.
However, as an Admin one is always looking to clean things up…

Anyway, On an unrelated issue, I was trying to find a way to do this and I wanted to delete a category that I use for displaying all products on the home page.... I wanted to send that same link from the home page to the ‘default category’ instead, but I was not having luck making this work.

I went to System - Manage Stores - Main Website Store Under the Root category drop down, I noticed that there were 2 listings for default category. I switched to the other default category and saved it. Checked the home page and now all products showing in the featured items and new items blocks were different. I checked URL rewrite page and those 5500 rewrites were now at about 1500. all good, but also some links on the home page are not working.

Not wanting to do anything that I was unsure how to fix, I reversed the process and changed back to the other ‘default category’ and saved.  Now again I went back to the rewrite management page and the 5500 are now at 3100.

Several links on the front page are not working, so I have to fix those, but otherwise everything seems to be happening fine. And apparently URL’s rules are much less.

Anyone have any idea what happened?

Robert

 
Magento Community Magento Community
Magento Community
Magento Community
 
hydn
Jr. Member
 
Total Posts:  12
Joined:  2012-08-12
 

My Url.php code on Magento 1.7 is:

if ($product->getUrlKey() == \'\' && !empty($requestPath)
                && 
strpos($existingRequestPath$requestPath) !== false
            
)

Does this fix still apply? Is it safe to edit that and change it to....

if (!empty($requestPath)
                && 
strpos($existingRequestPath$requestPath) !== false
            
)

We have over 1.6 GB of core_url_rewrites. :/

Thanks.

 
Magento Community Magento Community
Magento Community
Magento Community
 
ilianandreev
Jr. Member
 
Total Posts:  1
Joined:  2013-07-03
 

I can confirm the same bug with my shops. Using Magento > 1.7. As snowcountry pointed this is because:

edit: bug only exists here for products where we have 2 products having the same url-key.

The solution might be to append skus at the end of the url-key to make it unique and avoid these extra rewrites.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top
Page 1 of 2