Posting in the Magento forums has been disabled pending the implementation of a new and improved forum solution which should better serve the community.

For new questions please post at magento.stackexchange.com, the community-run support site for the Magento community. We will be providing updates on the new forum solution soon. For questions or concerns please email community@magento.com.

Magento Forum

Receiving large numbers of internal hits from “MagentoCrawler”. Is this normal? 
 
robertcowher
Jr. Member
 
Avatar
Total Posts:  2
Joined:  2012-06-20
 

All,

I’ve recently noticed that I’m receiving a large number of “hits"(about 3K a night) from my server’s internal IP from something called MagentoCrawler. These are displaying themselves in my /var/log/httpd/access_log as follows.

192.168.10.10 - - [22/Aug/2012:03:31:23 -0500] “GET /path/to/file.php HTTP/1.1” 301 369 “-” “MagentoCrawler”

I’m wondering if this is something Magento does internally or if it’s something malicious? Any help you can give is appreciated.

Sincerely,

Bob

 
Magento Community Magento Community
Magento Community
Magento Community
 
mgt_commerce
Member
 
Avatar
Total Posts:  66
Joined:  2009-06-06
 

Hi Bob,
did you install a full page cache or something like this which could come with a crawler?
Magento has by default no crawler installed and the requests are coming from a local system not from outside.

The crawler probably requests all urls from the database table “core_url_rewrite” where all catalog, product detail and cms pages are stored.

 
Magento Community Magento Community
Magento Community
Magento Community
Magento Community
Magento Community
Back to top