Magento

All,

I’ve recently noticed that I’m receiving a large number of “hits"(about 3K a night) from my server’s internal IP from something called MagentoCrawler. These are displaying themselves in my /var/log/httpd/access_log as follows.

192.168.10.10 - - [22/Aug/2012:03:31:23 -0500] “GET /path/to/file.php HTTP/1.1” 301 369 “-” “MagentoCrawler”

I’m wondering if this is something Magento does internally or if it’s something malicious? Any help you can give is appreciated.

Sincerely,

Bob

Hi Bob,
did you install a full page cache or something like this which could come with a crawler?
Magento has by default no crawler installed and the requests are coming from a local system not from outside.

The crawler probably requests all urls from the database table “core_url_rewrite” where all catalog, product detail and cms pages are stored.