netroby

WP-log-robots is a wordpress plugin, we could used it to loged the search engine robots activity.
after you had installed it . the WP-log-robots will log each robots activity and wrote it to log file .
if your site was : http://www.netroby.com. then the log file would be location at : http://www.netroby.com/robots_log.txt
<h3>Manual Installation</h3>
In case you want to make all steps manually:
<ol>
<li>
Upload to your plugins folder, usually <code>wp-content/plugins/</code> and unzip the file, it will create a <code>wp-content/plugins/wp-log-robots/</code> directory.
</li>
<li>
Make <code>wp-content</code> directory writeable by the web server.
</li>
<li>
Activate the plugin on the plugin screen.
</li>
</ol>
<h3>More Info</h3>
For more info, please visit http://www.netroby.com/ .
(For feedback/comments, please send an e-mail to: netroby[at]netroby.com ).

WP-log-robots

WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.

Samir Shah

Matt Calvert

This plugin issues an HTTP <code>410</code> response for URLs corresponding to content that has been permanently removed from your site. Originally created by Samir Shah, now maintained by Matt Calvert. When a post or page is deleted, the plugin logs the old URL and returns a <code>410</code> response when that URL is requested. You can also manually manage the list of obsolete URLs.
The <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11" rel="nofollow ugc">HTTP Specification</a> defines the <code>410 Gone</code> response for resources that have been permanently removed. It informs search engines and crawlers that the content will not return, improving crawl efficiency and SEO clarity.
This plugin is actively maintained by Matt Calvert as a personal project, informed by previous professional experience with similar 410-handling logic. No proprietary or employer-owned code has been used.

HTTP 410 (Gone) responses

Sends HTTP 410 (Gone) responses for deleted content, telling search engines the page is permanently removed.

WP-log-robots vs HTTP 410 (Gone) responses