netroby

WP-log-robots is a wordpress plugin, we could used it to loged the search engine robots activity.
after you had installed it . the WP-log-robots will log each robots activity and wrote it to log file .
if your site was : http://www.netroby.com. then the log file would be location at : http://www.netroby.com/robots_log.txt
<h3>Manual Installation</h3>
In case you want to make all steps manually:
<ol>
<li>
Upload to your plugins folder, usually <code>wp-content/plugins/</code> and unzip the file, it will create a <code>wp-content/plugins/wp-log-robots/</code> directory.
</li>
<li>
Make <code>wp-content</code> directory writeable by the web server.
</li>
<li>
Activate the plugin on the plugin screen.
</li>
</ol>
<h3>More Info</h3>
For more info, please visit http://www.netroby.com/ .
(For feedback/comments, please send an e-mail to: netroby[at]netroby.com ).

WP-log-robots

WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.

Samir Shah

This plugin will issue a HTTP <code>410</code> response to requests for articles that no longer exist on your blog. When you delete a post or page, it records the URL for that page and issues a <code>410</code> response when that URL is requested. You can also manually manage the list of obsolete URLs.
The <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.11" rel="nofollow ugc">HTTP Specification</a> defines the <code>410</code> response header for use when a resource has been permanently removed. It informs robots visiting your site that the requested content has gone, and that they should stop trying to access it.
If you come across any bugs or have suggestions, please use the plugin support forum. I can’t fix it if I don’t know it’s broken! Please check the <a href="https://wordpress.org/extend/plugins/wp-410/faq/" rel="ugc">FAQ</a> for common issues.

410 for WordPress

A plugin that sends HTTP 410 (Gone) responses to requests for articles that no longer exist on your blog.

WP-log-robots vs 410 for WordPress