netroby

WP-log-robots is a wordpress plugin, we could used it to loged the search engine robots activity.
after you had installed it . the WP-log-robots will log each robots activity and wrote it to log file .
if your site was : http://www.netroby.com. then the log file would be location at : http://www.netroby.com/robots_log.txt
<h3>Manual Installation</h3>
In case you want to make all steps manually:
<ol>
<li>
Upload to your plugins folder, usually <code>wp-content/plugins/</code> and unzip the file, it will create a <code>wp-content/plugins/wp-log-robots/</code> directory.
</li>
<li>
Make <code>wp-content</code> directory writeable by the web server.
</li>
<li>
Activate the plugin on the plugin screen.
</li>
</ol>
<h3>More Info</h3>
For more info, please visit http://www.netroby.com/ .
(For feedback/comments, please send an e-mail to: netroby[at]netroby.com ).

WP-log-robots

WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.

radgh

This simple and lightweight plugin adds a “Hide from search engines” checkbox above the publish box to pages. By 
ticking this box, a meta tag will be placed into the <code><head></code> section of your page specifying that robots should 
not index the page.
By default this only applies to pages. You may extend this functionality to posts or other custom post types with a small bit of code. See the FAQ’s tab for further instructions.

Noindex Pages

Ask search engines not to index individual pages by checking an option in the publish post box.

WP-log-robots vs Noindex Pages