netroby

WP-log-robots is a wordpress plugin, we could used it to loged the search engine robots activity.
after you had installed it . the WP-log-robots will log each robots activity and wrote it to log file .
if your site was : http://www.netroby.com. then the log file would be location at : http://www.netroby.com/robots_log.txt
<h3>Manual Installation</h3>
In case you want to make all steps manually:
<ol>
<li>
Upload to your plugins folder, usually <code>wp-content/plugins/</code> and unzip the file, it will create a <code>wp-content/plugins/wp-log-robots/</code> directory.
</li>
<li>
Make <code>wp-content</code> directory writeable by the web server.
</li>
<li>
Activate the plugin on the plugin screen.
</li>
</ol>
<h3>More Info</h3>
For more info, please visit http://www.netroby.com/ .
(For feedback/comments, please send an e-mail to: netroby[at]netroby.com ).

WP-log-robots

WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.

chrisguitarguy

George Pattichis

WordPress, by default, includes a simple robots.txt file that’s dynamically generated from within the WP application. This is great, but how do you easily change the content?
Enter WP Robots Txt, a plugin that adds an additional field to the “Reading” admin page where you can do just that. No manual coding or file editing required!
Simply visit https://your-site.com/wp-admin/options-reading.php and you can control the contents of your https://your-site.com/robots.txt
<a href="https://wordpress.org/plugins/wp-robots-txt/#developers" rel="ugc">Changelog</a>

WP Robots Txt

WP Robots Txt Allows you to edit the content of your robots.txt file.

WP-log-robots vs WP Robots Txt