netroby

<p>WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.</p>
<p>after you had installed it . the WP-log-robots will log each robots activity and wrote it to log file .</p>
<p>if your site was : http://www.netroby.com. then the log file would be location at : http://www.netroby.com/robots_log.txt</p>
<h3>Manual Installation</h3>
<p>In case you want to make all steps manually:</p>
<ol>
<li>
<p>Upload to your plugins folder, usually <code>wp-content/plugins/</code> and unzip the file, it will create a <code>wp-content/plugins/wp-log-robots/</code> directory.</p>
</li>
<li>
<p>Make <code>wp-content</code> directory writeable by the web server.</p>
</li>
<li>
<p>Activate the plugin on the plugin screen.</p>
</li>
</ol>
<h3>More Info</h3>
<p>For more info, please visit http://www.netroby.com/ .</p>
<p>(For feedback/comments, please send an e-mail to: netroby[at]netroby.com ).</p>


WP-log-robots

WP-log-robots is a wordpress plugin, we could used it to loged the search engine  robots activity.

Viktor Szépe

<h4>This plugin needs more documentation!</h4>
<p>You can edit your robots.txt and add remote content to it.<br />
E.g. you have several sites and want to use a centralized robots.txt.</p>
<h4>Features</h4>
<ul>
<li>Include or exclude WordPress’ own robots.txt (core function)</li>
<li>Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output)</li>
<li>Include or exclude a remote text file (the common part)</li>
<li>Include or exclude custom records from the settings page (the site specific part)</li>
</ul>
<h4>Where is robot.txt?</h4>
<p>WordPress handles robots.txt as a virtual URL – just the same way as posts and pages.</p>
<p>So when you browse to <code>https://example.com/robots.txt</code> WordPress generates robots.txt on the fly.</p>
<h4>TODO</h4>
<ul>
<li>add more description here</li>
<li>add a video too</li>
<li>add an admin notice for subdir installs (robots.txt is useless in a subdir)</li>
<li>‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that</li>
</ul>
<h4>Links</h4>
<p>Development of this plugin goes on on <a href="https://github.com/szepeviktor/multipart-robotstxt-editor" rel="nofollow ugc">GitHub</a>.</p>


Multipart robots.txt editor

Customize your site's robots.txt and include remote content to it

WP-log-robots vs Multipart robots.txt editor