Viktor Szépe

<h4>This plugin needs more documentation!</h4>
<p>You can edit your robots.txt and add remote content to it.<br />
E.g. you have several sites and want to use a centralized robots.txt.</p>
<h4>Features</h4>
<ul>
<li>Include or exclude WordPress’ own robots.txt (core function)</li>
<li>Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output)</li>
<li>Include or exclude a remote text file (the common part)</li>
<li>Include or exclude custom records from the settings page (the site specific part)</li>
</ul>
<h4>Where is robot.txt?</h4>
<p>WordPress handles robots.txt as a virtual URL – just the same way as posts and pages.</p>
<p>So when you browse to <code>https://example.com/robots.txt</code> WordPress generates robots.txt on the fly.</p>
<h4>TODO</h4>
<ul>
<li>add more description here</li>
<li>add a video too</li>
<li>add an admin notice for subdir installs (robots.txt is useless in a subdir)</li>
<li>‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that</li>
</ul>
<h4>Links</h4>
<p>Development of this plugin goes on on <a href="https://github.com/szepeviktor/multipart-robotstxt-editor" rel="nofollow ugc">GitHub</a>.</p>


Multipart robots.txt editor

Customize your site's robots.txt and include remote content to it

michael_greenberg

<p>PHPEnkoder is a port of the excellent <a href="http://hivelogic.com/enkoder" rel="nofollow ugc">Hivelogic<br />
Enkoder</a> to PHP and, more specifically,<br />
to WordPress. It is used to display text in a way that users can see<br />
and bots can’t.</p>
<p>The encoding system is directly and unabashedly stolen from the<br />
BSD-licensed source of Hivelogic Enkoder, which works by randomly<br />
encoding a piece of text and sending to the browser self-evaluating<br />
Javascript that will generate the original text. This works in two<br />
ways: first, a bot must first have a fairly complete Javascript<br />
implementation (in particular, it must have <code>eval</code>); second, the<br />
decoding process can be made arbitrarily computationally<br />
intensive. This is similar to the idea of charging computational<br />
payments to send e-mail, only this is actually implemented.</p>
<p>By default, PHPEnkoder scrambles e-mails in plaintext and in <code>mailto:</code><br />
links.  It additionally provides a shortcode for manual scrambling,<br />
used like so: <code>[enkode text="shown to non-JS browsers"]this will be<br />
scrambled[/enkode]</code>.</p>


PHPEnkoder

Encodes mailto: links and e-mail addresses with JavaScript to stifle webcrawlers.  Automatically turns plaintext e-mails into (enkoded) links.

Multipart robots.txt editor vs PHPEnkoder