chrisguitarguy

George Pattichis

<p>WordPress, by default, includes a simple robots.txt file that’s dynamically generated from within the WP application. This is great, but how do you easily change the content?</p>
<p>Enter <strong>WP Robots Txt</strong>, a plugin that adds an additional field to the “Reading” admin page where you can do just that. No manual coding or file editing required!</p>
<p>Simply visit https://your-site.com/wp-admin/options-reading.php and you can control the contents of your https://your-site.com/robots.txt</p>
<p><a href="https://wordpress.org/plugins/wp-robots-txt/#developers" rel="ugc">Changelog</a></p>


WP Robots Txt

WP Robots Txt Allows you to edit the content of your robots.txt file.

Viktor Szépe

<h4>This plugin needs more documentation!</h4>
<p>You can edit your robots.txt and add remote content to it.<br />
E.g. you have several sites and want to use a centralized robots.txt.</p>
<h4>Features</h4>
<ul>
<li>Include or exclude WordPress’ own robots.txt (core function)</li>
<li>Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output)</li>
<li>Include or exclude a remote text file (the common part)</li>
<li>Include or exclude custom records from the settings page (the site specific part)</li>
</ul>
<h4>Where is robot.txt?</h4>
<p>WordPress handles robots.txt as a virtual URL – just the same way as posts and pages.</p>
<p>So when you browse to <code>https://example.com/robots.txt</code> WordPress generates robots.txt on the fly.</p>
<h4>TODO</h4>
<ul>
<li>add more description here</li>
<li>add a video too</li>
<li>add an admin notice for subdir installs (robots.txt is useless in a subdir)</li>
<li>‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that</li>
</ul>
<h4>Links</h4>
<p>Development of this plugin goes on on <a href="https://github.com/szepeviktor/multipart-robotstxt-editor" rel="nofollow ugc">GitHub</a>.</p>


Multipart robots.txt editor

Customize your site's robots.txt and include remote content to it

WP Robots Txt vs Multipart robots.txt editor