CeeWP

CeeWP Robots.txt Manager is a lightweight and user-friendly plugin that allows you to easily manage your WordPress site’s virtual robots.txt file. Add custom rules without creating physical files, with full control over search engine access to your content.
Why Choose Robots.txt Manager?
<ul>
<li>No File Management: Works with WordPress’s built-in virtual robots.txt system</li>
<li>Two Operating Modes: Choose between appending to or replacing default rules</li>
<li>User-Friendly Interface: Clean admin panel with helpful examples</li>
<li>WordPress Standards: Follows all WordPress coding and security standards</li>
<li>Automatic Cleanup: Removes all data when uninstalled</li>
</ul>
Key Features:
<ul>
<li>Simple interface to add custom robots.txt rules</li>
<li>Replace Mode: Option to completely replace WordPress default robots.txt with custom rules</li>
<li>Append Mode: Add custom rules to existing WordPress robots.txt (default behavior)</li>
<li>Dynamic Textarea: Automatically adjusts height based on content</li>
<li>Live Preview: Direct link to view your robots.txt file</li>
<li>Smart Warnings: Alerts when search engines are discouraged</li>
<li>Privacy Respect: Honors WordPress privacy settings</li>
<li>Examples Included: Ready-to-use code snippets for common scenarios</li>
<li>Professional UI: Modern interface with WordPress dashicons</li>
</ul>
Perfect for:
<ul>
<li>SEO Professionals: Fine-tune search engine crawling behavior</li>
<li>Website Owners: Block access to private directories and files</li>
<li>Developers: Prevent indexing of development or staging content</li>
<li>E-commerce Sites: Control bot access to sensitive areas</li>
<li>Content Creators: Manage sitemap declarations and crawl rates</li>
</ul>
How It Works:
The plugin adds a new menu item under Settings > Robots.txt Manager where you can easily add your custom rules. Choose between two modes:
<ul>
<li>Append Mode (Default): Your custom rules are added to WordPress’s default robots.txt</li>
<li>Replace Mode: Your custom rules completely replace WordPress’s default robots.txt</li>
</ul>
The interface includes helpful examples, real-time textarea sizing, and a direct link to view your live robots.txt file.
<h3>Privacy</h3>
This plugin does not collect, store, or transmit any personal data. All settings are stored locally in your WordPress database and are only used to generate your robots.txt file.
<h3>Credits</h3>
Developed with ❤️ by CeeWP – https://ceewp.com/
Special thanks to the WordPress community for continuous inspiration and feedback.

CeeWP Robots.txt Manager

Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!

lastsplash (a11n)

Protect Your Content from AI Scraping
This plugin helps you prevent AI crawlers from using your content as training data for their products. By updating your site’s <code>robots.txt</code>, it blocks common AI crawlers and scrapers, aiming to protect your content from being used in the training of Large Language Models (LLMs).
<h3>Features</h3>
<h3>Blocks AI Crawlers</h3>
Includes:
<ul>
<li>OpenAI – Blocks crawlers used for ChatGPT</li>
<li>Google – Blocks crawlers used by Google’s Gemini AI products</li>
<li>Facebook / Meta – Used for Facebook’s AI training</li>
<li>Anthropic AI – Blocks crawlers used by Anthropic </li>
<li>Perplexity – Block crawlers used by Perplexity</li>
<li>Applebot – Blocks crawlers used by Apple</li>
<li>… and more!</li>
</ul>
<h3>Experimental Meta Tags</h3>
The plugin adds the “noai, noimageai” directive to your site’s meta tags, instructing AI bots not to use your content in their datasets. Please note that these tags are experimental and have not been standardized.
<h3>Custom robots.txt Rules</h3>
Have custom entries for your robots.txt file? You can now add them directly through the plugin!
<h3>Usage</h3>
After activation, the plugin will automatically update your <code>robots.txt</code> and add the necessary meta tags. No further configuration is required, but you can check the settings page for a full list of blocked crawlers.
<h3>Limitations</h3>
While this plugin aims to block specified crawlers, it cannot guarantee complete protection against all forms of scraping, as some bots may disregard <code>robots.txt</code> directives.
<h3>Support</h3>
For questions or support, <a href="https://wordpress.org/support/plugin/block-ai-crawlers/" rel="ugc">please post on the forums</a> or <a href="https://github.com/bobmatyas/wp-block-ai-crawlers/issues" rel="nofollow ugc">on GitHub</a>.

Block AI Crawlers

Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.

CeeWP Robots.txt Manager vs Block AI Crawlers