aicrawlerguard.com

<p>AI Crawler Guard detects popular AI-focused crawlers and lets you block or allow them from scraping your WordPress site. Everything runs locally—rules, decisions, logs, and history stay on your server.</p>
<ul>
<li>Built-in signatures for common AI bots, plus optional custom patterns</li>
<li>Global and per-crawler allow/block controls</li>
<li>Local crawler registry with first/last seen timestamps and hit counters</li>
<li>Lightweight logs with optional retention cap</li>
<li>No external calls or remote storage</li>
</ul>
<h3>Requirements</h3>
<ul>
<li>WordPress 5.8 or newer</li>
<li>PHP 7.4+</li>
<li>Works with Classic Editor, Gutenberg, WooCommerce, and major managed hosts</li>
<li>Respects wp-admin, wp-login, REST API, cron, and CLI contexts automatically</li>
</ul>
<h3>Performance</h3>
<ul>
<li>Detection runs once per front-end request (small O(n) UA scan)</li>
<li>No blocking of social preview bots (Facebook, Meta, Twitter, LinkedIn) by default</li>
<li>Assets load only on plugin admin pages and stay under 10 KB</li>
<li>Logs stay capped by your configured limit with auto-pruning</li>
</ul>
<h3>Privacy</h3>
<p>All detection logic and data storage live entirely inside WordPress. No data is sent to third-party services in v1.</p>


AI Crawler Guard

Detect and control AI crawler access with local logging and history.

apasionados

<p>This plugin adds lines to the virtual robots.txt file that WordPress creates automagically if the file is not present physically on the server to block the Common Crawl bot (CCBot) that is used to train AI algorithms like ChatGPT (in version 3.5 it makes up 60% of the training data mix) to crawl websites. Here is the <a href="https://commoncrawl.org/big-picture/frequently-asked-questions/" rel="nofollow ugc">information about the Common Crawl bot (CCBot)</a>.</p>
<blockquote>
<p>Please be aware of what this plugin does exactly: If you activate the plugin it will add some lines to the robots.txt file to tell Common Crawl bots to <strong>not to crawl and index</strong> parts of this website.</p>
</blockquote>
<p><strong>Please only activate this plugin if you know what you’re doing</strong>.</p>
<h4>What can I do with this plugin?</h4>
<p>This plugin adds the necessary lines to the virtual robots.txt file that WordPress creates automagically to block the Common Crawl bot (CCBot) is used to train AI algorithms like ChatGPT to crawl websites.</p>
<h4>What ideas is this plugin based on?</h4>
<p>Our plugin to <a href="https://wordpress.org/plugins/block-chat-gpt-via-robots-txt/" rel="ugc">block the OpenAI ChatGTP bot</a> after the launch of the OpenAI ChatGPT plugins that crawl websites.</p>
<h4>What is the robots.txt file?</h4>
<p>The robots.txt file is a plain text file located at the root folder of a domain (or subdomain) which tells web crawlers (like Googlebot) what parts of the website they should access and index.</p>
<p>The first thing a search engine crawler looks at when it is visiting a page is the robots.txt file and it controls how search engine spiders see and interact with the web pages.</p>
<h4>System requirements</h4>
<p>PHP version 7.4 or greater.<br />
We require 7.4 or higher because we believe that everybody should be running a modern PHP version. When releasing this plugin <a href="https://wordpress.org/about/requirements/" rel="ugc">WordPress recommends PHP 7.4 or higher</a>. We would recommend 8.0 or higher as 7.4 has no security updates since 28 Nov 2022.</p>
<h4>How to get this plugin in your Language!</h4>
<p>The first release is avaliable in English and Spanish. In the “languages” folder we have included the necessary files to translate this plugin.</p>
<p>If you would like the plugin in your language and you’re good at translating, please use the <a href="https://translate.wordpress.org/" rel="nofollow ugc">native WordPress Translation</a> functionality.</p>
<p>New to Translating a plugin? First read through the <a href="https://make.wordpress.org/polyglots/handbook/tools/glotpress-translate-wordpress-org/" rel="nofollow ugc">Translator Handbook</a>, then select your locale at <a href="https://translate.wordpress.org/" rel="nofollow ugc">Translating WordPress</a> and finally go to the <a href="https://translate.wordpress.org/projects/wp-plugins/block-ecommerce-assets-via-robots-txt/" rel="nofollow ugc">translation page for this plugin</a> to translate it.</p>
<h4>Further Reading</h4>
<p>You can access the description of the plugin in Spanish at: <a href="https://apasionados.es/blog/" rel="nofollow ugc">Block Common Crawl bot (CCBot) via robots.txt en español</a>.</p>
<h3>Contact</h3>
<p>For further information please send us an <a href="https://apasionados.es/contacto/index.php?desde=wordpress-org-block-chatgpt-assets-via-robots-txt" rel="nofollow ugc">email</a>.</p>


Block Common Crawl via robots.txt

Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI algorithms like ChatGPT through the WordPress virtual robots.txt file

AI Crawler Guard vs Block Common Crawl via robots.txt