WordPress crawlers plugins

Track crawlers, scrapers, LLM assistants, and AI agents on your website. Generate a robots.txt that blocks AI bots. Formerly Dark Visitors.

Tell AI (Artificial Intelligence) companies not to scrape your site for their AI products.

Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.

Control how AI crawlers use your content with Cloudflare Content Signals in robots.txt. Includes EU Directive 2019/790 rights reservation.

The easiest way to manage meta robots tag.

Advanced llms.txt generator with AI metadata, Schema.org types, canonical URLs, and differentiated licenses for optimal LLM indexing.

Easy way to discourage search engines from indexing only specific pages / posts with custom meta tags.

Easily manage your WordPress site's virtual robots.txt file with custom rules. No physical file needed!

Analyze how search engines and other bots crawl and understand your web page. The official PHP Wordpress plugin for log-hero.com.

Simply uses the HTTP_REFERER to determine if it's from 'Semalt', and blocks it with a nice 403 "Unauthorized" response.

Exports all entries of Seo Crawlytics and gives you the option to delete the entries that are older than 1 month.

Add some directives to your robots.txt file to keep your site safer

Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.

Protect your rights by reviewing and blocking AI bots from using your content without compensation.

Serves every published post and page as Markdown for AI agents and crawlers. No configuration, no API keys. Activate and it works.

Connect your WordPress site to the Supertab Connect platform for RSL license serving and enabling crawler authentication protocol.