WordPress crawler plugins

A lightweight plugin that protects email addresses from email-harvesting robots, by encoding them into decimal and hexadecimal entities.

Virtual Robots.txt automatically creates a robots.txt file for your site. Your robots.txt file can be easily edited from the plugin settings page.

Robots.txt for WordPress

Replace the default WordPress robots.txt workflow with a smarter, structured version you can preview before publishing, with Free, Pro, and Premium ed …

Spider Analyser是一款用于跟踪WordPress网站各种搜索引擎蜘蛛爬行日志的插件，并进行详细的蜘蛛爬行数据统计、蜘蛛行为分析、蜘蛛爬取分析及伪蜘蛛拦截等。

Prevent bots from creating accounts by blacklisting domains and usernames and present people with a human friendly security question.

Encodes mailto: links and e-mail addresses with JavaScript to stifle webcrawlers. Automatically turns plaintext e-mails into (enkoded) links.

Provide the easy managment of your robots.txt from admin side. It propose you the advanced then standard robots.txt content too.

The "Advanced Robots.txt Optimizer & Editor" WordPress plugin enhances website functionality, SEO and traffic management by allowing use …

With this plugin you can control the text which will be shown as description on the search engine results pages.

The Crawl Optimization Plugin is designed to help optimize your website’s crawl rate by removing unnecessary links and information from your HTTP head …

This plugin gives you the ability to set Meta Keywords in WordPress. Under pages it will registrate a separate taxonomy for keywords.

Marquee image crawler is a continuous scrolling image plugin. This plugin crawls the images left or right.

An easy-to-use plugin that generates and configures a proper robots.txt file, essential for effective search engine optimization (SEO).

Mass ping tool for SEO plugin is one of the fastest ways to get indexed by Google. After pinging, Google bots, crawlers will visit your website and in …

Real-Time Bot Protection. Anti Bots (robots), crawlers and spiders also from botnets. Increase security and avoid server overload and bandwidth.

Blocks the archive.org bots through the WordPress virtual robots.txt file.

Counts the visits from web spiders, crawlers and bots in your blog, with ability to get the blog rankings.

Blocks the ChatGPT-User bot that is used by plugins in ChatGPT to crawl websites through the WordPress virtual robots.txt file

Blocks some WooCommerce assets and Search pages that should not be indexed through robots.txt

Warm your WordPress cache with scheduled crawling, cache preloading, URL source control, logs, diagnostics, and WP-CLI.

Easily block over 30 of the most common AI user-agents known for crawling websites, keeping unauthorized bots at bay.

Powerful cloud based search engine for your website. Search as you type, real-time index updates, and embeddable UI. Free for small websites.

Protect email address from being collected by web crawlers.

Save time and resources by letting our plugin automate your blog.

Blocks some Feed and Comment resources in order to avoid spam bots

Blocks the Common Crawl bot (CCBot) that gathers data that is used to train AI algorithms like ChatGPT through the WordPress virtual robots.txt file

A lightweight, privacy-friendly WordPress cache warmup tool. No external services, no drama.

DeTx uses a global database of attackers (Bots) and also includes Artificial Intelligence and data analysis to identify Bots.

This plugin detects a few bots (e.g. Google, Google ads, Alta vista etc) and sends an email based on interval set by you. you can choose what format e …

Display the results of your Page Modified crawls right within your WordPress Admin.

rexCrawler is a regular expression web-crawler plugin. Able to parse websites, save data and show them on pages.

With SEObot Monitor for Googlebot you will be able to dump the server logs to Google Analytics, easily and automatically. This will allow you to dete …

Dynamic robots.txt for Multisite! Change parameters, or don't, for each site of your network without losing the defaults from WordPress Includes.

A custom plugin to log search engine bot visits and analyze their crawl budget directly inside WordPress (no external services required).

Powerful SEO auditor by Pixellize. Crawls your site to fix broken links, missing meta tags, and unoptimized images.

The professional architect for your llms.txt file. Control how AI agents (ChatGPT, Claude, Perplexity) consume your website content with a structured, …

A lightweight plugin to edit or create robots.txt directly from your WordPress dashboard. Developed by GoBookMart.

Show Password Required Post Content For Search Engine Bot

Track AI crawler visits such as GPTBot, ClaudeBot, and PerplexityBot server-side and visualize them in WordPress.

Detect and control AI crawler access with local logging and history.

Tracks and logs AI, Search and Social bots including ChatGPT, Perplexity, Googlebot, Bing and Meta crawlers.

A powerful website crawler and monitoring tool for WordPress.

A lightweight plugin that logs visits from common AI/LLM crawler user-agents and shows a simple report in wp-admin.

Block, log and monetise AI crawlers directly from your WordPress dashboard.

Verify AI crawlers with HTTP Signatures and enforce allow/deny/teaser policies; also serves llms.txt and AI-ready feeds.

Detect LLM user agents, enforce Google parity, and serve enhanced content through a configurable content service.

Official Trackbase Analytics plugin for WordPress. Add your tracking ID and enable server-side crawler analytics in minutes.

Monitor known AI crawlers and AI assistant fetches, with daily rollups, CSV exports, weekly summaries, and page briefs.

Serve every WordPress post and page as a clean .md file with full SEO frontmatter — discoverable by AI crawlers and search engines on the fly.

See which AI assistants — ChatGPT, Claude, Perplexity, Google and more — crawl your site, broken down by assistant, crawler, purpose and page.

Safely creates a virtual llms.txt dynamically. Features interactive bot targeting, map auto-syncing, and decoupled custom instructions.

Observe which AI bots visit which pages and when. WordPress implementation of the EdgeShaping framework.