Crawler Record
Crawler Record tells you the last time each of the most common search/chat bots visited (Google, ChatGPT, etc)- and the pages at which they looked.
Plugin info
Maintenance & Compatibility
Maintenance score
Actively maintained • Last updated 40 days ago
Is Crawler Record abandoned?
Likely maintained (last update 40 days ago).
Compatibility
Similar & Alternatives
Explore plugins with similar tags, and compare key metrics like downloads, ratings, updates, support, and WP/PHP compatibility.
Description
Crawler Record logs the last time specific user agents (for search and AI chat/LLMs) accessed your content including:
- Bing
- ChatGPT (OpenAI)
- Claude (Anthropic)
- Perplexity
- DuckDuckGo
You can view this information:
- Admin page: a grouped list of all crawlers, with last seen time, last page, and robots status for the crawler.
- Admin Bar (front-end): quick-glance status for the current singular or URL context (no dropdowns in wp-admin).
- Per Page/Post: From the edit screen on any page or post, you can see that last time each crawler visited that page.
Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. If Settings Reading “Discourage search engines” is enabled, all agents are shown as blocked with a prominent warning.
Performance-friendly by design: Write-throttling (default 10 minutes) and an auxiliary “last post ID per agent” record avoids heavy admin queries on large sites.
Privacy-friendly: Saves only bot visit timestamps and last URLs crawled — no personal data.
Learn how to use this plugin.
Highlights
- Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, and DuckDuckGo.
- Robots status computed from local robots.txt (physical or virtual) without outbound requests.
- Clear UI with grouped sections, microsecond timestamps, and a small diagnostics toggle showing the matched robots group and rule.
Known limitations
- robots.txt wildcards (
*) and end-of-line marker ($) are not interpreted; matching is prefix-based only. Future versions may add full spec support.
Privacy
This plugin stores:
– Timestamps of crawler visits (float, with microseconds)
– Last URL seen per crawler (per-URL records)
– Last post ID per crawler (for admin performance)
It does not collect or store personal data about site visitors. No data is transmitted to third parties.
License
GPLv2 or later. See LICENSE file.
Installation
- Upload the plugin folder to
/wp-content/plugins/, or install via the admin Plugins screen. - Activate the plugin.
- Visit Crawler Record under Admin Crawler Record to review crawler activity.
Frequently Asked Questions
This plugin can only begin to track crawler/bot visits from the time you have installed the plugun- not before. It may take several days or weeks (depending upon the popularity of your website) before any come for a visit.
If Settings Reading Discourage search engines is enabled, so none of these systems are allowed to read the pages on your site.
Review feed
Screenshots
Changelog
0.8.0
- Google updated its useragents so I updated the matching strings to account for these changes.
0.7.0
- The robots.txt checker wasn’t actually working. It does now.
- When you look at the report in the admin section, the robots checker is looking for a site-wide rule; when you look at the back-end of a page, the robots checker is looking at that particular page.
- If a page is blocked by the robots.txt file, a link appears sending you to the robots.txt file.
- Added a video to explain how to use this plugin.
0.6.0
- Clarified distinction between Googlebots.
- Better distinguishes Bingbots.
- Cosmetic changes to page in admin section.
- More clear documentation.
0.5.0
- First public release