Santos R. Guerra Faro

WordPress plugin to keep track of User Agents that come to our website. This information is usually extracted from the server log, but in many cases it is a cumbersome process and a lot of valuable time is lost, so I decided to make a simple plugin that allows you to get the information without consuming resources and easily export it for further analysis.
A User Agent is a text string that identifies the software and hardware used by a client when accessing a website or online service. It typically includes information about:
<ul>
<li>Web browser (type and version)</li>
<li>Operating system</li>
<li>Device (in the case of cell phones or tablets)</li>
<li>Rendering engine</li>
<li>Other client characteristics</li>
</ul>
Example User Agent: 
”’Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36””
User Agents’ collection and analysis provides valuable insights for developers, analysts and security professionals, allowing to improve user experience and website effectiveness.
The information collected by this plugin can be used for: 
* Search bots identification: Allows to identify search engine bots (such as Googlebot, Bingbot, etc.) that are crawling your site. 
* Detection of technical problems.* Resource optimization.* Identification of malicious traffic. 
* Verification of SEO tools. 
* Traffic analysis.

Get User-Agents

WordPress plugin tracks User Agents, avoiding server log analysis. Records and exports data easily for further analysis.

dizzysoft

Crawler Record logs the last time specific user agents (for search and AI chat/LLMs) accessed your content including:
<ul>
<li>Google</li>
<li>Bing</li>
<li>ChatGPT (OpenAI)</li>
<li>Claude (Anthropic)</li>
<li>Perplexity</li>
<li>DuckDuckGo</li>
<li>Meta</li>
<li>Apple</li>
</ul>
You can view this information:
<ul>
<li>Admin page: a grouped list of all crawlers, with last seen time, last page, and robots status for the crawler. </li>
<li>Admin Bar (front-end): quick-glance status for the current singular or URL context (no dropdowns in wp-admin).</li>
<li>Per Page/Post: From the edit screen on any page or post, you can see that last time each crawler visited that page.</li>
<li>Recent Pages: From the admin page you can select a crawlwer and get a list of the recent pages this UA has visited.</li>
</ul>
Robots-aware: The plugin checks your robots.txt and evaluates Allow/Disallow rules for a given path. If Settings → Reading → “Discourage search engines” is enabled, all agents are shown as blocked with a prominent warning.
Performance-friendly by design: Write-throttling (default 10 minutes) and an auxiliary “last post ID per agent” record avoids heavy admin queries on large sites.
Privacy-friendly: Saves only bot visit timestamps and last URLs crawled — no personal data.
Learn <a href="https://www.dizzysoft.com/crawler-record-plugin-for-wordpress/" rel="nofollow ugc">how to use this plugin</a>.
<h4>Highlights</h4>
<ul>
<li>Supports common user-agent variants for Google, Bing, OpenAI (ChatGPT), Anthropic (Claude), Perplexity, Meta, Apple, and DuckDuckGo.</li>
<li>Robots status computed from local robots.txt (physical or virtual) without outbound requests.</li>
<li>Clear UI with grouped sections, microsecond timestamps, and a small diagnostics toggle showing the matched robots group and rule.</li>
</ul>
<h4>Known limitations</h4>
<ul>
<li>Can only track crawlers from the time the plugin is installed- cannot look into the past.</li>
<li>robots.txt wildcards (<code>*</code>) and end-of-line marker (<code>$</code>) are not interpreted; matching is prefix-based only. Future versions may add full spec support.</li>
<li>To reduce database load and avoid slowing the website, bot tracking is throttled, so not every crawler visit will necessarily be recorded.</li>
</ul>
<h3>Privacy</h3>
This plugin stores: 
– Timestamps of crawler visits (float, with microseconds) 
– Last URL seen per crawler (per-URL records) 
– Last post ID per crawler (for admin performance)
It does not collect or store personal data about site visitors. No data is transmitted to third parties.
<h3>License</h3>
GPLv2 or later. See LICENSE file.

Crawler Record

Crawler Record tells you the last time each of the most common search/chat bots visited (Google, ChatGPT, etc)- and the pages at which they looked.

Get User-Agents vs Crawler Record