The AI Crawler and SEO Manager by akbansa plugin generates an llms.txt file for crawler discovery and logs visits by AI-based crawlers such as GPTBot, …

Anshul

<p>The AI Crawler and SEO Manager by akbansa plugin generates an <code>llms.txt</code> file for crawler discovery and logs visits by AI-based crawlers such as GPTBot, ClaudeBot, and others. It also provides an analytics page in the WordPress admin dashboard to view crawler activity by path, URL, and timestamp.</p>
<h3>Key Features:</h3>
<ul>
<li>Generates <code>llms.txt</code> file for crawler discovery.</li>
<li>Removes <code>llms.txt</code> on plugin uninstall</li>
<li>Schedules automatic generation of <code>llms.txt</code> daily at midnight (frequency configurable: daily, twice daily, hourly)</li>
<li>Uses atomic file writes for <code>llms.txt</code> and <code>sitemap.xml</code> for safety</li>
<li>Integrates with Yoast SEO, Rank Math, or All in One SEO for metadata (falls back to DB if not present)</li>
<li>Admin panel to select/exclude categories and content types for <code>llms.txt</code></li>
<li>Logs every AI crawler request from the frontend (user agent, crawl info, page URL, IP, referrer)</li>
<li>Filters crawlers using <code>config.json</code> and stores all details in a dedicated DB table</li>
<li>Analytics dashboard in admin to view recent crawler activity</li>
<li>Robust AJAX security and error handling</li>
<li>Detects and logs visits from AI crawlers based on their User-Agent.</li>
<li>Stores crawler logs in a database for analysis.</li>
<li>Supports popular AI-based crawlers (e.g., OpenAI, Anthropic, GPTBot, etc.).</li>
<li>Option to customize the list of crawlers via a <code>config.json</code> file.</li>
</ul>
<h3>Short Description</h3>
<p>Advanced AI-powered SEO and analytics for WordPress. Generates llms.txt for AI crawler discovery, logs AI bot visits, and provides comprehensive analytics dashboard.</p>
<h3>Usage</h3>
<ul>
<li>After activation, the plugin will automatically start generating the <code>llms.txt</code> file and log visits from AI-based crawlers.</li>
<li>The crawler logs are stored in the database for analysis.</li>
<li>You can view analytics by navigating to <strong>LLMS Crawlers</strong> in the WordPress admin menu.</li>
</ul>
<h3>Admin Analytics</h3>
<p>The admin panel will display a list of the most recent crawler visits, showing the following information:<br />
– Path<br />
– URL<br />
– Crawler Name<br />
– Timestamp</p>
<h3>Support</h3>
<p>If you encounter any issues, please feel free to reach out via the plugin support page or create an issue on the GitHub repository.</p>
<h3>License</h3>
<p>This plugin is licensed under the GPLv2 or later.</p>
<h3>New Features</h3>
<ol>
<li><strong>Admin Settings Filter</strong>: You can now filter active pages, products, categories, and posts for generating the <code>llms.txt</code>.</li>
<li><strong>Sitemap Handling</strong>: If a <code>sitemap.xml</code> exists, it will be used to get the list of pages. If not, the database will be used as a fallback.</li>
<li><strong>SEO Plugin Integration</strong>: The plugin integrates with Yoast SEO, All in One SEO, and Rank Math to fetch metadata for the <code>llms.txt</code> file.</li>
<li><strong>Frequency Setting</strong>: You can choose the frequency for regenerating the <code>llms.txt</code> file (daily, weekly, monthly, hourly).</li>
<li><strong>Traffic Capture and AI Agent Logging</strong>: A front-end script captures traffic, sends it to the backend, and logs AI agents based on a <code>config.json</code> file.</li>
<li><strong>Sitemap Update</strong>: The <code>sitemap.xml</code> is automatically updated with the absolute path to the <code>llms.txt</code> and the updated timestamp.</li>
</ol>


Alternatives to AI Crawler and SEO Manager by akbansa

Top alternatives