Selvakumar Duraipandian

Markdown for AI Agents is a lightweight WordPress plugin that enables HTTP content negotiation for your site’s content. When a client (like an AI agent or a custom script) requests a page with the <code>Accept: text/markdown</code> header, the plugin intercepts the request and returns a clean, structured Markdown representation of the post or page content.
This is ideal for AI crawlers, RAG (Retrieval-Augmented Generation) systems, and non-browser clients that prefer machine-friendly text over complex HTML.
Important note: This plugin is primarily a developer/integration tool. Human visitors browsing your site will never see any difference — the Markdown output is only served when explicitly requested via the <code>Accept: text/markdown</code> HTTP header. Normal browser requests always receive the standard HTML page.
Key Features:
<ul>
<li>Automatically detects <code>Accept: text/markdown</code> headers.</li>
<li>Converts HTML content to clean Markdown using the League HTMLToMarkdown library.</li>
<li>Strips away theme layout, navigation, headers, footers, and sidebars — serving only the main content.</li>
<li>Adds useful HTTP response headers: <code>Content-Type: text/markdown</code>, <code>Vary: Accept</code>, and <code>X-Markdown-Word-Count</code>.</li>
<li>Respects WordPress visibility rules and filters.</li>
<li>No configuration required — works out of the box for posts, pages, and custom post types.</li>
</ul>
<h3>How It Works</h3>
This plugin uses a standard web technique called HTTP content negotiation. The same URL on your site can serve different representations of the same content depending on what the client asks for:
<ul>
<li>A regular browser sends <code>Accept: text/html</code> → receives your normal HTML page.</li>
<li>An AI agent sends <code>Accept: text/markdown</code> → receives a clean Markdown version of the same page.</li>
</ul>
No extra URLs, no duplicate content, no configuration needed. The plugin hooks into WordPress’s <code>template_redirect</code> action, detects the <code>Accept</code> header, captures the rendered HTML, converts it to Markdown, and returns it with appropriate headers.
<h3>Why Markdown for AI Agents?</h3>
When building RAG (Retrieval-Augmented Generation) applications or AI pipelines that ingest web content, HTML is extremely noisy. A typical WordPress page contains thousands of tokens worth of HTML tags, inline styles, navigation menus, scripts, and layout markup — none of which carries meaning for an AI model.
Serving clean Markdown instead can reduce token consumption by up to 60%, which means:
<ul>
<li>Lower API costs — fewer tokens ingested when loading pages into vector stores or LLM pipelines.</li>
<li>Faster processing — less text for the model to parse, filter, and discard.</li>
<li>Better retrieval accuracy — higher signal-to-noise ratio improves the quality of RAG results.</li>
<li>Simpler pipelines — no need for custom HTML stripping logic on the client side; the plugin handles it server-side.</li>
</ul>
Any AI agent, crawler, or ingestion script that sends <code>Accept: text/markdown</code> in its request header will automatically receive the clean Markdown version — no extra URLs, no separate endpoints, no changes to your content workflow.

Markdown for AI Agents

Serve clean Markdown versions of WordPress content to AI agents using HTTP content negotiation.

Azey

AI agents from ChatGPT, Claude, Perplexity, and others read your content to generate answers — but they parse through HTML, JavaScript, and navigation just to reach it. Serve Markdown gives them a direct path: clean Markdown with structured metadata. When a crawler requests it — via an <code>Accept: text/markdown</code> header or a <code>.md</code> URL — your site responds instantly. No noise, no guesswork.
<h4>How It Works</h4>
Serve Markdown adds three capabilities to your WordPress site:
1. Content Negotiation 
When an AI crawler sends <code>Accept: text/markdown</code>, your site returns Markdown instead of HTML — no URL changes needed.
2. .md URL Suffix 
Append <code>.md</code> to any post or page URL (e.g. <code>example.com/my-article.md</code>) to get the Markdown version directly.
3. Markdown Auto-Discovery 
Every page includes a <code><link rel="alternate" type="text/markdown"></code> tag in the HTML head so crawlers can find your Markdown automatically.
<h4>What the Markdown Output Looks Like</h4>
Every Markdown response includes YAML frontmatter with structured metadata followed by your post content. Here is a real example:
<pre>
---
url: 'https://akumarjain.com/textexpander-year-in-review-2025/'
title: TextExpander Year in Review 2025
author:
 name: Ajay
 url: 'https://akumarjain.com/author/akumarjain/'
date: '2026-01-05T09:05:00+05:30'
modified: '2026-01-10T14:30:06+05:30'
type: post
categories:
 - 2025
 - Year in Review
tags:
 - '#ToolsIUse'
 - Productivity
 - Remote Work
image: 'https://akumarjain.com/wp-content/uploads/text-expander-year-in-review-2025.webp'
published: true
---

# TextExpander Year in Review 2025

In my daily work, I need to type the same words and phrases repeatedly...
</pre>
<h4>Features</h4>
Content Serving
<ul>
<li>Content negotiation via <code>Accept: text/markdown</code> header</li>
<li><code>.md</code> URL suffix on any post or page</li>
<li>Auto-discovery <code><link rel="alternate"></code> tag in page head</li>
<li>Each feature can be toggled independently</li>
<li>Choose which post types are exposed (posts, pages, custom post types)</li>
</ul>
Frontmatter and Metadata
<ul>
<li>YAML frontmatter with title, author, date, categories, tags, featured image, and more</li>
<li>Toggle individual metadata fields on or off</li>
<li>Add custom static key-value pairs (e.g., <code>license</code>, <code>language</code>)</li>
<li>Map WordPress custom fields (post meta) into the frontmatter</li>
</ul>
Access Control
<ul>
<li>Per-post opt-out via a sidebar checkbox in the editor</li>
<li>Exclude entire categories from Markdown serving</li>
<li>Exclude entire tags from Markdown serving</li>
<li>Only published posts are served — drafts and private posts are never exposed</li>
</ul>
Crawler Insights
<ul>
<li>Request log showing every Markdown request with timestamp, URL, bot name, and method</li>
<li>Automatic bot detection: ClaudeBot, GPTBot, ChatGPT, OAI-SearchBot, PerplexityBot, Googlebot, Bingbot, Cohere, Meta AI, Bytespider, Applebot, and more</li>
<li>Stats dashboard: total requests, requests today, unique bots</li>
<li>Filter log by individual bot</li>
<li>Configurable retention period with automatic pruning</li>
<li>One-click log clearing</li>
</ul>
Editor Integration
<ul>
<li>“Preview Markdown” button in the post editor sidebar</li>
<li>“View Markdown” link in the Posts and Pages list table</li>
<li>Meta box shows on all enabled post types</li>
</ul>
<h4>Performance</h4>
Markdown conversion only runs when specifically requested — regular visitors see the same HTML pages they always have. The crawler log uses a lightweight custom table with three safeguards against unbounded growth: time-based retention, a row count cap, and a table size cap.
<h4>Privacy</h4>
The crawler log stores IP addresses and user-agent strings from Markdown requests. This data stays in your WordPress database and is never sent to external services. You can disable logging entirely, configure a retention period, or clear the log at any time from the settings page.

Serve Markdown

Serve Markdown versions of your posts and pages to AI agents and crawlers. Content negotiation, .md URLs, auto-discovery, and crawler logging.

Markdown for AI Agents vs Serve Markdown