Why Your Website Should Respect Accept Headers for AI Agents

The Future is Already Requesting Markdown

If you're building for the web in 2025, you need to know this: AI agents aren't browsing your HTML anymore. They're asking for markdown.

OpenCode's next version is switching to request text/markdown versions of web pages by default. This isn't just a technical curiosity; it's a fundamental shift in how AI agents consume web content.

Remember: Accept Headers Are Already a Web Standard

Here's something many developers forget: Accept headers have been part of the HTTP specification since the beginning. They're not new. We've been using them for years to negotiate between HTML, XML, and JSON responses. The Accept header is how clients tell servers what content types they prefer.

Yet somehow, most websites only serve HTML. They ignore Accept headers entirely, missing out on a fundamental piece of HTTP content negotiation. Now, with AI agents becoming primary web consumers, this oversight is becoming a real problem.

Why Markdown Matters for AI

Here's the thing about HTML: it's verbose. All those div tags, classes, inline styles, and JavaScript snippets? They're noise to an LLM. When an AI agent fetches your beautifully crafted React app, it's wading through thousands of tokens of markup just to find the actual content.

Markdown strips away the cruft. It's pure content with minimal formatting. Exactly what LLMs need to understand your site efficiently.

Consider the token economics:

• HTML page: 5,000-10,000 tokens (or more)
• Same content as markdown: 500-1,500 tokens
• Token savings: 70-90%

That's not just a performance improvement. It's the difference between an AI agent being able to process your entire site versus hitting rate limits after a few pages.

Real-World Implementation

I've just implemented Accept header content negotiation on this very site. You can now request any page in three formats:

• Accept: text/html - Traditional HTML (default)
• Accept: text/markdown - Clean markdown format
• Accept: text/plain - Plain text, no formatting

Want to see this exact blog post in markdown? Try this:

curl -H "Accept: text/markdown" https://groff.dev/blog/respect-accept-headers-for-ai-agents

How It Works

The implementation uses Next.js middleware to detect the Accept header and serve appropriate content. When a request comes in with Accept: text/markdown, the middleware rewrites to an API endpoint that:

1. Reads the original MDX content
2. Strips out React components and imports
3. Returns pure markdown with proper Content-Type headers

The beauty is that your existing pages remain unchanged. You're not maintaining two versions of your content. You're serving the same content in different formats based on who's asking.

Why This Matters Now

AI agents are becoming primary consumers of web content. They're not just search crawlers anymore. They're actively reading, understanding, and acting on information from websites.

If you want AI agents to:

• Discover your content effectively
• Process it efficiently
• Understand it accurately
• Take actions based on it

...then you need to speak their language. And increasingly, that language is markdown.

The Competitive Advantage

Sites that respect Accept headers will have a significant advantage in the AI-first web:

1. Better discoverability - AI agents will prefer and remember sites that are easy to consume
2. Lower costs - Fewer tokens mean cheaper API calls for everyone
3. Faster processing - Less parsing overhead means quicker responses
4. More accurate understanding - Clean content without HTML noise improves comprehension

Implementation Tips

If you're ready to make your site AI-friendly:

1. Start simple - You don't need to support every page immediately
2. Focus on content pages - Blog posts, documentation, and articles are perfect candidates
3. Use proper Content-Type headers - Always return text/markdown; charset=utf-8
4. Test with real tools - Try fetching your pages with curl using different Accept headers
5. Monitor usage - Track how many requests come in with markdown Accept headers

The Bottom Line

The web is evolving from human-first to human-and-AI-first. Accept headers aren't just about content negotiation anymore. They're about making your content accessible to an entire new class of consumers.

OpenCode is just the beginning. As more AI tools adopt markdown-first approaches, sites that don't support Accept headers will be left serving expensive, inefficient HTML to agents that just want the content.

Make your site future-proof. Respect the Accept header.

---

Want to discuss AI agent optimization strategies? Connect with me on LinkedIn or follow me on X @groffdev.