Interested in my skills? Hire me through the TopTal platform

Serving Markdown to LLMs from Your Rails App

When a model reads your site, it downloads the HTML: navigation, scripts, styles, metadata, and somewhere in the middle, the content it actually needs. On this blog, a post page is about 28.9 KB of HTML, but the Markdown that produced it is about 8.4 KB. Everything else costs the reader tokens, and the model has to guess which part is the article.

TL;DR: Serve the same URL as Markdown when a client asks for it, list your pages in an /llms.txt index, and treat that output like a public API: same content as the HTML, stable URLs, nothing private.

One URL, two representations

The idea underneath is HTTP content negotiation, which is older than any LLM. A resource has one URL and can come in several representations. The client says which one it prefers, either with an Accept header or with a format extension like .md, and the server picks one. Markdown even has its own media type, text/markdown, defined in RFC 7763.

Follow a request to see where it splits:

flowchart LR
  A["GET /posts/slug"] --> B{"Wants text/markdown<br/>or ends in .md?"}
  B -- no --> C["Full HTML page"]
  B -- yes --> D["Model's to_markdown"]
  C --> E["Response with Vary: Accept"]
  D --> E

Both branches load the same record. Only the last step is different, which is what keeps the two versions from drifting apart.

Rails 8.1 does most of the work

Before 8.1, you had to register the Markdown type yourself and render plain text with the right content type. Rails 8.1 made Markdown rendering a headline feature:

The framework now registers text/markdown with md and markdown as extensions, so no initializer is needed.
A new markdown: renderer calls to_markdown on whatever you pass it, and sends strings as they are.
Adding a Markdown branch to an action's existing respond_to block opts that action in. In my tests, Rails also added Vary: Accept, which tells caches that the response depends on that header, as described in RFC 9110.

So the work is mostly a modeling decision: what does to_markdown return? On this blog, it's the title as a heading, the summary as a quote, and the raw post body. The posts were already written in Markdown, so nothing needs converting. If your content is stored as HTML or rich text, convert it once, when it's saved, rather than on every request.

One trap cost me a failing test. ERB templates for the Markdown format still escape HTML, so "Q&A" comes out as Q&A. That's correct for HTML and wrong for Markdown. Use ERB's raw-output tag in those templates, and add a test with an ampersand in a title.

Give models a table of contents

A single page as Markdown helps once a model has found it. llms.txt is a proposed convention for helping it find the page in the first place: a Markdown file at the root of the site with an H1 for the site name, a short blockquote summary, and lists of links with one-line descriptions. The same proposal suggests serving each page's Markdown at its own URL with .md added, which is exactly what content negotiation already gives you.

In Rails, that's one route pointing to the posts index in Markdown format, and a small template that links to every post's .md version along with its summary. It's generated from the database, so it never goes stale.

Then make both discoverable from the HTML side. Add a link tag in each page's head with rel set to "alternate" and the type set to text/markdown. Crawlers that already parse the head will find the Markdown version without guessing.

What belongs in the Markdown version

The Markdown output is a public interface, so the same rules apply as for any API:

Same content as the HTML. Serving models something other than what humans see is cloaking. It erodes trust, and search engines penalize it.
Absolute, canonical URLs. A model often reads a page without the site around it, so relative links stop meaning anything.
Lead with context. Put the title and a one-sentence summary at the top, the way a good README does, so a client that truncates long pages still gets the point.
Only published, public content. Drafts, admin pages, and anything behind a login stay out of both the .md routes and llms.txt. Scope the query, don't just hide the link.
Keep the source intact. Leave code fences and diagrams as Markdown source rather than rendered output. A mermaid block is more useful to a model than an SVG.
Curate the index. llms.txt is a table of contents, not a sitemap dump. One line per page that answers "should I read this?" works best.
Access control lives elsewhere. llms.txt invites models to read; it doesn't block anyone. To limit crawlers, use robots.txt (RFC 9309) and rate limiting.

Measuring the gain

On this blog, running Rails 8.1.4, one post's full HTML page was 28,863 bytes, its article element alone was 9,589 bytes, and the Markdown response was 8,358 bytes. That's about 3.5× less to download and tokenize than the full page, with no navigation or markup to strip.

To check it on your own app, request the same post twice, once normally and once with .md added, and compare the response sizes. Then paste both into your model provider's tokenizer and compare token counts, which is the number that actually affects cost and context space.

Tradeoffs

Every page with a Markdown version is another output to keep correct, so test it the way you test JSON: content type, exact body, escaping, and the Vary header. llms.txt is still a proposal, and how much crawlers use it varies. It costs one route and one template, so the downside is small.

Takeaway

Treat models as one more client of your resources. Negotiate the representation, keep it the same as what humans see, and make it as easy to find as your HTML.