AI agents often fetch public pages with a plain HTTP request. A Next.js page can look complete in a browser and still be empty to a reader that does not run JavaScript. Keep the useful content in the server response, publish a small index, and make the clean text version easy to find.
1. Render the content on the server
Use Server Components for public copy and docs data. A client component can add interaction, but the page title, explanation, examples, and links should already be present in the HTML response.
Check the first response without executing scripts:
curl -fsS https://your-site.com/docs | head -c 1200
If the response contains only an app shell, move the public content into a Server Component or statically render the route. Do not add private data to a public render just to make the page easier to crawl.
2. Publish a short llms.txt
Place llms.txt at the site root. Say what the project is, then link to the few pages that explain the product and its API. Keep the file maintained as routes change.
# Example product
> A short, factual description of what the product does.
## Docs
- [Getting started](https://your-site.com/docs/start)
- [API reference](https://your-site.com/docs/api)
llms.txt is a useful index convention, not a replacement for robots.txt or a sitemap, and its presence does not guarantee that a model will cite the site. See Backwatch's llms.txt checker.
3. Offer Markdown as a second representation
For each important page, provide either a .md route or a response negotiated with Accept: text/markdown. If one URL returns HTML or Markdown depending on that header, return Vary: Accept so caches do not mix the two representations.
Next.js can advertise the Markdown URL in metadata:
import type { Metadata } from "next";
export const metadata: Metadata = {
alternates: {
types: { "text/markdown": "/docs/start.md" },
},
};
Confirm both representations:
curl -sSI https://your-site.com/docs/start.md
curl -sSI -H 'Accept: text/markdown' https://your-site.com/docs/start
The Markdown response should use a Markdown content type. The negotiated HTML route should vary on Accept. Backwatch's Markdown tester checks the common paths and headers.
4. Make missing routes honest
Return an actual 404 for a missing page, not a 200 response containing “not found.” Keep robots.txt, the sitemap, and any public /.well-known files reachable through the app router, reverse proxy, and CDN.
Run a focused check after deployment:
npx backwatch your-site.com --pillar readability
Backwatch checks the public responses it can observe from outside. A passing scan does not prove that every AI product indexes or uses the content.