Make a site readable to AI agents with nginx

An nginx rule can block an agent before your application sees its request. Start by checking the public response at the edge: status, content type, cache headers, and body. Keep any Markdown route explicit, and do not let a proxy turn missing pages into successful HTML shells.

Serve Markdown files with a Markdown media type

For static .md files, configure the type in the relevant server block:

location ~* \.md$ {
    default_type text/markdown;
    try_files $uri =404;
}

This only serves files that exist. It does not generate Markdown from an HTML page. Keep the file contents current with the page they describe, and link to the Markdown version from the HTML page.

Mark negotiated responses as variable

If your application returns Markdown when the request has Accept: text/markdown and HTML otherwise, both responses should include Vary: Accept. Add the header where that representation negotiation occurs:

location /docs/ {
    proxy_pass http://docs_app;
    proxy_set_header Host $host;
    proxy_set_header Accept $http_accept;
    add_header Vary Accept always;
}

Apply that only when the upstream actually varies the representation by Accept. If Markdown has a separate .md URL, the path already identifies the representation and does not need this variation rule.

Keep public discovery files reachable

Make sure /robots.txt, /sitemap.xml, and public documentation routes reach the intended upstream or static directory. Avoid a broad dotfile denial that accidentally blocks published discovery files under /.well-known. Keep secrets and private configuration outside public roots.

Test the deployed routes:

curl -sSI https://docs.example.com/robots.txt
curl -sSI https://docs.example.com/docs/start.md
curl -sSI -H 'Accept: text/markdown' https://docs.example.com/docs/start
curl -sSI https://docs.example.com/this-page-does-not-exist

Expect a real 404 for the missing path. For a negotiated response, confirm that the response changes format and includes Vary: Accept. A CDN in front of nginx may add its own cache behavior, so inspect the public response after both layers are active.

Check the site

Backwatch's Markdown negotiation tester checks the representation and cache headers. The AI bot access checker checks reachability and robots policy.

npx backwatch docs.example.com --pillar readability

This is an outside-in check of public behavior, not a review of your private nginx configuration. Re-run it after proxy or CDN changes, and investigate any inconclusive result rather than treating it as a pass.