Cloudflare can serve a fast public page and still stop an agent at the edge. The useful distinction is between a request for public content and a request for a private application route. Keep public docs fetchable while preserving authentication and abuse controls everywhere else.
Start with the public policy
Review the zone's robots.txt and any AI crawler controls. A simple policy for a public documentation site might look like this:
User-agent: *
Allow: /
Disallow: /account/
Disallow: /private/
Sitemap: https://docs.example.com/sitemap.xml
Robots rules are requests to compliant crawlers; they do not protect private data. Enforce authentication at the application or edge for anything private. Also decide separately whether to allow user-triggered retrieval crawlers, search crawlers, and model-training crawlers. Their names and behavior can change.
Review challenges and bot rules
Request a public docs page with a normal browser and with the user-facing crawler names your product supports. If the second request receives a challenge, inspect the matching WAF, rate-limit, and bot rules. Do not create a zone-wide bypass based only on a User-Agent string: clients can send arbitrary names.
Where your Cloudflare plan supports verified-bot signals, use those signals and scope an exception to the public host and paths that need it. Keep account, API, and admin routes under their existing controls. Prefer a narrow rule that avoids an interactive challenge for public documentation over skipping every security check.
Check cache behavior
If the same URL returns HTML or Markdown based on Accept, the origin must send Vary: Accept, and the cache configuration must respect that header. A separate .md URL is simpler to cache because the representation is explicit.
Inspect the response headers from outside:
curl -sSI https://docs.example.com/robots.txt
curl -sSI -A 'ChatGPT-User' https://docs.example.com/
curl -sSI -H 'Accept: text/markdown' https://docs.example.com/
Look for an expected status, the right content type, and an edge response from the intended route. A 200 challenge page is still a failed content fetch, so inspect the body when a response looks suspicious.
Validate from the outside
Run Backwatch against the public hostname after changing a rule:
npx backwatch docs.example.com --pillar readability
The AI bot access checker focuses on reachability, robots policy, and bot protection. It cannot verify your private Cloudflare configuration or guarantee how a third-party AI product will use the page.
Keep the boundary clear
Allowing a crawler to read a public page does not require exposing an authenticated app, accepting arbitrary API calls, or disabling rate limits. If the content should not be public, block it at the application boundary rather than relying on robots.txt.