# AI crawler access: robots.txt is only one layer

_2026-11-09_

An AI crawler can be allowed by `robots.txt` and still receive a block page from a firewall, CDN, or application. It can also reach the homepage while being unable to fetch the documentation it needs. Check each layer from outside, and make the decision about public content separately from the decision about private routes.

## Understand the layers

- **robots.txt** states preferences for crawlers that honor the file. It does not enforce access control.
- **A WAF or bot system** can challenge or block a request before it reaches the application.
- **Application authentication** decides whether the requested content is public.
- **A cache or proxy** can return a stale page, an HTML challenge, or the wrong representation.

Changing one layer does not change the others. If a page contains private data, protect it with authentication even if robots.txt says not to crawl it.

## Separate kinds of crawlers

Some crawlers fetch pages for search or training. Others fetch a page after a person asks an AI product to look at it. Their names and policies differ and can change. Decide which public pages each category may read; do not assume every crawler should be allowed or blocked.

Avoid broad security exceptions based on a User-Agent alone. A client can send any string. If your CDN provides verified-bot signals, use those where available, and scope any exception to the public host and paths that need it. Keep account, admin, and API routes protected.

## Check the response, not just the status

A challenge page can return HTTP 200. Request the public page with the user-facing crawler names you intend to support and inspect both headers and body:

```sh
curl -sS -D - -A 'ChatGPT-User' https://example.com/docs/
curl -sS -D - -A 'Claude-User' https://example.com/docs/
```

Use the current names and guidance from each provider when you configure production rules. These examples are a smoke check, not proof that the request came from the provider.

Then run a focused outside-in scan:

```sh
npx backwatch example.com --pillar readability
```

Backwatch checks public reachability, robots policy, and related response evidence. It cannot inspect private WAF rules or promise that an AI product will index or use a page.

For practical configuration, see [Cloudflare setup](/guides/cloudflare), [nginx setup](/guides/nginx), and the [AI bot access checker](/tools/ai-bot-access-checker).