///// Checks / Readability
robots.txt AI crawler policy
Whether robots.txt lets AI search crawlers read your public pages.
- Check id
robots-ai-policy-quality- Pillar
- Readability
- Runs in
- The free CLI
- Scoring
- Up to 2 points
- Status
- Pre-release, in calibration
Why it matters
AI search can only cite pages its crawlers may read. Blocking training crawlers is a separate choice.
How to pass
Allow the AI search crawlers in robots.txt, and block training crawlers only if you choose to.
Run it
Find robots-ai-policy-quality in the report. Add --md for a fix prompt you can paste into Claude Code, Codex or Cursor.