robots.txt blocks Claude-User (user-directed retrieval)
The observed robots.txt policy blocks Claude-User. Anthropic describes Claude-User as the agent used when users ask Claude to access websites. It is distinct from ClaudeBot for model development collection and Claude-SearchBot for search quality and indexing.
Why this matters
If the business wants Claude user-directed retrieval, an unintended block can prevent Anthropic's system from retrieving content for those requests and may reduce visibility in that context. It does not prove a training opt-out, does not control Claude search indexing, and allowing it does not guarantee citations or referrals.
How to fix it
Choose the intended policy for each Anthropic bot. Preserve private-path restrictions; remove an unintended Claude-User block if user-directed retrieval access is desired. Review ClaudeBot and Claude-SearchBot separately, then verify the deployed robots.txt response before measuring outcomes.
Turn website findings into an action plan
Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.
Start Full Audit