Skip to content

robots.txt blocks Claude-SearchBot (Claude search index)

The observed robots.txt policy blocks Claude-SearchBot, which Anthropic describes as supporting search quality and indexing. It is distinct from ClaudeBot for model development collection and Claude-User for user-directed retrieval.

Why this matters

If the business wants Claude search access, an unintended block can prevent Anthropic's search crawler from indexing or optimizing pages for that search experience and may reduce visibility in that context. It does not prove a training opt-out, does not control user-directed retrieval, and allowing it does not guarantee indexing, citations or referrals.

How to fix it

Choose the intended policy for each Anthropic bot. Preserve private-path restrictions; remove an unintended Claude-SearchBot block if search access is desired. Review ClaudeBot and Claude-User separately, then verify the deployed robots.txt response before measuring outcomes.

Turn website findings into an action plan

Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.

Start Full Audit