Skip to content

robots.txt declares restrictive Content-Signal AI-use preferences

Content Signals are a machine-readable policy mechanism documented and implemented by Cloudflare. They distinguish search indexing, real-time AI input, and AI training. This check found ai-input=no and/or ai-train=no in robots.txt.

Why this matters

The directive expresses a publisher preference about those uses. Cloudflare's own crawl tooling enforces Content Signals, but support is not universal across every AI provider, so AuditHQ should not treat the directive as proof that Google, OpenAI, Perplexity, or every other answer engine will exclude the site.

How to fix it

First confirm the publisher's intent. Keep restrictive values if they are deliberate. If they were added accidentally and broader AI use is wanted, update the Content-Signal values or remove the restriction. Provider-specific crawler controls should still be reviewed separately.

Turn website findings into an action plan

Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.

Start Full Audit