robots.txt declares a Crawl-delay directive
A Crawl-delay directive requests a delay between crawler requests. Its effect depends on the crawler and the robots.txt group. Google does not support this field; Bing and Anthropic document support for it. Other audit tools often report this finding as "robots.txt crawl-delay".
Why this matters
A deliberate delay may protect server availability, while an obsolete delay can constrain crawlers that honour it. The finding alone does not establish an indexing problem or explain missing AI citations.
How to fix it
Identify the applicable crawler group and review server requests and errors before changing the delay. Keep intentional restrictions. Bing Crawl Control provides an alternative for Bing; a robots.txt delay takes precedence over that setting. For Google, investigate crawl access and server responses rather than looking for the retired Search Console crawl-rate control. Verify the deployed file and compare observed crawler behaviour.
Turn website findings into an action plan
Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.
Start Full Audit