Skip to content

Should you remove crawl-delay from robots.txt?

A crawl-delay finding is a policy decision, not an automatic SEO failure. Google does not support it. Other crawlers may honour it. First identify the crawler and the server-load problem the rule was meant to solve.

Free robots.txt text reviewer

Find crawl-delay lines in your robots.txt

Paste the plain-text file to list its delay entries, line numbers and declared user-agent groups. This reviews your text in the browser; it does not fetch a website or determine effective crawl permission, server capacity or indexing.

Use the file from the affected hostname. Up to 50,000 characters; larger files are rejected in full.

Load the illustrative example or paste a file, then review it.

Before changing a delay

Google does not support Crawl-delay. Anthropic documents support for this extension. Confirm the affected crawler and the purpose of the limit before editing; removing a line is not a guarantee of faster indexing or AI citations.

Free editable handover sheet

A blank worksheet for the hostname, crawler group, reason for the limit, exact edit, rollback and before/after server evidence. Open it in a spreadsheet and keep your completed copy privately. No signup or upload is needed; this is a change record, not a crawl-permission test.

Which crawlers use crawl-delay?

Google documents crawl-delay as unsupported. Bing supports the directive, and a robots.txt delay takes precedence over its Crawl Control settings. Anthropic also supports crawl-delay for its bots. Removing the line therefore changes crawler-specific behaviour; it is not a universal way to make Google index a site faster.

Find the rule that actually applies

Open the robots.txt file on the affected hostname, not a copy stored in an old project folder. Record the User-agent group containing the delay, its value, and the neighbouring Allow and Disallow rules. Then compare it with the crawler named in your logs. A rule in a ClaudeBot group is a different decision from one in the generic group. Subdomains may serve different files.

  1. Save the current live file and identify who owns its generation: CMS, application, CDN or static hosting.
  2. List the important URLs whose freshness matters, such as a service page, new article or product listing.
  3. Check recent crawler requests and server errors before assuming the delay is the bottleneck.

Choose a change using server evidence

Keep a deliberate rate limit if removing it would create a real availability problem. If the delay is historical and no longer needed, change only that directive in the relevant group. For Bing, consider its Crawl Control interface instead. For Google, investigate response times, error rates and crawl access rather than looking for a crawl-delay setting. Do not remove private-path restrictions while cleaning up an unrelated delay.

Worked example: a Claude-only delay

Suppose a file contains a ClaudeBot group with Crawl-delay: 10, followed by a separate Claude-SearchBot group. That declaration belongs to the ClaudeBot group; it is not evidence that Google waits ten seconds, or that Claude-SearchBot uses that value. This is an illustrative policy-reading example, not a result from your website. Use the text reviewer above to locate declarations, then check the named provider policy and your live requests before deciding what to change.

User-agent: ClaudeBot
Crawl-delay: 10
Disallow: /private/

User-agent: Claude-SearchBot
Disallow: /private/
  1. If the goal is Google indexing, this delay is not the Google setting to change.
  2. If the goal is Claude search access, review the Claude-SearchBot group and delivery controls separately.
  3. If the goal is reducing ClaudeBot server load, keep the capacity evidence and intended rate limit with the change record.

Verify the deployed policy

After publishing, retrieve the live file again and compare it with the saved version. Check the public hostname and any redirecting hostname separately. The useful outcome is a correctly served policy plus healthy crawler requests, not merely a successful Git commit.

  1. Confirm the intended delay changed and the required Disallow rules remain.
  2. Check that the affected public pages still return their expected content and status.
  3. Watch crawler requests, response latency and errors over a comparable period.
  4. Use indexing and search-performance evidence to assess freshness separately from crawl activity.

What a crawl-delay change cannot promise

A crawler being permitted or being able to request more pages does not guarantee indexing, ranking or AI citations. If the page is inaccessible, duplicated or unhelpful for the query, removing a delay will not repair those issues. For a client handover, record the original purpose, the exact changed group, the verification date and the signal you will compare next. A public scan can identify the directive; it cannot establish your server capacity from that line alone.

Frequently asked questions

Does Google support crawl-delay?

No. Google lists it as an unsupported robots.txt field. Look at access, response errors and crawl evidence when diagnosing Google crawling.

Should every website remove crawl-delay?

No. Identify the affected crawler and whether the limit still protects server availability. Change the specific rule using evidence.

Will removing it increase AI citations?

There is no guaranteed relationship. Crawl access is one prerequisite; relevance, indexing and the answer system still determine visibility.

Turn website findings into an action plan

Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.

Start Full Audit