Skip to content

robots.txt blocks Perplexity-User (live browsing agent)

Perplexity-User is Perplexity's on-demand crawler, triggered when a real user's search pulls your page into an answer in real time, separate from PerplexityBot, the bulk crawler that indexes content in advance. This check found robots.txt disallowing Perplexity-User specifically. Both crawlers matter, but they cover different moments in a prospect's research.

Why this matters

Even if PerplexityBot is allowed, blocking Perplexity-User may stop Perplexity fetching your page when a person's query asks it to. The bulk crawler being allowed can make the setup look fine at a glance, so this gap can go unnoticed.

How to fix it

Your developer needs about 15 minutes. Allow Perplexity-User in robots.txt unless you specifically intend to block Perplexity's live, user-triggered visits. This is worth fixing even if PerplexityBot is already allowed, since the two crawlers serve genuinely different moments in the buying journey.

Turn website findings into an action plan

Create an account to start a Full Audit of your website, with supporting evidence and a prioritised action plan.

Start Full Audit