How AI crawlers work
Unlike standard indexers, AI crawlers fetch your content to inject it directly into inference context windows or RAG databases.
Which systems crawl you
ChatGPT-User: Operates when a user specifically asks ChatGPT to browse a link.OAI-SearchBot: Feeds SearchGPT protocols.Anthropic-ai: Claude's data collection arm.
Optimization tips
Ensure your firewall allows these User-Agents. If you're on Cloudflare, check your bot management settings. Use our test tool to verify!