AI Crawler Simulator
See which AI tools can read your site. We load your page 13 times, once as each of the main AI crawlers that OpenAI, Anthropic, Perplexity, Apple, Amazon, Meta and others send out (GPTBot, ClaudeBot, PerplexityBot and more). We also check your robots.txt for each one, including Google-Extended, the rule Google uses for Gemini training. That way you can spot 403 errors, redirects, and quiet blocks before they hurt your AI visibility. These are outside crawlers, not BuzFind agents.
What AI crawlers are, and why some can’t get in
Tools like ChatGPT, Claude and Perplexity send their own bots to read websites, such as GPTBot, ClaudeBot and PerplexityBot. If your site blocks them, those tools know less about your business when someone asks them about a service like yours.
Two things can block them
- Your robots.txt file. It’s a small text file on your site that tells each bot what it may read. One wrong line can shut out every AI crawler.
- Your server or security service. Some hosting and security services block AI bots by default, even when your robots.txt lets them in.
Should you let them in?
That’s your call. If you want AI tools to know about your business and mention it, it makes sense to let their crawlers in. If you’d rather they don’t use your content, you can block them on purpose. What matters is that it’s your choice and not an accident.
If you see a blocked crawler
- Open your robots.txt (yoursite.com/robots.txt) and look for that bot’s name, or a “Disallow: /” rule for all bots.
- Check the bot settings in your hosting or security service.
- After you change something, run the check here again.
Letting crawlers in doesn’t guarantee an AI tool will mention you. Useful, clear content about what you do still matters most.