Menu
open source · runs in this tab, uploads nothing
The AI bots reading your site. The visitors AI sends back.
Paste a domain to see which of 28 AI crawlers its robots.txt lets in. Drop an access log to count the visits ChatGPT, Perplexity and 10 other AI chats sent you. Both run in your browser.
or paste lines
Your file never leaves your browser.
- AI crawlers known by name
- 28
- companies behind them
- 13
- AI chats spotted in referrers
- 12
- bytes of your data uploaded
- 0
Crawlers fromOpenAIAnthropicPerplexityGoogleMicrosoftCommon CrawlByteDanceAmazonAppleMetaDuckDuckGoYou.comMistral AI
AI crawler robots.txt checker
See ChatGPT and Perplexity traffic in your logs
rastrolog doesn't recognise this log format. Here's the first line:
The file ends early (a truncated or damaged gzip). These numbers cover every complete line before that.
No AI crawlers or AI chat visits in this log.
Not counted as AI traffic. Google AI Overviews send a plain google.com referrer, so they look like search.
Keep counting after you close this tab
Your server logs · Python 3.10+
The same report in your terminal, plus middleware for FastAPI and Django.
pip install rastrolograstrolog parse access.logYour pages · one script tag, under 2 KB
Sends one ai_referral event to GA4, Plausible, PostHog, Fathom, Umami, Matomo or Tag Manager.
<script src="https://cdn.jsdelivr.net/npm/rastrolog@0.2.1/dist/snippet.min.js" integrity="sha384-f1WmE4FFMG3T7OCN/djpfKWMNHW3DxGQ3dICa4ZZlIlqLgLnzr4Yzfs0SFK0Wzdm" crossorigin="anonymous" defer></script>What it detects
28 crawler tokens and 12 AI products. 27 of the tokens come from the vendor's own documentation, and the list is open: signals.json.
| Company | Crawler tokens |
|---|---|
| OpenAI | GPTBot · ChatGPT-User · OAI-SearchBot |
| Anthropic | ClaudeBot · Claude-User · Claude-SearchBot |
| Perplexity | PerplexityBot · Perplexity-User |
| Google-Extended · Googlebot · Google-Agent · Google-GeminiNotebook | |
| Microsoft | bingbot |
| Common Crawl | CCBot |
| ByteDance | Bytespider (third-party source) |
| Amazon | Amazonbot · Amzn-SearchBot · Amzn-User |
| Apple | Applebot · Applebot-Extended |
| Meta | meta-externalagent · meta-externalfetcher · meta-webindexer |
| DuckDuckGo | DuckAssistBot |
| You.com | YouBot |
| Mistral AI | MistralAI-User · MistralAI-Index · MistralAI-Training |
AI referrers
ChatGPT · Claude · Perplexity · Gemini · NotebookLM · Copilot · You.com · Le Chat · Duck.ai · Meta AI · Grok · DeepSeek
$ rastrolog alerts
One email when an AI company ships a new crawler
When a company adds or renames a crawler, you get its name, what it does with your pages and the exact robots.txt lines to block it. That's the whole newsletter.
Mistral AI uses it to collect pages for training its models. Mistral AI's docs
To block it, add these lines to robots.txt:
User-agent: MistralAI-Training
Disallow: /