Free robots.txt tool
AI Crawler Checker
Is your site blocking ChatGPT, Claude or Perplexity?
See which AI bots your robots.txt lets in, page by page. Siteiz separates the crawlers that put you in AI answers from the ones that only collect training data, so you know what a block actually costs you.
Free · no signup · results in seconds
How it works
How to check which AI crawlers you allow
- 01
Enter your website
Type a domain or a full page address. We read the robots.txt at the root of that site.
- 02
Rules are matched per crawler
Each AI crawler gets its own verdict for that exact page, using the same matching rules the crawlers follow (RFC 9309).
- 03
Fix what blocks you
If an AI search crawler is shut out, copy the lines that let it in. You can also paste edited rules and test them before you deploy.
The crawlers we check
Not every AI bot does the same job
Blocking a search crawler hides you from that assistant's answers. Blocking a training crawler only keeps your pages out of model training. Most robots.txt mistakes come from mixing the two up.
| User agent | Operator | Job | What it does |
|---|---|---|---|
| OAI-SearchBot | OpenAI | AI search | Indexes pages so ChatGPT can show and link them in search answers. |
| Claude-SearchBot | Anthropic | AI search | Indexes pages so Claude can find and cite them when it searches the web. |
| PerplexityBot | Perplexity | AI search | Indexes pages so Perplexity can show and cite them in its answers. |
| ChatGPT-User | OpenAI | On request | Visits a page when a ChatGPT user asks it to open or read that page. |
| GPTBot | OpenAI | Training | Collects public pages that may be used to train OpenAI models. |
| ClaudeBot | Anthropic | Training | Collects public pages that may be used to train Anthropic models. |
| Google-Extended | Usage control | Controls whether Google may use your pages for Gemini training and grounding. It does not affect Google Search. | |
| Applebot-Extended | Apple | Usage control | Controls whether Apple may use pages Applebot crawled to train its models. It does not affect Siri or Spotlight results. |
| Meta-ExternalAgent | Meta | Training | Collects public pages that may be used to train Meta AI models. |
| CCBot | Common Crawl | Training | Builds the open Common Crawl archive, which many AI models are trained on. |
| Bytespider | ByteDance | Training | Collects public pages for ByteDance, the company behind TikTok and the Doubao assistant. |
Learn more
After the door is open
Can ChatGPT read your website?
Access is step one. What AI crawlers do with the page once they are let in.
Do cookie banners block AI crawlers?
How consent tools can hide your content from crawlers that robots.txt lets in.
What is an AI visibility audit?
The wider picture: how AI systems access, read and cite your pages.
FAQ
AI crawler and robots.txt questions
Is this AI crawler checker free?
Yes. There is no signup and no email. Enter a website and you get the verdict for every AI crawler we track in a few seconds.
How do I check if my website blocks ChatGPT?
Enter your site above and look at three rows: OAI-SearchBot, which indexes pages for ChatGPT search, ChatGPT-User, which opens a page when someone asks ChatGPT to read it, and GPTBot, which collects training data. If OAI-SearchBot is blocked, ChatGPT search cannot index your pages.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects public pages that may be used to train OpenAI models. OAI-SearchBot indexes pages so ChatGPT can show and link them in search answers. They are controlled separately, so you can block GPTBot to opt out of training and still allow OAI-SearchBot to stay visible in ChatGPT search.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot is a choice about how your content is used. Blocking search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot keeps you out of those assistants' answers. Many sites allow the search crawlers and decide on training separately.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a robots.txt token, not a separate crawler. Google states it does not affect inclusion or ranking in Google Search. It controls whether your content is used for Gemini training and grounding. AI Overviews in Google Search are crawled by Googlebot, not Google-Extended.
My robots.txt allows AI crawlers. Can they still be blocked?
Yes. robots.txt is only one layer. A CDN or firewall rule, a bot-protection setting or a login wall can turn crawlers away even when robots.txt says yes. This tool reads robots.txt only. The free AI visibility scan requests the page itself the way a crawler does and shows what actually comes back.
What does Allowed by default mean?
No rule in your robots.txt mentions that crawler or matches that page, so the crawler may visit. That is the normal state for most sites and nothing needs fixing.
What is llms.txt and do I need one?
llms.txt is a proposed plain-text file at the root of a site that gives AI tools a short, curated map of your most useful pages. It is not an official standard and support from AI companies is limited so far, so treat it as optional. We show whether one exists so you know where you stand.
One file is not the verdict.
robots.txt decides whether AI may visit. The free AI visibility scan shows what it finds when it does, graded A to F with the exact fixes.