Open your-domain.ch/robots.txt and look for the names GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended: if it says "Disallow: /" under a name, that crawler is blocked. Next, check whether the page text also appears in the source code without JavaScript, and that no login, cookie wall or bot protection hides the content. For visibility in AI answers, at least the providers' search crawlers must get through.
Before an AI can cite your website, it has to be allowed to read it. That sounds obvious, but it fails more often than you'd think: because of a line in robots.txt that an agency added as a precaution back in 2023, a website builder that only loads text via JavaScript, or bot protection that's been switched on since the last hosting update. The following test needs no developer and takes two minutes for the first step, ten for all four.
Step 1: read robots.txt
Type ihre-domain.ch/robots.txt into your browser's address bar. The file is public and consists of blocks that start with User-agent:. Use Ctrl+F (Mac: Cmd+F) to search for these names:
| Crawler | Operator | Stated purpose | Consequence if blocked |
|---|---|---|---|
| GPTBot | OpenAI | Collecting content to train the models | Content isn't used for training; search answers aren't affected |
| OAI-SearchBot | OpenAI | Index for search in ChatGPT | Website doesn't appear as a source in ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetching a page when a user asks about it in chat | Page can't be opened in the conversation |
| ClaudeBot | Anthropic | Collecting content for training | Same as GPTBot |
| Claude-SearchBot, Claude-User | Anthropic | Search and fetching on user request | Website is missing from Claude answers with web search |
| PerplexityBot | Perplexity | Index for Perplexity search | Website doesn't appear as a source on Perplexity |
| Perplexity-User | Perplexity | Fetching on user request | Page can't be opened |
| Google-Extended | Controls usage for Gemini applications | No effect on Google search and AI overview |
What matters is what's listed under each name. This block fully blocks OpenAI's training crawler while letting the search crawler through:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /If a name is missing entirely, the general block User-agent: * applies. If that says Disallow: /, everything is blocked, including Google. If it says only Disallow: /admin/ or similar, the AI crawlers are free to access. What the file can and can't do in general is explained in the glossary on robots.txt.
Step 2: The JavaScript test
Google renders JavaScript and therefore also sees text that's only generated in the browser. The providers don't document whether the crawlers from OpenAI, Anthropic and Perplexity execute JavaScript. Assume they don't, and check what's left without JavaScript:
- Open an important page, for example your services page.
- Call up the source code: right-click, "View page source", or put
view-source:in front of the address. - Use Ctrl+F to search for a sentence from the visible text, for example the first sentence of your service description.
If you find the sentence in the source code, the content is readable for every crawler. If you don't find it, but only script references and empty containers, your website only delivers the text afterwards. This mainly affects websites built as pure web apps, and some website builders with "dynamic" content blocks. The solution is server-side rendering or static delivery, and that's a job for your developer, not a five-minute fix.
Step 3: Login, cookie wall, interstitial pages
Crawlers don't click anything away. Check three things:
- Login wall: If guide content sits behind a sign-up, it doesn't exist for the AI. Either open it up or consciously accept it as a customer area.
- Cookie wall: A banner that merely overlays the content isn't a problem. A wall that only delivers the content after consent also hides it from crawlers. Test it with the source code trick from step 2.
- Interstitial pages: Age checks, language selection pages without a direct path to the content, and "please wait" pages break the access. Every important page must be reachable directly under its address, without detours.
Step 4: Your host's bot protection
Many hosts and services such as Cloudflare offer a switch that blocks known AI crawlers, sometimes by default. The robots.txt may then be correct, but the crawler is still turned away with a 403 error. Here's how to find out:
- Look in your hosting or Cloudflare dashboard for settings with terms like "AI Bots", "Bot Fight Mode" or "Block AI crawlers".
- Ask your developer to filter last week's server logs for the user agents from the table. A 200 status response means access, 403 means blocked.
- Ask your host if you don't have access to logs. The question "Do you block GPTBot, OAI-SearchBot and PerplexityBot on my website?" can be answered in one sentence.
Allow or block? The decision guide
The question has two levels: training and search. They can be answered separately.
| Your goal | Training crawlers (GPTBot, ClaudeBot) | Search and user crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) |
|---|---|---|
| Appear as a provider in AI answers | Your decision, no direct influence on mentions | Allow |
| Protect content from imitation | Blocking is possible without losing visibility | Allow, otherwise no source citation |
| Customer area or paid content | Block by path, e.g. /kunden/ | Block by path |
| Stay completely invisible | Block | Block, knowing that competitors will then provide the answer |
For most SMEs, the answer is simple: let search crawlers through, and decide on training by gut feeling. The post Appearing in AI answers: three strategies for SMEs describes the strategies that follow for visibility.
What the test doesn't show
The test shows whether an AI is allowed to read. It doesn't show whether it quotes you. For that, the page also has to deliver the answer: up top, clear, with date and sender. What belongs there is covered in the AEO checklist. And whether ChatGPT, Perplexity and Gemini actually find your website today is shown by the free AI check in about a minute, without you having to read a single line of robots.txt.
Frequently asked questions
I've blocked GPTBot. Have I disappeared from ChatGPT?
Not necessarily. OpenAI separates the training crawler GPTBot from the search crawler OAI-SearchBot and from retrieval on user request (ChatGPT-User). If you only block GPTBot, you can still appear in ChatGPT search answers. If you also block OAI-SearchBot, you disappear from those answers.
Does Google-Extended affect whether I appear in the AI overview?
No. According to Google, Google-Extended only controls whether content is used for Gemini applications. Search and the AI overview use the normal Googlebot. If you want to be excluded from the AI overview, you need to use nosnippet or max-snippet, not Google-Extended.
How do I tell if my host is blocking AI bots even though robots.txt allows them?
Check your hosting or Cloudflare dashboard for settings such as "Block AI bots" or "Bot Fight Mode". You'll see it for certain in the server logs: requests with the user agent GPTBot or PerplexityBot that get a 403 status are blocked, regardless of what robots.txt says.
Do I need to list every crawler individually?
No. Anything not listed in robots.txt is allowed. You only need entries if you want to block something. If you want to explicitly allow all AI crawlers, it's enough not to set a disallow rule for them.