Should you block AI crawlers? GPTBot, ClaudeBot and your robots.txt
George · 3 min read
The short answer: for a new product that wants to be found, keep search engines and AI search crawlers allowed. Blocking crawlers that collect pages to train models is a reasonable choice, and it does not take you out of Google or out of AI answers. Whatever you choose, check what your robots.txt actually says, because templates, hosts and old projects often decide for you.
More and more people ask an AI assistant what tool to use instead of searching. That makes a quiet file at the root of your site, robots.txt, more important than it used to be. One line in it can decide whether an assistant can read your site at all.
Two kinds of AI crawler
AI companies run more than one crawler, and they do different jobs.
Crawlers that collect pages for training. These gather content to train future models. Examples include GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl, and Google-Extended, which is not a separate crawler but a token that controls whether Google may use your pages for its Gemini models.
Crawlers that fetch pages to answer questions. These read a page when someone asks about a topic, and the answer can link back to you. Examples include OAI-SearchBot for ChatGPT search, Claude-SearchBot and Claude-User from Anthropic, and PerplexityBot.
Each company documents its own crawlers, and the list changes, so check their pages when you set your rules.
What blocking each one means
- Block training crawlers: your pages are not collected for future training by the companies that respect it. Your search ranking and your place in AI answers are not affected by this alone.
- Block AI search crawlers: assistants that respect it can not read your pages when someone asks about your topic, so you can be left out of their answers.
- Block search engines: you can drop out of search results altogether. This is almost never what a launching product wants.
How robots.txt decides
robots.txt is a set of groups. Each group starts with one or more User-agent lines and lists Allow and Disallow rules. A crawler follows the group that names it. Only if no group names it does it follow the group for User-agent: *.
That last point surprises people. If you have a group for GPTBot, GPTBot ignores everything in your * group. Within a group, the longest matching rule wins, and Allow wins a tie.
A common setup that blocks training but stays in AI search looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
What robots.txt can not do
robots.txt is a request, not a lock. Well-behaved crawlers follow it; others ignore it. It also does not remove a page from search results by itself: a blocked page can still be listed if other sites link to it. To keep a page out of search results, let crawlers read it and add a noindex tag.
Check what yours really says
It is easy to be wrong about your own robots.txt. Some hosts and security settings add AI rules on their own, a file copied from another project brings its rules along, and a Disallow: / left from development can hide a whole site.
The free robots.txt and sitemap checker reads your file the way crawlers do and marks 16 crawlers, from Googlebot to GPTBot, as allowed or blocked, with the line that decided it. It also checks that your sitemap can be found and read, and warns if your home page still has a noindex.
The short version
- Training crawlers and AI search crawlers do different jobs.
- Blocking training is a choice; blocking search keeps you out of answers.
- A crawler named in its own group ignores the * group.
- robots.txt is a request, and it does not remove pages from search on its own.
- Check what your file actually says before you launch.