who can crawl your website?
paste your address. we read your robots.txt the way crawlers do and show which search engines and AI crawlers it lets in, then check that your sitemap can be found and read.
free to useno signupresults straight away
how it works
- 1
paste your website
we read its robots.txt, its home page and the sitemaps it lists.
- 2
see who is let in
16 crawlers, from Googlebot to GPTBot, each marked allowed or blocked, with the line that decided it.
- 3
fix what you did not mean
a stray Disallow, a missing sitemap or a leftover noindex, each with what to change.
what it checks
- Googlebot, Bingbot, Applebot and DuckDuckBot
- AI search crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot
- AI training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and more
- whether robots.txt exists, and whether it is a real file and not a web page
- the sitemaps robots.txt lists, or /sitemap.xml when it lists none
- that each sitemap is a real urlset or sitemap index, with addresses on your own site
- a noindex left on your home page, in a meta tag or a header
why check robots.txt before you launch
one line can hide a whole site
"Disallow: /" under "User-agent: *" tells every well-behaved crawler to stay out. It is often added while a site is being built and forgotten at launch. The site works perfectly in a browser and never shows up in search, and nothing tells you why.
AI crawlers are not all the same
Some AI crawlers collect pages to train models, like GPTBot and ClaudeBot. Others fetch pages to answer a question someone just asked, with a link back to you, like OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking training crawlers is a reasonable choice. Blocking the search ones means your product can be left out when someone asks an assistant for a tool like yours.
templates and hosts make choices for you
Some site builders, hosts and security settings add rules for AI crawlers on their own, and a robots.txt copied from another project brings its rules along. Reading your robots.txt the way each crawler does is the only way to know what you are actually telling them.
a sitemap is your list of pages
A sitemap tells search engines which pages exist and when they changed. A missing one is not fatal for a small site, but a broken one is worse than none: an address that falls through to your home page, an empty list, or addresses on a different domain. Listing it in robots.txt lets every crawler find it, not only the ones you submitted it to.
what robots.txt can not do
robots.txt is a request, not a lock. Well-behaved crawlers follow it; others ignore it. It also does not take a page out of search on its own: a blocked page can still be listed if other sites link to it. To keep a page out of results, use a noindex tag on a page crawlers are allowed to read.
a few quick answers.
How do I block AI training but stay in AI search?
Add a group for each training crawler with "Disallow: /", such as "User-agent: GPTBot" and "User-agent: ClaudeBot", and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Run the check again to see each one marked the way you meant.
Why does a crawler show "by the * group"?
Your robots.txt has no group naming that crawler, so it follows the rules for "User-agent: *". A crawler that is named in its own group ignores the * group completely, which is how a site can let one crawler in and keep others out.
What if I have no robots.txt?
Then every crawler may read everything, which is fine for most new sites. A robots.txt is still the best place to list your sitemap, so search engines and other crawlers find it.
Does it check every page?
It checks the rules for your home page, which is what decides whether a site is found at all. A crawler marked allowed can still be kept out of other paths, like /admin; the full robots.txt is shown so you can read those rules.
Do you store my site or the results?
No. Your robots.txt, home page and sitemaps are read once to make the report and nothing about them is saved.
ready? launch it on ShipDesk.dev
your own launch page, a link to your site and a full week on the board. free, and it takes about a minute.
launch yours for free