AI crawlers
AI crawlers are the automated agents that fetch web pages on behalf of AI systems, and they are not one family: some collect material to train models, others fetch sources to answer a question being asked right now.
Treating them as one group is the most expensive mistake in this field, because the rule you write to keep your content out of a training set can also remove you from the answers where you wanted to appear.
What is the difference between a training crawler and an answer crawler?
OpenAI documents three bots and they do different jobs. GPTBot gathers content that may be used to train foundation models. OAI-SearchBot exists to surface websites in ChatGPT's search results. ChatGPT-User visits a page because a person just asked something that needs it.
Only the first has anything to do with training. The other two are how you get cited, and they are governed by separate rules in your robots.txt. Other vendors draw the same line with their own names.
What happens if you block GPTBot?
You keep your content out of that training corpus, which is a legitimate choice and for some publishers the right one. What you do not do is stop AI from citing you, because citation comes from the search and user-triggered bots, which are governed by different rules.
The reverse mistake is more common and more damaging: a blanket rule that blocks everything with the word bot in it, written by someone who meant to protect against training, taking the answer crawlers with it. The site then disappears from answers and nobody connects the two events.
We see the accidental version constantly. Of the 321 websites in our own calibration sample, 51 returned a firewall block before robots.txt was ever read: 16% of the sample, shut to a crawler whose rules nobody got to state.
How do you tell them apart?
Three things to check, in order. The first two take minutes and the third is where most sites discover the problem they did not know they had.
- Read the vendors' published documentation: each names its bots and states what each one is for.
- Write explicit rules per agent in robots.txt. An explicit Allow states intent; an absent rule states nothing.
- Check what your firewall or CDN does, because it can block a crawler before robots.txt is ever read, and it will not tell you.
What is this based on?
The vendors' own documentation and the standard that governs how the rules are read. All three are public and free.
- OpenAI crawler documentation
Names GPTBot, OAI-SearchBot and ChatGPT-User and states what each is for. It is the clearest published evidence that training and answering are separate jobs.
- RFC 9309, the Robots Exclusion Protocol
Settles how a robots.txt is read, including that the longest matching pattern wins and that Allow wins a tie. Most accidental blocks are a misreading of exactly this.
- The llms.txt convention
What you offer an agent once you have let it in. Permission and orientation are separate problems and they are often confused.
Frequently asked questions about AI crawlers
Does blocking GPTBot stop AI from citing me?
- No. GPTBot collects training material, while citation comes from the search and user-triggered crawlers, which follow separate rules. Blocking one and not the others is a coherent position: stay out of the training corpus, remain available as a source.
Should I block AI crawlers?
- The answer depends on what you sell. If your business is the content itself, keeping it out of a training corpus is defensible. If your business is being found, blocking the answer crawlers removes you from the place where your customers are now asking, and most blanket rules do that by accident.