Technical / on-page
GPTBot
GPTBot is OpenAI's web crawler, identified by its user agent, used to collect publicly available content for training its models. Site owners can allow or block it via robots.txt. GPTBot is distinct from OAI-SearchBot, the crawler OpenAI uses for live search and referral.
Why it matters for AEO
The GPTBot vs OAI-SearchBot distinction is a high-value practical detail: blocking the training crawler does not necessarily affect search visibility, and vice versa. Understanding which crawler does what lets you make a deliberate choice about how your content is used and whether it can be cited. Documenting your crawler policy in robots.txt, alongside an llms.txt file, makes that choice explicit to every AI system that visits.
Related terms
Newsletter
Get the AEO field notes
Occasional, editorial updates on AI search — what changed, newly verified stats, and citation-tracking notes. No spam, unsubscribe anytime.