What is GPTBot?
Crawls public web content that may be used to train OpenAI's generative AI foundation models.
- Operator
- OpenAI
- Purpose
- AI training
- Feeds
- Training of OpenAI generative AI foundation models
- robots.txt token
GPTBot- Follows robots.txt
- Yes
- IP ranges
- Published list
- Source
- OpenAI documentation
GPTBot user agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Published as an example string; OpenAI says the version number may change. robots.txt fetches may add a 'robots.txt' marker to the string.
Match it in robots.txt with the token GPTBot. User agents can be faked, so check requests against OpenAI's published IP ranges before trusting them. Match the request IP against OpenAI's published JSON list; no reverse-DNS method is documented.
Should you block GPTBot?
Disallowing GPTBot signals your content should not be used to train OpenAI's foundation models; it is independent of OAI-SearchBot, so it does not by itself remove you from ChatGPT search.
Yes. OpenAI says GPTBot follows robots.txt. “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models.”
Block GPTBot
User-agent: GPTBot Disallow: /
Allow GPTBot
User-agent: GPTBot Allow: /
Other OpenAI crawlers
OpenAI uses separate user agents for separate jobs, so you can allow one and block another.
- OAI-SearchBot (AI search index): Crawls sites so they can be surfaced and linked in ChatGPT's search features.
- ChatGPT-User (User-triggered fetch): Visits a page when a ChatGPT or Custom GPT user's request needs it (including GPT Actions); it does not crawl the web automatically.
GPTBot FAQ
What is GPTBot?
Crawls public web content that may be used to train OpenAI's generative AI foundation models.
What is the GPTBot user agent?
GPTBot identifies itself as: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. In robots.txt, match it with the token "GPTBot".
Does GPTBot respect robots.txt?
Yes. OpenAI says GPTBot follows robots.txt.
How do I block GPTBot?
Add "User-agent: GPTBot" followed by "Disallow: /" to the robots.txt file at the root of your site. Disallowing GPTBot signals your content should not be used to train OpenAI's foundation models; it is independent of OAI-SearchBot, so it does not by itself remove you from ChatGPT search.
Crawlable isn't the same as cited
Letting AI crawlers in is step one. Citedify checks whether ChatGPT, Claude, Perplexity and Google AI Overviews actually recommend your brand, and what to fix when they don't.
Facts on this page come from OpenAI's documentation, checked 2026-09-28. Sources: developers.openai.com/api/docs/bots