What is OAI-SearchBot?

OAI-SearchBot user agent

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Published as an example string; the version number may change. The variant carries a 'robots.txt' marker and is used when fetching robots.txt.

Match it in robots.txt with the token OAI-SearchBot. User agents can be faked, so check requests against OpenAI's published IP ranges before trusting them. Match the request IP against OpenAI's published JSON list; no reverse-DNS method is documented.

Should you block OAI-SearchBot?

Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (they can still appear as navigational links); robots.txt changes take about 24 hours to apply.

Yes. OpenAI says OAI-SearchBot follows robots.txt. “we recommend allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges”

User-agent: OAI-SearchBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Other OpenAI crawlers

OpenAI uses separate user agents for separate jobs, so you can allow one and block another.

• GPTBot (AI training): Crawls public web content that may be used to train OpenAI's generative AI foundation models.
• ChatGPT-User (User-triggered fetch): Visits a page when a ChatGPT or Custom GPT user's request needs it (including GPT Actions); it does not crawl the web automatically.

OAI-SearchBot FAQ