What is this?
This check fails when your site's robots.txt disallows OAI-SearchBot, OpenAI's crawler for ChatGPT search. A blocked retrieval crawler cannot fetch the page, so the page cannot be used as a source for a ChatGPT search answer. AccessKnight scores it at 8 points, the heaviest single check in the AI Crawler Access pillar.
OpenAI runs two independent crawlers with two different jobs. GPTBot collects content for model training. OAI-SearchBot fetches pages so ChatGPT search can quote them in a live answer. They read separate robots.txt groups and neither decision affects the other, which is why blocking GPTBot does not remove you from ChatGPT search, and why allowing GPTBot does not put you back into it while OAI-SearchBot is disallowed.
Most blocks are not deliberate. Three patterns account for nearly all of them: a blanket User-agent: * group with Disallow: / left over from a staging environment; an SEO plugin whose block-AI-crawlers toggle writes the rule for you without naming which crawlers it covers; and a CDN or WAF bot rule that rejects the request before robots.txt is ever read. The third is invisible in robots.txt, so a file that looks correct can still leave the crawler shut out.
Who does this affect, and why does it matter?
ChatGPT search is affected directly and immediately, because it is the surface OAI-SearchBot feeds. Anything built on OpenAI's search stack inherits the same block. Google AI Overviews, Perplexity, Copilot, and Claude are unaffected by this specific directive, because each uses its own crawler and its own index — this rule closes one door, not the corridor.
The person who absorbs the loss is the site owner, usually without knowing it. A disallowed retrieval crawler produces no error, no email, and no report anywhere in your own tooling. The page keeps ranking in Google, keeps earning traffic, and simply never turns up as a source when someone asks ChatGPT a question the page answers well.
AccessKnight scores this check at 8 of 8 points, the heaviest single check in the AI Crawler Access pillar. It is also one of only two gating checks in the whole scan: when it fails, the headline Machine-Readability Score is capped at 40 regardless of how well the other seventeen checks score, and the report states both the cap and the reason for it.
The cap is deliberate rather than punitive. A perfect score on structure, semantics, and extractability describes a page that is beautifully built for a retrieval crawler that has been told not to fetch it. The asymmetry is worth stating plainly: blocking OAI-SearchBot means the page cannot be retrieved for ChatGPT search, while allowing it only makes the page eligible to be retrieved. Being fetchable is a precondition, not an outcome.
The practical consequence is one of influence rather than traffic. When someone asks an assistant a question your page answers, the assistant assembles its answer from whatever it can reach. A disallowed retrieval crawler removes your page from that pool entirely, so the answer gets built from pages that did let the crawler in — and because there is no click, there is no line in your analytics telling you it happened.
How do I fix it?
Add an explicit OAI-SearchBot group to robots.txt with Allow: /, because a group naming the crawler overrides the wildcard rules that cause most of these blocks. Then rule out the two causes robots.txt cannot show you: a plugin that wrote the rule for you, and a CDN rejecting the crawler before the file is ever read.
- Open your robots.txt at https://yourdomain.com/robots.txt and look for any group that disallows OAI-SearchBot, including a blanket User-agent: * group with Disallow: / that would catch it by default.
- Add an explicit User-agent: OAI-SearchBot group containing Allow: /. A group that names the crawler wins over the wildcard * group, so the specific rule is the one that decides access.
- Write the token exactly as OAI-SearchBot, with no version suffix. The parser matches user-agent tokens, so a group written as User-agent: OAI-SearchBot/1.0 is not recognised as that crawler's group and the wildcard rules apply instead.
- Check your CMS or SEO plugin for an AI-crawler blocking toggle. Several write the disallow into robots.txt for you, and most do not distinguish retrieval crawlers from training crawlers.
- Test the file as the crawler itself, which catches an edge block that robots.txt cannot describe: curl -sA "OAI-SearchBot" https://example.com/robots.txt. Getting the file back means the path is open; a 403, a challenge page, or a timeout means your CDN or WAF is rejecting the request before robots.txt is ever consulted, and the fix belongs in the bot rules there rather than in the file.
- Re-scan with AccessKnight and confirm the check reports 8 of 8 and the headline score is no longer capped at 40.
User-agent: *
Disallow: /
User-agent: OAI-SearchBot
Disallow: /User-agent: *
Disallow: /admin/
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /How does AccessKnight detect this?
AccessKnight fetches /robots.txt from the origin of the final URL after redirects, parses it into per-user-agent groups, and resolves the rules for OAI-SearchBot against the path / — the site root, not the scanned page's path. A group naming the crawler wins over the wildcard * group, which is consulted only when no specific group matches; inside the winning group the longest matching rule decides and Allow wins a tie. An empty Disallow: line counts as allow-all at the lowest specificity, so any more specific rule overrides it. If robots.txt is missing, unreachable, or served as an HTML soft-404, everything is treated as allowed and the check passes. It is worth 8 points and is one of two checks that cap the headline score at 40. One limitation worth knowing, and how to cover it yourself: the fetch is made with AccessKnight's own user agent, so the check reports the policy your robots.txt declares rather than whether OpenAI's crawler is actually served when it arrives. To test that second question, request the file as the crawler — curl -sA "OAI-SearchBot" https://example.com/robots.txt — and treat a 403, a challenge page, or a timeout as an edge block rather than a robots.txt problem.
What strings mean this?
If one of these strings is in front of you right now — in a config file, an HTTP header, an AccessKnight report, or another checker’s output — it is describing the failure explained on this page.
- robots.txt
- “User-agent: OAI-SearchBot”
- robots.txt
- “Disallow: /”
- User-agent
- “OAI-SearchBot”
- AccessKnight report
- “OAI-SearchBot is blocked — you will not appear in ChatGPT search, even if GPTBot already crawled you for training. The two systems are independent.”
- AccessKnight report
- “Capped at 40: OAI-SearchBot (ChatGPT search) is blocked in robots.txt. The site is effectively invisible to major AI surfaces until this is fixed.”
Frequently asked questions
What is the difference between OAI-SearchBot and GPTBot?
Does blocking GPTBot remove me from ChatGPT search?
I allowed OAI-SearchBot but the check still says it is blocked. Why?
Why is my AccessKnight score capped at 40?
Do I still need Bingbot allowed if OAI-SearchBot is allowed?
Where does this come from?
Every external claim on this page traces to one of the documents below — the vendors’ and standards bodies’ own documentation. What AccessKnight adds is the scan, not the standard.