What is this?
This check fails when the page carries a noindex directive in either of the two places one can live: a meta robots tag in the HTML head, or an X-Robots-Tag HTTP response header. Both mean the same thing — drop this page from the index — and AccessKnight treats them identically, scoring the check at 3 points.
The header version is the one that catches people out. A meta tag is visible in the page source; an X-Robots-Tag is not, and it is often set globally by a server config, a reverse proxy, or a framework's default for a preview environment. A page can look completely clean in the browser and still be carrying a noindex the moment it leaves the server.
It is also worth separating noindex from a robots.txt disallow, because the two fail in opposite directions. A disallow says do not fetch this. A noindex says fetch it, then do not keep it. Combining them is a known trap: a crawler that is disallowed never fetches the page, so it never reads the noindex, and the URL can linger in an index as a bare link. If you want a page gone, let the crawler in so it can read the directive.
Who does this affect, and why does it matter?
Every retrieval surface is affected at once, which is what separates this from the crawler checks. Google Search and AI Overviews, Bing and Copilot, ChatGPT search, Perplexity, and Claude all assemble answers from indexed pages. A directive that removes the page from an index removes it from everything downstream of that index, in one move.
The human cost is usually a launch that quietly never happened. A preview-environment default, a CMS setting that discourages search engines, an SEO plugin's per-page toggle: each is set once and forgotten, and the page ships carrying it. The site looks live, the links work, analytics records the visits of everyone who was sent the URL directly — and the page is simply not in any index.
AccessKnight scores this check at 3 of 30 points, which understates it badly, so the module compensates elsewhere: this is the second of the two gating checks. When it fails, the headline Machine-Readability Score is capped at 40 regardless of how the other seventeen checks score, and the report names the directive as the reason.
The cap is the honest reading of the situation. Three points is what the directive is worth as a line of markup. Being absent from every index is what it costs in practice. A page can earn full marks on crawler access, semantics, structure, and extractability and still be unreachable by every answer engine, because none of them can cite what their index does not contain.
The reason to fix it before anything else is sequence rather than size. Removing a noindex does not put the page anywhere; it makes the page eligible to be indexed, which is the precondition every other improvement depends on. Work spent on schema, heading structure, and answer formatting earns nothing while the directive is still in place, so this is the check to clear before the rest of a scan report is worth reading.
How do I fix it?
Find the directive before you delete anything, because it lives in one of two places: a meta robots tag in the head, or an X-Robots-Tag response header the page source will never show you. Remove it at its source rather than overriding it elsewhere, then keep the crawler allowed so it can read the change.
- View the page source and search the head for a meta tag named robots. A content value containing noindex or none is the directive.
- Check the response headers, which the page source will never show you: run curl -sI https://example.com/page and look for an X-Robots-Tag line.
- Trace whichever one you find back to its source — a CMS setting that discourages search engines, an SEO plugin's per-page indexing toggle, a server or reverse-proxy rule, or a framework default keyed to the deployment environment.
- Remove the directive rather than trying to override it somewhere else. AccessKnight joins the meta tag and the header before testing, so a noindex in either one fails the check, and engines generally resolve a conflict in favour of the more restrictive directive.
- If the page was previously indexed and you want it removed, keep the crawler allowed in robots.txt so it can fetch the page and read the noindex. A disallowed crawler never sees the directive, which is why disallow-plus-noindex leaves URLs stranded in the index.
- Re-scan with AccessKnight and confirm the check reports 3 of 3 and the headline score is no longer capped at 40.
<!-- 1. In the HTML head -->
<meta name="robots" content="noindex, nofollow">
<!-- 2. As a response header, invisible in the page source -->
X-Robots-Tag: noindex<!-- 1. Delete the meta robots tag. Absence already means indexable;
"index, follow" is only ever for making the intent explicit. -->
<meta name="robots" content="index, follow">
<!-- 2. Remove the X-Robots-Tag at its source (server config, reverse
proxy, or a framework default keyed to the environment), then
confirm the header is gone:
curl -sI https://example.com/page | grep -i x-robots-tag -->How does AccessKnight detect this?
AccessKnight reads two sources: every meta tag whose name attribute is robots, matched case-insensitively and joined with commas when the page carries more than one, and the X-Robots-Tag response header captured during the page fetch. It lowercases the combined string, splits it on whitespace, commas, colons, and semicolons, then fails the check if any resulting token is exactly noindex or exactly none. Because the split includes colons, a bot-scoped header such as X-Robots-Tag: googlebot: noindex is caught as well. Matching is on whole tokens, so a longer word that merely contains the letters noindex is not flagged. The report names the source it found — the meta robots tag or the X-Robots-Tag header — and when the meta tag carries the directive, that is the source reported. The check is worth 3 points and is one of the two that cap the headline score at 40.
What strings mean this?
If one of these strings is in front of you right now — in a config file, an HTTP header, an AccessKnight report, or another checker’s output — it is describing the failure explained on this page.
- meta robots tag
- “<meta name="robots" content="noindex">”
- meta robots tag
- “<meta name="robots" content="none">”
- HTTP header
- “X-Robots-Tag: noindex”
- AccessKnight report
- “Page is indexable (no noindex directive)”
- AccessKnight report
- “A noindex directive was found (meta robots tag) — search and AI engines drop this page from their indexes, so it cannot be retrieved or cited by any AI answer surface, regardless of robots.txt.”
- AccessKnight report
- “Capped at 40: the page carries a noindex directive. The site is effectively invisible to major AI surfaces until this is fixed.”
Frequently asked questions
What is the difference between noindex and a robots.txt disallow?
Where do I find an X-Robots-Tag header?
Does 'none' mean the same thing as noindex?
Does a noindex stop AI tools like ChatGPT from using my page?
Why is my AccessKnight score capped at 40 when I only lost 3 points?
Where does this come from?
Every external claim on this page traces to one of the documents below — the vendors’ and standards bodies’ own documentation. What AccessKnight adds is the scan, not the standard.