Why a page isn’t in Google: noindex and robots.txt

How noindex and robots.txt keep pages out of search, how to find the rule that blocks a page, and how to remove it.

Two different rules can keep a page out of Google, and they work in opposite ways. A noindex rule lets Google fetch the page and asks it not to list it. A robots.txt rule stops Google from fetching the page, but it doesn’t stop Google from listing the address.

Both are easy to leave on by accident, such as a “hide from search engines” setting used while the site was being built, or a header meant for a staging server.

What SignalDNS checks

We fetch the page as Android Chrome, since Google indexes the phone version of a page, and read up to 2 MiB of the final page’s HTML before any JavaScript runs. For “Can search engines list it?” we look at:

  • Robots rules: the robots, googlebot, and bingbot meta tags in the page’s head, and every X-Robots-Tag header on the final response.
  • robots.txt: /robots.txt on the final page’s host, up to 500 KiB as Google reads it, and whether it lets Googlebot fetch the page.
  • Redirects: how many it took to reach the page.
  • The canonical tag and hreflang links, covered in Canonical URLs and hreflang.

The page asks search engines not to list it (noindex)

The page has noindex, or none (noindex, nofollow), in a robots meta tag or the X-Robots-Tag header. Once Google crawls the page and sees it, Google drops the page from its results, even if other sites link to it. SignalDNS marks this as needing attention rather than a problem, because it’s often deliberate, as on a thank-you page.

We count a rule when it applies to Google:

  • <meta name="robots"> applies to every crawler, and <meta name="googlebot"> to Google only.
  • In the header, a rule with no crawler name applies to every crawler. A name in front, as in X-Robots-Tag: googlebot: noindex, limits the rules after it to that crawler, until another crawler is named. A name never carries over to a second X-Robots-Tag header, so X-Robots-Tag: bingbot: noindex followed by X-Robots-Tag: noindex keeps the page out of Google.
  • A rule for another crawler, such as <meta name="bingbot"> or otherbot: noindex in the header, is named in the result but doesn’t affect Google.

nofollow alone doesn’t keep a page out; it only asks crawlers not to follow its links. In the HTML, the rule looks like this:

<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex, nofollow">

In the response headers, which curl -I https://www.example.com/page shows:

HTTP/2 200
content-type: text/html; charset=utf-8
x-robots-tag: noindex

How to fix it: if the page should be in search, find what adds the rule, which our result names, and remove it.

  1. A CMS setting. In WordPress, the “Discourage search engines from indexing this site” box on the Reading settings screen adds noindex to every page, and SEO plugins set it page by page. Shopify uses the seo.hidden metafield, Squarespace a “Hide page from search results” toggle, and Wix “Let search engines index this page”.
  2. A header from the server, CDN, or host. Search their configuration for X-Robots-Tag. Some hosts add it outside production: Cloudflare Pages sends it on every preview deployment.
  3. Framework code. In Next.js, check the robots field in the page’s metadata, such as robots: { index: false }.

Then request indexing in Search Console’s URL Inspection tool. Google has to crawl the page again to see the change, which Google says can take months for some pages.

robots.txt blocks the page

robots.txt, at the root of each host such as https://www.example.com/robots.txt, tells crawlers which paths they may fetch. The result names the rule that blocks the page and its group, and SignalDNS marks it as needing attention.

We read the file the way Google does:

  • One group applies. Googlebot follows the group that names Googlebot, and uses the User-agent: * group only when none does. Groups for the same crawler are combined, but the two are never mixed. Groups for other crawlers, such as Google-Extended, don’t affect Googlebot.
  • The longest matching rule wins. When an Allow and a Disallow rule match with the same length, Allow wins.
  • Paths are case-sensitive and match from the start of the path. * matches any characters, and $ marks the end of the address.

For example:

User-agent: *
Disallow: /

User-agent: Googlebot
Disallow: /checkout/
Allow: /checkout/help

Googlebot uses only its own group, so it may fetch everything except /checkout/, though /checkout/help is allowed because its rule is longer. Other crawlers are blocked everywhere.

robots.txt controls crawling, not indexing. Google can’t read a blocked page, so it can’t see a noindex on it, and if other pages link to it, Google can still list the address without a description.

How to fix it: if the page should be in search, remove or narrow the rule, or add a longer Allow rule for the page. Search Console’s robots.txt report shows the copy Google last fetched and lets you ask for a new fetch. If the file itself fails to load, see When robots.txt can’t be read.

When robots.txt can’t be read

We follow Google’s handling of each status:

  • 404 or 410. There’s no file, so everything is allowed. You don’t need one. OK.
  • Another 4xx, such as 403. Google treats every 4xx except 429 as no file, so everything is allowed. OK.
  • More than five redirects. Google stops after five and treats the file as missing. OK.
  • 5xx, 429, or another status that isn’t success or a missing file. The file is failing, and SignalDNS marks it as needing attention. For a server error, Google stops crawling the site for the first 12 hours while it retries. Then, for up to 30 days, it uses the last copy it fetched, or crawls without restrictions if it has none. After that, if the site is otherwise available, it acts as if there were no robots.txt.
  • No response, or a file we can’t read. A timeout, or a file sent compressed although we asked for none, is “Couldn’t check”. Google treats network failures like server errors, but ours may not happen for Google.

How to fix it: make /robots.txt return 200 with the rules you want, or 404 if you don’t need any. If the whole site is returning errors, start with why a website doesn’t load.

noindex or robots.txt: which to use

To keep a page out of search results, use noindex and let Google crawl the page. Don’t also block it in robots.txt: Google wouldn’t see the noindex, and the address could stay listed without a description.

Use robots.txt to manage crawling, such as keeping crawlers away from endless filter combinations or easing load on your server. Google says it isn’t a way to keep a page out of Google. Related cases:

  • To remove a page that’s already listed, add noindex, make sure robots.txt allows the page, and wait for Google to crawl it. Search Console’s Removals tool hides the address in the meantime, for about six months.
  • For private content, use a password or a login. Robots rules are requests; they don’t stop anyone opening the page.
  • Noindex: lines in robots.txt aren’t supported by Google. We don’t call them a mistake, but they don’t keep a page out.
  • Site-wide switches differ. Squarespace’s “Block Search engine crawlers” adds Disallow: / to robots.txt, while WordPress’s “Discourage search engines” box adds noindex.

The page is three or more redirects away

It took three or more redirects, counting HTTP redirects and meta refresh, to reach the page from the address you entered. The page can still be listed, but each redirect is another request that visitors and crawlers wait through. Googlebot generally follows up to 10, and ignores what a redirecting address sends. SignalDNS marks three or more as needing attention, and one or two as OK. The other checks apply to the page at the end of the chain.

How to fix it: link straight to the final address in menus, sitemaps, and canonical tags, and make each old redirect go there in one step. See three or more redirects.

Setting it up in your CMS or framework

What we can’t tell from outside

We read the head of the HTML before JavaScript runs. Google also follows a robots meta tag in the body, or one a script adds. When Google sees noindex in the HTML as sent, it may not run the page’s scripts, so removing the tag with JavaScript may not work. We fetch as a phone browser, not as Googlebot, so a server that treats Googlebot differently can give Google another answer. And we can’t see what Google has indexed: Search Console’s URL Inspection tool shows that.

Sources

Last reviewed .