Robots.txt vs Noindex: Which One Should You Use?

Robots.txt vs noindex is one of the most misunderstood choices in technical SEO, because both seem to “hide” pages from Google. In reality they do different jobs: robots.txt controls crawling, while noindex controls indexing. Use the wrong one and you can end up with private pages showing in search results, or important pages quietly disappearing. This guide explains how each works, when to use which, and the one combination you should never use.

This guide is part of our complete website SEO audit guide, which walks through every step of auditing a small site.

What robots.txt actually does

Robots.txt is a plain text file at the root of your domain, for example https://example.com/robots.txt. It gives crawlers instructions about which paths they may request. A simple file looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /*?sort=

Sitemap: https://example.com/sitemap_index.xml

Key facts about robots.txt:

• It controls crawling, not indexing. A disallowed URL can still be indexed if other pages link to it. Google may show it in results with a bare URL and no description, because it was never allowed to read the page.
• It works at path level. You can block whole folders or URL patterns with a few lines, which makes it efficient for large groups of URLs.
• It is public. Anyone can read your robots.txt, so listing secret folders in it advertises them.
• It is voluntary. Reputable crawlers respect it; scrapers and malicious bots may not.

Google’s own introduction to robots.txt states plainly that it is not a mechanism for keeping a web page out of Google.

What noindex actually does

Noindex is an instruction placed on the page itself, telling search engines not to include it in their index. There are two ways to add it:

As a meta tag in the HTML head:

<meta name="robots" content="noindex, follow">

Or as an HTTP response header, which also works for PDFs and other non-HTML files:

X-Robots-Tag: noindex

Key facts about noindex:

• It controls indexing. Once Google crawls the page and sees the tag, the page is dropped from results.
• It works at page level. Each page needs its own tag or header, though CMS plugins can apply it to whole post types or archives.
• It requires crawling. Google must be able to fetch the page to see the instruction. This is the crucial point most people miss.
• Links can still be followed. With noindex, follow (the default behaviour), crawlers can still discover pages linked from the noindexed page.

Robots.txt vs noindex: side-by-side comparison

The short version of robots.txt vs noindex: if you want a page out of search results, use noindex. If you want crawlers to stop requesting a group of URLs, use robots.txt.

The mistake to avoid: blocking a noindexed page

The most damaging robots.txt vs noindex error is using both at once. Many site owners add a noindex tag and also disallow the same URL in robots.txt, reasoning that two locks are better than one. It works the opposite way:

1. Robots.txt tells Google not to crawl the page.
2. Because Google cannot crawl it, it never sees the noindex tag.
3. If the URL has links pointing to it, Google can still index it, without knowing you asked it not to.

The result is the exact outcome you were trying to prevent. If a page is already showing in results and you want it gone, make sure it is crawlable, add noindex, and wait for Google to recrawl it. Only once it has dropped out should you consider blocking crawling, and usually you do not need to at all.

When to use robots.txt vs noindex: real scenarios

Here is how the decision plays out for pages small sites commonly need to manage:

Use noindex for

• Thank-you and confirmation pages after a form submission or purchase.
• Thin archive pages, such as tag archives with one or two posts or date-based archives on a blog.
• Internal search results pages that users can reach but that offer no unique value in Google.
• Staging or test pages that must stay accessible to your team (better still, put staging behind a password).
• Legal boilerplate you are happy for users to find on your site but not in search results.

Use robots.txt for

• Admin and back-end paths such as /wp-admin/.
• Infinite URL spaces created by calendar widgets, filter combinations or sort parameters, where crawlers could waste time on thousands of near-identical URLs.
• Cart and checkout steps that hold no value for search.
• Resource-heavy endpoints that put load on your server when crawled.

Use neither for

• Duplicate pages you want consolidated. A canonical tag is the right tool; see our canonical URL guide.
• Genuinely private content. Use password protection or authentication. Neither robots.txt nor noindex stops a person from opening the URL.
• CSS and JavaScript files your pages need to render. Blocking them can stop Google seeing your layout and content properly.

How to check robots.txt and noindex on your pages

Because these settings are invisible on the page itself, they are easy to forget. Build a habit of checking:

1. Read your robots.txt. Open /robots.txt in a browser and read every line. Look for leftover rules like Disallow: / from a development site.
2. Check individual pages. The free LiteScroll SEO Inspector Chrome extension shows a page’s metadata as you browse, so you can confirm whether a robots meta tag is present without opening the source.
3. Check at scale. The LiteScroll SEO Crawler scans up to 25 pages free and flags indexability, which quickly reveals pages carrying an unexpected noindex.
4. Confirm with Google. Search Console’s URL Inspection shows whether a URL is blocked by robots.txt or excluded by noindex, and the Pages report groups these across the site.

Remember that noindex can also be sent as an HTTP header, which never appears in the HTML. If a page is excluded and you cannot find a meta tag, check the response headers in your browser’s developer tools under the Network tab. Our indexability checklist covers this and eleven other reasons pages fail to get indexed.

A simple decision checklist

When you are weighing robots.txt vs noindex for a specific page and are unsure which to use, answer these questions in order:

• Should the content be private? Use authentication, not robots.txt or noindex.
• Is it a duplicate of another page you want to rank? Use a canonical tag.
• Should people be able to visit it, but not find it in Google? Use noindex and keep it crawlable.
• Is it a large group of low-value URLs that wastes crawl activity, and is none of it already indexed? Use robots.txt.
• Is the page already indexed and you want it removed? Use noindex first; block crawling only after it has dropped out, if at all.

If you believe that blocking a page in robots.txt removes it from Google, you are not alone; it is one of the classic technical SEO myths small site owners still believe.

Key takeaways

• Robots.txt controls crawling; noindex controls indexing. They are not interchangeable.
• To keep a page out of search results, use noindex and leave it crawlable.
• Never combine a robots.txt block with noindex on the same URL if your goal is removal.
• Use robots.txt to stop crawlers wasting time on admin paths and endless parameter URLs.
• Check both settings after every launch, migration or plugin change.

Settling the robots.txt vs noindex question for each section of your site takes a few minutes and prevents some of the most frustrating indexing problems. For more practical, lightweight SEO checks, head to LiteScroll.com.