Robots.txt vs Noindex vs Nofollow: Which One Do You Actually Need?

Every week we audit a site where a page the owner "blocked" still ranks, or a page they wanted indexed silently disappeared. In nearly every case the cause is the same: using one of these three controls when the situation called for another.

They operate at different stages of how a search engine processes a URL:

robots.txt: controlling crawling

robots.txt lives at the root of your host and tells compliant crawlers which paths they may request.

User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /api/
Allow: /api/public/
Sitemap: https://www.example.com/sitemap.xml

What it is for

• Keeping crawlers out of infinite spaces: internal search results, faceted filters that generate millions of combinations, calendar widgets.
• Protecting server resources from crawl load on expensive endpoints.
• Steering crawl budget on very large sites toward the URLs that matter.

What it is not for

Removing pages from Google. If a URL is disallowed but other pages link to it, Google can still index the URL itself, showing it in results with no description ("No information is available for this page"). Worse, if the page has a noindex tag, Google cannot see it, because it is not allowed to fetch the page. The noindex is never read.

Hiding private content. robots.txt is a public file and a polite request. Anyone can read it to find the paths you consider sensitive, and non-compliant bots ignore it. Use authentication.

Common mistakes we find

• Blocking /wp-content/, /assets/ or /_next/. Google needs CSS and JavaScript to render pages; blocking them degrades indexing and can make pages look empty.
• A staging Disallow: / that shipped to production.
• Blocking a directory and expecting its pages to drop out of the index.
• Case sensitivity: Disallow: /Admin does not block /admin.
• A robots.txt that returns 500. Google treats a server error as "crawl nothing" for the whole site until it recovers.

noindex: controlling indexing

noindex is a per-page directive, delivered either as a meta tag in the HTML or as an HTTP header.

<meta name="robots" content="noindex">

X-Robots-Tag: noindex

The header form works for non-HTML files (PDFs, images) and for whole response types at the server level.

What it is for

• Pages that must exist for users but have no search value: login, cart, checkout, account pages, thank-you pages.
• Thin or duplicate pages you want to keep live: tag archives, printer-friendly views, filtered listings.
• Removing a page from search results while keeping it accessible by URL.

How it behaves

Google must crawl the page to see the directive. After it does, the page is dropped from the index on the next processing cycle, usually within days. Over time, Google also crawls noindex pages less often and eventually treats their links as nofollow, so a noindex page is not a good place to park important internal links.

Common mistakes

• noindex combined with robots.txt Disallow on the same URL. The block wins and the noindex is never seen.
• noindex left on a page after a redesign or injected by a tag manager rule. Always check the rendered HTML, not just the template source.
• noindex on paginated category pages beyond page 1. Google recommends letting them index with self-referencing canonicals; noindex severs the crawl path to deep products.
• noindex on a page that is also listed in the XML sitemap. Not harmful, but it sends conflicting signals and clutters the coverage report.

nofollow: controlling link signals

rel="nofollow" on a link tells search engines not to associate your site with the target and not to pass ranking signals through that link.

<a href="https://example.com" rel="nofollow">Example</a>

Since 2019, Google treats nofollow as a hint, not a directive, and introduced two more specific values:

• rel="sponsored" for paid links and ads.
• rel="ugc" for user-generated content such as comments and forum posts.

What it is for

• Paid placements and affiliate links (use sponsored).
• Links in comments and other user-submitted content (use ugc).
• Links to pages you do not vouch for.

What it is not for

• Hiding pages. The target can still be discovered through other links, sitemaps or external sites, and can still be indexed.
• Sculpting PageRank. Internal nofollow to "concentrate" authority has not worked for well over a decade. Signals are not redistributed to your other links; they simply evaporate.
• Keeping Googlebot out of a section. Use robots.txt for crawling or noindex for indexing.

The page-level <meta name="robots" content="nofollow"> applies to every link on the page. It has very few legitimate uses on a normal site.

Decision table

How to audit your current setup

1. Fetch /robots.txt and list every Disallow. For each, ask: do I want these URLs out of the index (then this is the wrong tool) or just uncrawled?
2. Crawl the site and list every URL returning noindex. Compare against the sitemap and against pages with organic traffic in Search Console. Anything valuable on that list is a bug.
3. Use the Search Console URL Inspection tool on three or four important pages. Confirm "Crawl allowed: Yes" and "Indexing allowed: Yes".
4. Search for internal links carrying nofollow. Remove it from internal links unless there is a specific reason.
5. Check the Pages report for "Blocked by robots.txt" and "Excluded by noindex tag" and confirm every URL there is intentional.

Summary

• robots.txt controls crawling. It does not remove URLs from Google and it hides noindex tags from Google.
• noindex controls indexing. It only works on pages Google is allowed to crawl.
• nofollow, sponsored and ugc control link signals. They are not privacy or crawl tools.
• Most "blocked but still ranking" and "disappeared from Google" tickets are one of these three used for the wrong job.