Robots.txt vs Noindex: Which One Should You Use?
Robots.txt controls crawling and noindex controls indexing. Learn the difference, when to use each, and why combining them…
In this guide
A canonical URL tells search engines which version of a page you want indexed when the same or very similar content is reachable at more than one address. Get it right and ranking signals consolidate on one clean URL; get it wrong and Google may index a filtered, parameterised or HTTP version instead of the page you actually care about. This guide shows you how to set canonical tags, how to test them, and how to fix the mistakes that come up most often on small websites.
This guide is part of our complete website SEO audit guide, which walks through every step of auditing a small site.
What a canonical URL is (and what it is not)
Most websites produce duplicate URLs without anyone noticing. A single product page might be reachable at all of these addresses:
https://example.com/blue-mug/https://example.com/blue-mug(no trailing slash)https://www.example.com/blue-mug/https://example.com/blue-mug/?utm_source=newsletterhttps://example.com/shop/kitchen/blue-mug/(category path)
To a person these are one page. To a crawler they are five URLs with identical content. The canonical URL is the one address you nominate as the “real” version. You declare it with a rel="canonical" link element in the page head, and search engines use it as a strong hint when choosing which URL to show in results.
Two points are worth understanding before you start:
- A canonical tag is a hint, not a directive. Google weighs it alongside redirects, internal links, sitemaps and HTTPS. If your signals contradict each other, Google may pick a different canonical URL than the one you declared.
- A canonical tag does not block crawling or indexing on its own. It consolidates duplicates. If you want a page kept out of the index entirely, that is a job for
noindex, which we compare in our guide to robots.txt vs noindex.
How to set a canonical URL correctly
The standard method is a single link element inside the <head> of each page:
<link rel="canonical" href="https://example.com/blue-mug/" />Follow these rules every time:
- Use an absolute URL. Include the protocol and domain. Relative paths like
/blue-mug/are technically allowed but invite mistakes when pages are copied between staging and live environments. - Point to the preferred protocol and host. If your site runs on
https://withoutwww, every canonical URL should say exactly that. - Match your trailing slash convention. Pick one style and use it in canonicals, internal links and the sitemap.
- Add one canonical per page. Multiple canonical tags with different values cause Google to ignore all of them.
- Self-reference on unique pages. Every indexable page should carry a canonical pointing to itself. This protects it when someone links to it with tracking parameters attached.
- Point only to indexable pages. The target must return a 200 status, must not be noindexed and must not be blocked in robots.txt.
If you use WordPress, an SEO plugin usually outputs a self-referencing canonical automatically and lets you override it per post. On other platforms, check your theme or template files for the head section and confirm there is exactly one canonical element.
Other ways to signal the canonical
The link element is the most common method, but there are alternatives:
| Method | Best for | Strength |
|---|---|---|
| 301 redirect | Retired URLs, HTTP to HTTPS, www to non-www | Strongest — users and bots both land on the canonical |
rel="canonical" link element | Duplicates that must stay accessible (parameters, print versions, category paths) | Strong hint |
Link HTTP header | PDFs and other non-HTML files | Strong hint |
| XML sitemap inclusion | Reinforcing the preferred URL across the site | Weak hint |
| Internal linking | Showing which version you actually use | Supporting signal |
For HTTP header canonicals, the syntax looks like this:
Link: <https://example.com/guide.pdf>; rel="canonical"Google documents all of these methods in its guide to consolidating duplicate URLs, which is worth bookmarking.
When to use a canonical URL vs a redirect
A simple test decides it: does anyone still need to reach the duplicate URL?
- If no (an old slug, a retired page, the HTTP version of your site), use a 301 redirect. It is cleaner and stronger.
- If yes (a URL with sorting parameters that shoppers use, a printer-friendly version, the same article syndicated in two sections), keep the page live and use a canonical tag.
Common real-world cases for canonical tags include:
- Product pages reachable through several category paths.
- Faceted navigation and sort orders, such as
?sort=priceor?colour=blue. - Campaign tracking parameters like
utm_source. - Session IDs appended by older e-commerce platforms.
- Content you republish on another domain, where the copy should point back to the original.
Paginated archives are a frequent exception. Page 2 of a blog category is not a duplicate of page 1, so it should normally carry a self-referencing canonical rather than pointing to page 1.
Quick tip: Before adding a canonical to a duplicate, check where your internal links point. If your menu links to the non-canonical version, you are sending Google mixed signals. Fix the links first, then the tag.
How to test canonical tags on your site
Testing happens at two levels: one page at a time, and across the whole site.
Check a single page
- Open the page, right-click and choose View page source.
- Search the source for
rel="canonical". - Confirm there is exactly one, that it uses an absolute URL, and that the target is the version you expect.
- Open the canonical target in a new tab and confirm it loads with a 200 status rather than redirecting.
If you prefer not to dig through source code, the free LiteScroll SEO Inspector Chrome extension shows a page’s metadata while you browse, which makes spot-checking canonicals on a handful of pages quick.
Check canonicals across the site
Page-by-page checks do not scale. A crawler visits each URL and records its canonical URL so you can spot patterns. The LiteScroll SEO Crawler scans up to 25 pages free and flags canonical configuration alongside titles, meta descriptions, H1s and indexability, then lets you download the results as a PDF or CSV. In the export, sort by the canonical column and look for:
- Pages with no canonical URL declared at all.
- Canonicals that point to a different URL than the page itself, and whether that is intentional.
- Canonicals that use HTTP, the wrong host or the wrong trailing slash.
- Several unrelated pages canonicalised to the homepage.
If you are new to working through exports like this, our guide on how to read an SEO crawl report explains how to prioritise what you find.
Confirm what Google chose
In Google Search Console, the URL Inspection tool shows both the “User-declared canonical” and the “Google-selected canonical”. When they differ, Google has overruled your hint, which usually means your signals conflict.
Common canonical URL mistakes and how to fix them
These are the problems that turn up again and again during audits of small sites:
| Mistake | Why it hurts | Fix |
|---|---|---|
| Canonical URL points to a redirect | Creates a chain of conflicting signals | Update the tag to the final destination URL |
| Canonical points to a noindexed or 404 page | The preferred page cannot be indexed, so neither version may rank | Point to a live, indexable page |
| Every page canonicalised to the homepage | Tells Google your inner pages are duplicates of the homepage | Switch to self-referencing canonicals |
| Canonical and noindex on the same page | Contradictory instructions | Choose one: consolidate with canonical, or exclude with noindex |
| Multiple canonical tags | Google may ignore all of them | Remove duplicates, often added by a theme and a plugin at once |
| HTTP canonicals on an HTTPS site | Points signals at the insecure version | Update templates to use HTTPS |
| Canonical injected only by JavaScript | May be missed or processed late | Output the tag in the server-rendered HTML |
| Staging domain in canonicals | Sends signals to a site you do not want indexed | Search and replace the domain after migration |
The “multiple canonicals” problem deserves special mention on WordPress. If you switch SEO plugins, or your theme outputs its own head tags, you can end up with two canonical elements. View source and search for canonical to catch it.
A canonical URL checklist for small sites
Run through this list whenever you launch a site, migrate, or change your SEO plugin:
- Every indexable page has exactly one self-referencing canonical.
- Every canonical URL uses HTTPS and your preferred host (www or non-www).
- Trailing slash style matches internal links and the XML sitemap.
- No canonical points to a redirect, a 404 or a noindexed page.
- Parameter and tracking URLs canonicalise to the clean version.
- Your sitemap lists only canonical URLs.
- Internal links point to canonical URLs, not to duplicates.
- Search Console shows the Google-selected canonical matching yours for key pages.
Canonical issues often sit next to other crawl and indexing problems, so pair this list with our indexability checklist for a fuller picture.
Next steps
Canonical URLs are one of those settings that work silently when correct and cause confusing ranking problems when wrong. Start with a quick crawl to see what canonical each page declares, fix anything pointing at redirects, error pages or the wrong host, then make sure your internal links and sitemap agree with your tags. Once the signals line up, Google has every reason to respect your choice.
You can find more free, lightweight tools for checks like this at LiteScroll.com.
