Duplicate Content: Diagnosis Before Medicine

There is no duplicate content penalty. Google filters duplicates at serving time; it doesn't punish you for having them. The real costs are quieter: link signals split across variants, crawl budget burned on copies, and Google occasionally choosing the wrong URL as canonical. Those are three different diseases with three different treatments, and most sites apply the medicine — usually a blanket rel=canonical — before doing any diagnosis. Spend an hour identifying which kind of duplication you actually have first.

The three kinds of duplication

Exact technical duplicates. The same document served at multiple URLs: http vs https, www vs bare, trailing slash vs not, ?utm_source= and other parameters, uppercase path variants. These are infrastructure bugs, not content decisions.

Near-duplicates. Pages that are 80–95% identical: faceted URLs (?sort=stars&license=mit), pagination, print views, and — the programmatic-SEO special — template pages where the boilerplate outweighs the unique data. Two app pages that share 1,900 words of template and differ in 60 words of specs are near-duplicates to a search engine, which is the mechanism behind most thin-content-at-scale indexing collapses.

Cross-domain duplicates. Syndication you agreed to and scraping you didn't.

Find exact duplicates with a hash pass

On a static site this takes one command against the build output:

# byte-identical pages in the build
find dist -name '*.html' -exec md5sum {} + | sort | uniq -w32 --all-repeated=separate

Byte-identical is rare in practice because build timestamps, nonces, and CSRF tokens vary per page. Hash the extracted main content instead:

find dist -name '*.html' | while read -r f; do
  htmlq -t 'main' < "$f" | tr -d '[:space:]' | md5sum | awk -v f="$f" '{print $1, f}'
done | sort | uniq -w32 --all-repeated=separate

On this site's ~54,000-page build that pass runs in about two minutes and has caught real bugs twice: a category filter that generated the same listing under two slugs, and a comparison template that rendered identically when both apps lacked spec data. For near-duplicates, the same approach with shingling or simhash works, but honestly a desktop crawler's near-duplicate report at a 90% similarity threshold gets you the list with less code.

Read Search Console's indexing report properly

The Pages report names duplicate states precisely, and each one means something different:

The third row is the one that deserves attention. Google overrules canonicals when other signals disagree: internal links pointing at the non-canonical variant, the sitemap listing the "wrong" URL, or redirects that contradict the tag. The canonical element is a hint, not a directive — when Google ignores yours, the fix is aligning every other signal, not repeating the hint louder.

The fix matrix

Two rules cover most mistakes. First, never robots.txt-block a URL you want consolidated — a blocked page can't be crawled, so its canonical tag is never seen and its signals never merge; blocking is for crawl-budget protection on pages you don't care about, which is a crawl budget decision, not a duplication fix. Second, don't stack noindex with a canonical pointing elsewhere — the two signals contradict each other (one says "this page is a copy of X", the other says "remove this page"), and Google's guidance on consolidating duplicate URLs is explicit that you pick one intent per URL.

Syndication changed in 2023 and most advice hasn't caught up

Google removed cross-domain rel=canonical from its syndication guidance in 2023. The current position: if you syndicate your content out and don't want the copies outranking you, the copies should carry noindex — a canonical on the syndicated copy is no longer the recommended mechanism, because syndicated pages differ (different boilerplate, ads, wrappers) and Google frequently ignored those canonicals anyway. Negotiate noindex into the syndication deal or accept that a stronger domain may outrank you with your own words. A visible link back to the original helps attribution but guarantees nothing about ranking.

Scrapers, the involuntary version: mostly ignore them. Google is good at original-source attribution when your page was indexed first — which is one more reason honest, fast indexing pipelines matter. DMCA requests are worth the ten minutes only when a scraper actually outranks you on queries that convert.

Bottom line

Run the hash pass, read the GSC duplicate rows, and only then choose a tool: redirects for URLs that shouldn't exist, canonicals for variants that must coexist, noindex for pages that should vanish, and content differentiation — not canonicals — for template pages that are 90% boilerplate. If Google is overriding your canonicals, stop declaring and start aligning: internal links, sitemap entries, and redirects all voting for the same URL, the same discipline that makes canonical edge cases tractable. The diagnosis takes an afternoon. Prescribing before it is how sites end up noindexing pages that were earning traffic.