Duplicate content checker
Compares content pages at an 80% similarity threshold, skipping archives and pagination.
What TMOD checks
- Extracts the visible text of every crawled page and compares each pair, reporting any that are 80% or more similar.
- Excludes structurally non-unique pages, archives, category listings and paginated views, detected by structure rather than URL pattern, so it works on any CMS and in any language.
- Reports the specific pair of URLs involved rather than just a count, so you can see immediately whether it is one article at two addresses or a template producing near-identical pages.
Why it matters
Internal duplication splits the signals for a topic across several URLs. Instead of one page that clearly covers something, you have three that partly cover it, competing with each other. Search engines pick one and often not the one you would have chosen.
For AdSense specifically, duplication reads as a padding tactic, a site inflating its page count without adding value. That interacts badly with the page-count and thin-content checks: a site that appears to have thirty pages but really has eight articles repeated is judged on the eight.
The great majority of flags here are not plagiarism or laziness. They are the same page reachable at multiple URLs: with and without www, http and https, with and without a trailing slash, with tracking parameters, or a printer-friendly variant. Those are canonicalization problems wearing a duplicate-content costume, and they are usually fixable in one template change.
How to fix it
01Check canonicalization first
Before rewriting anything, add self-referential canonical tags site-wide and make sure http redirects to https and one of www/non-www redirects to the other. This resolves the majority of duplicate flags without touching a single word of content.
02Merge genuine duplicates and redirect
Where two pages really do cover the same ground, pick the stronger one, fold anything unique from the other into it, and 301 the loser to the winner. Deleting without redirecting throws away whatever links and history the old URL had.
03Differentiate templated pages properly
If your location or product pages differ only by a name, they will trip this check and the scaled-content check together. Each needs genuinely specific content, different details, different examples, or it should not exist as a separate indexed page.
04Handle parameters at the source
Tracking, sorting and filtering parameters generate near-infinite URL variants of the same content. A self-referential canonical on the clean URL handles this. Blocking parameters in robots.txt does not, because a blocked URL can still be indexed from external links.
Canonical tags and the three ways they go wrong
A canonical tag is a hint, not an instruction. Search engines take it seriously when it agrees with everything else they can see, and quietly ignore it when it does not, which is why a canonical alone rarely fixes a duplicate problem that the rest of the site is contradicting.
The first failure is the site-wide canonical pointing at the homepage, usually a template variable that was never filled in. Every article on the site then declares itself a copy of the front page. The second is a canonical pointing at a URL that redirects, 404s or is blocked from crawling, which asks a crawler to prefer a page it cannot have. The third is a canonical that disagrees with your own sitemap or your internal links, where the site says one thing in the head of the page and another everywhere else.
The reliable version is dull. Every page declares itself, absolute URL, one protocol, one hostname, one trailing-slash convention, and the redirects, the internal links and the sitemap all agree with it. Consistency is doing more work here than the tag is.
Duplication you do not fully control
Some overlap comes from outside. Syndicating an article to a partner site, republishing on a platform, or letting an aggregator carry your text all create copies you did not make. The convention is that the copy carries a canonical back to your original, or is noindexed. Ask for it when you agree to the syndication, because asking afterwards rarely works.
Scrapers are the version people worry about most and the one worth worrying about least. Automated copies of a site are common, and search engines are generally good at resolving which version came first, particularly when the original is well linked and indexed quickly. It is a nuisance rather than a ranking event, and there is no version of this check that would tell you about it anyway, since it compares your pages against each other rather than against the web.
The one to take seriously is your own staging site. A development copy left publicly reachable and crawlable duplicates the entire production site, at a different hostname, and it does it while you are not looking. Password-protect it or serve it a noindex header, and check what a crawler can actually reach rather than assuming obscurity is protection.
Questions
Does this check whether my content was copied from other sites?
No. This compares your pages against each other, not against the wider web. External plagiarism detection needs an index of the entire internet, which is a fundamentally different product. The separate copyright check looks at intellectual-property risk signals on your pages, but it is not a plagiarism scanner either.
Why 80% similarity?
It is high enough that pages sharing a header, footer, navigation and sidebar do not trip it, those are identical across a site by design and are excluded from the comparable text anyway. Two pages that still read 80% the same after that are substantively the same page. Lowering the threshold produces mostly false positives on any templated site.
My category and tag pages overlap heavily. Will they be flagged?
No. Archive, category and paginated pages are detected structurally and excluded before comparison, because they overlap by design and no search engine treats them as duplicate content. Only real content pages are compared.