Ye ole’ Duplicate Content Penalty

Duplicate Content and the “duplicate content penalty” have been around for 4 years at least – especially since Google started weeding out gazilllions of duplicate datafeed sites in 2004 (including some of mine that were REALLY duplicate before I took countermeasures…) … e.g. Aaron Wall and Caveman wrote about that years ago already.

Talk talk talk …

Duplicate Content has been topic on panels and discussions on ALL those dozens of search marketing conferences I’ve been to the last 5 years …
And even Google finally published an explanantion of dupe content penalties to their blog end of 2006 (wow – quick response) and a year later even to their Google Webmaster Guidelines … (please note how they rewrite their guideline post in a clever way to not trip their own dupe content filters

so for dupe content, web masters have been

trained,
warned

and even been threatened (from scraper sites or malicious competitors) with duplicate content of their website.

Why?

Because once Google detects your site or page to contain duplicate content, it removes it from the results or at least unranks it 100s of positions back.

So we’re urged to create unique and propelling content for the users and the search engines. Or be doomed.

All of us? No, not all of us… I found a nice exception…

Double Standard for Dupe Content?

Based on our recent research on highly trusted domains of Canadian university domains we found at least one interested example where the trust of completely duplicate domains seems to disable what webmaster’s have been trained on for years…

The University of Toronto maintains the utoronto.ca and the toronto.edu domain.

So dig this – they run the same content on both domains, don’t redirect and yet

both rank and got nice traffic

according to Compete.com the dupe domain toronto.edu still had 30k unique visitors last December – 10% of the real site!

No sign of a duplicate content penalty.

Dupe Site Stats compared

Here we go with some of the stats for the two domains

Url	Juice	G!PR	DomKW	DFS	aFD	whoisdate	Y!BL	Y!edu	Y!ac.uk	Y!uniDE	Y!uniCA	Y!gov
http://www.utoronto.ca/	Juice/9	8	38,201	01.08.1995	01.12.1997	01.08.1995	232,000	10,100	746	489	243,734	225
http://www.toronto.edu/	Dupe/22	8	5,713	01.08.1995	01.06.1997	08.05.1986	2,910	1,310	5	16	357	10

I had to split the table as even these few columns break the layout here

 

 

Ok, lot’s of weird headings – I know – what you see is a subset of the 55 different search engine marketing related detail parameters we track historically – time to explain our abbreviations for those common ones we use here explained briefly

Juice	Juice Indicator (cemper.com)
G!PR	Google Page Rank (PR)
Domkw	Keywords driving traffic to the domain (SemRush)
DFS	date first seen using netcraft.com
aFD	First time a domain was cached by archive.org
whoisdate	Domain Creation Date
Y!BL	Yahoo Backlinks
Y!edu	Yahoo .edu domain backlinks
Y!gov	Yahoo .gov domain backlinks
Y!uniDE	Yahoo Backlinks from german universities
Y!uniCA	Yahoo Backlinks from canadian universities
G!BL	Google backlinks (attention: crippled!)
G!idx	Google indexed pages
G!cache	Google page cache date (attention: often crippled!)
Y!BLint	Yahoo internal Backlinks (from a page to its home-site)
Y!LD	Yahoo domain backlinks
Alexa	Alexa traffic rank
Cuqv	Unique Visitors per Month (Compete.com)
M!FL	MSN forward links

Interpretation

Now let’s see what we can read from these few datapoints already

• Both domains seem to exist since 1986 – altough the .ca registry was messed up in 1995. Yes, they are very old & crusty.
• Both domains maintain a PR of 8. No sign of a pagerank removal to signal a penalty
• utoronto.ca ranks for 38,201 unique keyword traffic phrases according to SemRush (an awesome organic keyword tracker!)
• toronto.edu ranks for 5,713 unique phrases according to SemRush
• There are ZERO internal links from toronto.edu going to toronto.edu … Broken? No – this just shows that they just serve the same content made for utoronto.ca user the toronto.edu domain.
• There’s a fraction of backlinks going to the toronto.edu
  • 3000 accidential links you would LOVE to have
  • 1310 accidential .edu links!
  • 357 accidential .ca uni links (not to mention those 16 german and 5 british university links!)
  • 10 accidential .gov links – at least give me those
• Our Juice Indicator tool correctly identifies the .edu as duplicate (“Dupe/22“) – at least the homepage doesn’t rank as it should for unique phrases (well yes, it’s the same stuff as on utoronto.ca which is JUICy – what do you expect)
• The number of Google backlinks reported are EQUAL for both domains. This fact again shows that the Google backlink command was crippled 5 years ago and changed it’s rules internally several times since (as to which weird numbers to show)) it appears that they somehow understand that the two sites are equal, yet let both of them rank
• The Google cache command shows something accurate for the homepage – but note that the Google cache results had been crazy for 2+ years now
  
• Also Live/MSN show 5 million links going OUT of the toronto.edu as opposed to 38.4M of the utoronto.ca – we can assume that those would count in Live.

To sum up, it’s quite interesting to see that BOTH domains rank pretty well, for thousands of phrases – despite the duplicate content.

It’s still about Age & Trust

What’s the double standard here? The duplicate domain toronto.edu simply has

• TONS of great links
• is OLD & CRUSTY
• therefore seems to be TRUSTED ENOUGH by Google to serve 100% dupe content

So again, it’s still a matter of AGE & TRUST - large sites get away with dupe content as we see. Or as Todd Malicoat said years ago already (think it was him) – big brands get away with a lot more than small webmasters!

What would you do if you were the SEO for the University of Toronto?

Redirect the .edu to the utoronto.ca with a 301?
Probably, but what if according to Patrick Altoft the anchor text passing via 301 stops anytime – they would lose rankings from those 3000 links then… I’m sure.

What’s your take?