Ye ole’ Duplicate Content Penalty

Duplicate Content and the “duplicate content penalty” have been around for 4 years at least – especially since Google started weeding out gazilllions of duplicate datafeed sites in 2004 (including some of mine that were REALLY duplicate before I took countermeasures…) … e.g. Aaron Wall and Caveman wrote about that years ago already.

Talk talk talk …

Duplicate Content has been topic on panels and discussions on ALL those dozens of search marketing conferences I’ve been to the last 5 years …
And even Google finally published an explanantion of dupe content penalties to their blog end of 2006 (wow – quick response) and a year later even to their Google Webmaster Guidelines … (please note how they rewrite their guideline post in a clever way to not trip their own dupe content filters

so for dupe content, web masters have been

trained,
warned

and even been threatened (from scraper sites or malicious competitors) with duplicate content of their website.

Why?

Because once Google detects your site or page to contain duplicate content, it removes it from the results or at least unranks it 100s of positions back.

So we’re urged to create unique and propelling content for the users and the search engines. Or be doomed.

All of us? No, not all of us… I found a nice exception…

Double Standard for Dupe Content?

Based on our recent research on highly trusted domains of Canadian university domains we found at least one interested example where the trust of completely duplicate domains seems to disable what webmaster’s have been trained on for years…

The University of Toronto maintains the utoronto.ca and the toronto.edu domain.

So dig this – they run the same content on both domains, don’t redirect and yet

both rank and got nice traffic

according to Compete.com the dupe domain toronto.edu still had 30k unique visitors last December – 10% of the real site!

No sign of a duplicate content penalty.

Dupe Site Stats compared

Here we go with some of the stats for the two domains

Url	Juice	G!PR	DomKW	DFS	aFD	whoisdate	Y!BL	Y!edu	Y!ac.uk	Y!uniDE	Y!uniCA	Y!gov
http://www.utoronto.ca/	Juice/9	8	38,201	01.08.1995	01.12.1997	01.08.1995	232,000	10,100	746	489	243,734	225
http://www.toronto.edu/	Dupe/22	8	5,713	01.08.1995	01.06.1997	08.05.1986	2,910	1,310	5	16	357	10

I had to split the table as even these few columns break the layout here

 

 

Ok, lot’s of weird headings – I know – what you see is a subset of the 55 different search engine marketing related detail parameters we track historically – time to explain our abbreviations for those common ones we use here explained briefly

Juice	Juice Indicator (cemper.com)
G!PR	Google Page Rank (PR)
Domkw	Keywords driving traffic to the domain (SemRush)
DFS	date first seen using netcraft.com
aFD	First time a domain was cached by archive.org
whoisdate	Domain Creation Date
Y!BL	Yahoo Backlinks
Y!edu	Yahoo .edu domain backlinks
Y!gov	Yahoo .gov domain backlinks
Y!uniDE	Yahoo Backlinks from german universities
Y!uniCA	Yahoo Backlinks from canadian universities
G!BL	Google backlinks (attention: crippled!)
G!idx	Google indexed pages
G!cache	Google page cache date (attention: often crippled!)
Y!BLint	Yahoo internal Backlinks (from a page to its home-site)
Y!LD	Yahoo domain backlinks
Alexa	Alexa traffic rank
Cuqv	Unique Visitors per Month (Compete.com)
M!FL	MSN forward links

Interpretation

Now let’s see what we can read from these few datapoints already

• Both domains seem to exist since 1986 – altough the .ca registry was messed up in 1995. Yes, they are very old & crusty.
• Both domains maintain a PR of 8. No sign of a pagerank removal to signal a penalty
• utoronto.ca ranks for 38,201 unique keyword traffic phrases according to SemRush (an awesome organic keyword tracker!)
• toronto.edu ranks for 5,713 unique phrases according to SemRush
• There are ZERO internal links from toronto.edu going to toronto.edu … Broken? No – this just shows that they just serve the same content made for utoronto.ca user the toronto.edu domain.
• There’s a fraction of backlinks going to the toronto.edu
  • 3000 accidential links you would LOVE to have
  • 1310 accidential .edu links!
  • 357 accidential .ca uni links (not to mention those 16 german and 5 british university links!)
  • 10 accidential .gov links – at least give me those
• Our Juice Indicator tool correctly identifies the .edu as duplicate (”Dupe/22“) – at least the homepage doesn’t rank as it should for unique phrases (well yes, it’s the same stuff as on utoronto.ca which is JUICy – what do you expect)
• The number of Google backlinks reported are EQUAL for both domains. This fact again shows that the Google backlink command was crippled 5 years ago and changed it’s rules internally several times since (as to which weird numbers to show)) it appears that they somehow understand that the two sites are equal, yet let both of them rank
• The Google cache command shows something accurate for the homepage – but note that the Google cache results had been crazy for 2+ years now
  
• Also Live/MSN show 5 million links going OUT of the toronto.edu as opposed to 38.4M of the utoronto.ca – we can assume that those would count in Live.

To sum up, it’s quite interesting to see that BOTH domains rank pretty well, for thousands of phrases – despite the duplicate content.

It’s still about Age & Trust

What’s the double standard here? The duplicate domain toronto.edu simply has

• TONS of great links
• is OLD & CRUSTY
• therefore seems to be TRUSTED ENOUGH by Google to serve 100% dupe content

So again, it’s still a matter of AGE & TRUST - large sites get away with dupe content as we see. Or as Todd Malicoat said years ago already (think it was him) – big brands get away with a lot more than small webmasters!

What would you do if you were the SEO for the University of Toronto?

Redirect the .edu to the utoronto.ca with a 301?
Probably, but what if according to Patrick Altoft the anchor text passing via 301 stops anytime – they would lose rankings from those 3000 links then… I’m sure.

What’s your take?

.edu links have been a buzzworthy topic for years now, thanks to the trust and authority those domains have. This led us to compare take a look at the often overlooked British universities which are also super strong authority domains in our eyes, so getting a link from an .ac.uk domains is a very desirable thing – for anyone inside AND outside the UK market.
Now it’s been a while since and as we promised, we’ll look at other countries’ university domains and – as we found to be worthwhile – authority domains in general. Today we’ll continue our research thread with Canadian universities

just to follow up with the domain strength of German universities very soon!

Domain Ranking for SEOs and link builders

Just as with the British universities, we wanted to find out if the reputable Canadian schools that are known worldwide get the same amount of trust and reputation from Google & co.
We also were curious again if we could find some hidden gems of link building of rather unknown schools that everyone might overlook because of their (non-)size (in real world) but that would outperform other big schools in terms of trust, authority and backlink power.

A new set of tools and methods

Back in October/November 2008 we did already spend considerable time on researching good sources for lists of schools, dozens of SEO parameters for each of the 100s of domains and then weighting and comparing them.
Well, that was an easy job back then!

The goal to research Canadian and German universities was a bit tougher…

More work for .CA vs. AC.UK universities

Official University Rankings

the TIMES has a pretty popular ranking of UK schools, but now we needed a good, reputable source for Canada and Germany’s school… plus – we were eager to compare those schools to each other… just as we do SEO wise.

FOUND THAT – with ARWU and THE-QS rankings (read below for more)

A Canadian school domain list

Funny, it cost us some time just to FIND a list of Canadian and German universities – including their webpage!

FOUND THAT!

National Academic Backlink Power metric involved coding = $$$

National what?
Sounds too fancy? It’s about the links from other universities in the same country!



The national academic backlink power is measured in the quantity of backlinks from universities in the same nation. We believe that a school is stronger if there’s a lot of content and material published on it’s domain and in addition linked to from other universities.

If there wouldn’t be the TLD issue!

So this job was a piece of cake job for the UK research, because all we had to do was ask Yahoo how many .ac.uk backlinks they found for a specific domain (and we had that automated already)…

See, it works like this query "linkdomain:http://www.ox.ac.uk/ site:ac.uk" and whoppa – Oxford has 422k backlinks from other British schools

However, for Canada (and Germany) there’s nothing ready like that. This is because Canadian and German universities don’t have a separate domain extension (TLD) like .edu for the US or .ac.uk for the UK.

So if you ask simply Yahoo for backlinks to a Canadian university from other Canadian university and try it like this "linkdomain:utoronto.ca site:.ca" all you know is that there are roughly 800k backlinks from Canadian domains overall – not restricted to Canadian universities.

SOLVED THAT

we CAN count the number of backlinks from Canadian or German schools now – for ANY domain we’d like to this is a GEM! we will blog about that authority backlink counter, but just to tell you, it involved coding, and long runtimes

good thing is – we found some more nice tricks to get more links than we ever expected – high quality, high detail backlink data – beyond any 1000 link limit.

And we also have some interesting findings regarding one school’s website that has duplicate domains, duplicate content and what trust seems to be able to help nevertheless. Read about that soon

Comparable SEO rankings between .CA / .DE / .AC.UK domains

Our goal was to compare Canadian universities to German ones to the UK ones…

SOLVED THAT!

Please note, that we ALWAYS measure a universities strength based on the number of national academic backlinks, AND .edu AND .gov links, since we imply that the US .edus and .govs provide strength and trust to ANY country’s site.

This means that the results could be a bit skewed towards English language universities. This appears to be based on a sort of two-world system in the academic world. You hardly find English language papers quote German language papers. Vice versa is a little bit better tough – a little bit!

How we did the valuation

From a basis of roughly 200 Canadian universities, we scraped more than 20 different search related parameters, weighted them and let all that go into one overall ranking.
Some of the parameters we included were obviously

• other canadian universites (.ca links from special list of domains – see above)
• backlink counts from US schools (.edu links)
• governmental pages (.gov)
• domain age, measure in various way
• forward links
• yes, we also took the number of forward links of those domains into account, since every good and trusted site links out a lot naturally and is "not too greedy with it’s links".
• we also included parameters like Google Pagerank (which is pretty pointless), the overall backlinks we counted and traffic metrics of Alexa (which can be manipulated quite easily, is said to be skewed towards SEOs) or the Compete.com Rank (which didn’t even give us results for all domains, so we took it out again.

Domain age is important – but…

Again, one of the major criteria looked at was the domain inception date. The age of a domain is one of the major criteria in Google’s algorithm to check if and how much it could possibly trust a site. Usually it’s the older the better.
Once again we found that Canada cough migrated their WHOIS registry several times in the years 1995, 2000, 2001 and 2003, like so many other countries (UK for instance). this means that the domain creation date that you find to be pretty reliable for .com domains is pretty useless for the university .ca domains registered – most of them returned dates like 08/01/1995 or 1/1/1996 … the picture looks better than with the ac.uk domains, but nevertheless we uses the same method to gauge the real inception of a domain, as explained earlier in the post about .ac.uk domains.

Is the more the better?

In general we followed the idea of "more is better" when it came to link counts. This isn’t perfect – and as explained here, we have ideas how to improve that.
The date was valued as "older is better" – obviously.

And the winner is….

In the following table we compare our SEO ranking to those of the two new academic ratings ARWU and THE-QS.
==

SEO Ranking	URL	Name of University	ARWU Ranking	THE-QS Ranking	Webometrics Ranking	SEO Rating
1	http://www.utoronto.ca/	University of Toronto	19	41	24	1.6
2	http://uwaterloo.ca/	University of Waterloo	201-302	129	94	1.7
3	http://www.mcgill.ca/	McGill University	201-302	20	91	1.8
4	http://ubc.ca/	The University of British Columbia	35	34	38	1.8
5	http://yorku.ca/	York University		252	113	1.8
6	http://carleton.ca/	Carleton University	402-503	346	187	1.9
7	http://ulaval.ca/	Université Laval	201-302	268	110	1.9
8	http://queensu.ca/	Queens University	201-302	117	177	1.9
9	http://www.ualberta.ca/	University of Alberta	101-151	74	61	1.9
10	http://uvic.ca/	University of Victoria	201-302	244	171	1.9
11	http://uwo.ca/	University of Western Ontario	201-302	159	193	1.9
12	http://concordia.ca/	Concordia University		357	400	2.0
13	http://mcmaster.ca/	McMaster University	89	117	181	2.0
14	http://umanitoba.ca/	University of Manitoba	201-302	307	166	2.0
15	http://ucalgary.ca/	University of Calgary	201-302	170	69	2.0
16	http://www.sfu.ca/	Simon Fraser University	201-302	164	62	2.1
17	http://uqam.ca/	Université du Québec à Montréal		334	140	2.1
18	http://umontreal.ca/	University de Montréal	101-151	91	63	2.1
19	http://www.uottawa.ca/	Université d’Ottawa	201-302	222	228	2.2
20	http://www.mun.ca/	Memorial University			324	2.3
21	http://uoguelph.ca/	University of Guelph	201-302			2.3
22	http://usask.ca/	Universtiy of Saskatchenwan	201-302			2.3
23	http://dal.ca/	Dalhousie University	201-302	197	339	2.4
24	http://unb.ca/	University of New Brunswick			331	2.5
25	http://uwindsor.ca/	University of Windsor			627	2.6
26	http://brocku.ca/	Brock University			645	2.6
27	http://polymtl.ca/	Polytechnique Montréal			641	2.6
28	http://uwindsor.ca/	University of Windsor				2.6
29	http://uwinnipeg.ca/	The University of Winnipeg			555	2.6
30	http://upei.ca/	University of Prince Edward Island				2.6

As you can see – University of Toronto has a .edu domain as well – and we’ll talk about that soon more in detail.
Westervelt college decided to redirect it’s nice domain http://westerveltcollege.com/ to the free-hostish looking http://westervelt.x2idea.net/ … well – not sure what’s going on in their brand department… and biggest fun of all is, the root domain http://www.x2idea.net/ runs a default Debian linux start page… frightening… would you want to study there?

Oh – and it’s a 302 redirect

Even more hard work for a couple numbers

Once again we’ve been gathering and shuffling the base data that was "easy" to get. Just as for the ac.uk research we did.
However this time it got harder – we needed to implement special back link checking tools to find the actual number of universities linking to a site – with a seed list of universities to check against.

But great thing is – it forced us to find and implement some more nice tricks to get more links than we ever expected to get from public sources – high quality, high detail backlink data and way beyond any 1000 link limit!

What to do with this list

This SEO-Ranking gives you a priority and idea about allocation of time budget to plan for when hunting for links. Going after the UPEI.CA school (#30) with a general grade of 2.6 is by far less promising than going after the top one University of Toronto.

How to compare to other university domains

As promised earlier, we’ll try to make the valuation model comparable and therefore started to publish our "SEO Rating" number – which in general is a number between 1 and 5 – comparable to European school grades and a weighted mixture of detail grades for different aspects we looked at.
From today on, those SEO Ratings can be compared between countries. We will update the post about the .ac.uk universities in a few hours to hold the new official rankings from ARWU as well as THE-QS.

We think this will become even more exciting as we’ll launch the research for German universities in a few days.

Using this model for non-academic domains

What develops further is our domain group valuation model, where we’re able to look at a whole range of domains of an industry to rank them SEO-wise – be it from the financial, real estate or tech sector doesn’t matter.
We’re going to continue our research and preparations.

Bonus Tip for finding great authority links

Another great approach to use this list is to use it as input for a common forward link tool like our own private tool. This will result in a list of domains that are commonly linked from these authorities – and will result in a second shortlist of sites to target for link building.
After all it can be very tedious and expensive to get links from Canadian university sites (well, this is true for .ac.uk or .edu sites also)!

Update 05/26/2009

After updating our table with the new finding from another traditional university ranking we got finally some data for some universities that we did not find on any of the typical ranking before.

The ranking was accomplished by the Cybermetrics Lab, a research group belonging to the Consejo Superior de Investigaciones Cientificas (CSIC). It is called the Webometrics Ranking of World Universities and it is published twice a year since 2004.

Do you want this research for GERMAN domains? Let us know!

I love it when I can just dispatch urgent blocking issues like this to my inhouse dev team, to at least not have to rely on external sources to fix their tools…

• a classic post by Roger/Martinibuster about the sex of linkbuilders… "seems like getting a request from Georgina, Bunny, or Debbie has a better chance than one from Hank, Frank, or Bruce." old but very funny
• new Wordpress plugin by Gab for automatic internal link building great stuff and was just recently updated
• Patricks Quantum physics and Pagerank… sorry, this is over my head… but interesting to see what some people spend time with
• Monte still uses indians for linkbuilding (to some extend) … well, some clients were burnt in India, we were burnt in India.. not something I’d vote for, but still nice post
• bit older, but nice analysis on how mainstream media link out … I’ll put all those sites into our common forward links tool now to see which link targets to approach first
• using url search top get thousands of visitors… nice idea</font

101 Link Building Tipps by Aaron & Andy

Backlink Quality by Jim

Link Types by Todd

and then there’ comment and trackback spam again…

• It’s been almost FIVE years now, that some guys called me a trackback spammer for pinging them without linking to them … oh well, we all started somewhere
  

• Blue Hat SEO just released a plugin to do that automatically on a large scale
  

• Just in Time ShoeMoney starts to blog about how to improve his comment and trackback spam fight on his blog
  

• It appears that now in mid 2008 not much has changed in the spamming world since 2003, expect that some bloggers use NOFOLLOW and more of the links get caught in filters and algos… but it appears by just scaling up their bots, some people are still successfull to boost their spammy sites… ATTENTION: Not something I would recommend of course!

ENJOY!

According to this blog post Google allows you to add a NOFOLLOW Cass class to your webmaster console and tell google (in secret) that a block that might be labelled “white-navigation-links” are in fact sold links and that Google should ignore them.

Well, as of today I haven’t seen that “No Follow Class” option in any of the webmaster console accounts I manage for my own stuff and clients.

What about you?

Last night in the “You & A” session Matt Cutts responded to Jay Young talking about building an “Archive of pagerank data” and giggled when he confirmed that they already take care of people querying the PR data too much or too heavily.

In fact he confirmed that years ago already they started to return stupid random data based on some signature they found in the guy’s query signature. He added, that TODAY, they of course are a lot smarter in making sure people don’t overdo the pagerank queries.

For me this is a clear confirmation that they have implemented some (obviously needed) measures to

• detect bot type patterns in the PR queries – i.e. if more than 50 pages from a domain are requested within 2 minutes, it must be a bot, mustn’t it
• implement kind of “quotas” for PR queries (I had clients using Aarons SEO toolbar seeing only white or gray anymore)
• return random or white/grey PR data based on above findings

This again is a clear argument (as Adam Audette said as well in his great presentation) that PR cannot and SHALL NOT be used as a basis for decisions like

• PR sculpting / siloing (how do you know what you’re doing if you get crap data input?)
• link building – or removing built or paid links based on PR changes

Interesting sessions here btw… but more even more important and interesting private talks

Thanks to Aaron for providing this quality-document with a lot of great pointers and suggestions.

Google changed the way the return results for some queries.
Google changed the way they filter results for some queries.
Google changed the SERPs for some queries.

Seeing some clients rocket up the SERPs … good thing, eh?

Have a good weekend!

As you might know (or not know) some of the private label content clubs provide "announce blog networks" that allow you to post some content including your links to your sites for a relative low fee (or included)… and there your links are – among the neighbourhood of the other 500 members trying to power their thin adsense site with links from even thinner announce blog network links… well – as you can tell, I haven’t been impressed with those offers …

in fact you’d better NOT put your links up there as the negative effect of bad links became evident in the past 6 months… Yes, I’ve seen sites tank because of bad links!

And thanks to you own ad network detection tools that I wrote about here as well, it didn’t take long for me to spot that several of those blogs are not only available to members of the network (ie ArticleUnderground’s network) but are also member sites of Text-Link-Ads, PayPerPost and other big scale, WAY ABOVE THE RADAR ad networks that are all being cracked down … hand’s off!