Ye ole’ Duplicate Content Penalty

Duplicate Content and the “duplicate content penalty” have been around for 4 years at least – especially since Google started weeding out gazilllions of duplicate datafeed sites in 2004 (including some of mine that were REALLY duplicate before I took countermeasures…) … e.g. Aaron Wall and Caveman wrote about that years ago already.

Talk talk talk …

Duplicate Content has been topic on panels and discussions on ALL those dozens of search marketing conferences I’ve been to the last 5 years …
And even Google finally published an explanantion of dupe content penalties to their blog end of 2006 (wow – quick response) and a year later even to their Google Webmaster Guidelines … (please note how they rewrite their guideline post in a clever way to not trip their own dupe content filters

so for dupe content, web masters have been

trained,
warned

and even been threatened (from scraper sites or malicious competitors) with duplicate content of their website.

Why?

Because once Google detects your site or page to contain duplicate content, it removes it from the results or at least unranks it 100s of positions back.

So we’re urged to create unique and propelling content for the users and the search engines. Or be doomed.

All of us? No, not all of us… I found a nice exception…

Double Standard for Dupe Content?

Based on our recent research on highly trusted domains of Canadian university domains we found at least one interested example where the trust of completely duplicate domains seems to disable what webmaster’s have been trained on for years…

The University of Toronto maintains the utoronto.ca and the toronto.edu domain.

So dig this – they run the same content on both domains, don’t redirect and yet

both rank and got nice traffic

according to Compete.com the dupe domain toronto.edu still had 30k unique visitors last December – 10% of the real site!

No sign of a duplicate content penalty.

Dupe Site Stats compared

Here we go with some of the stats for the two domains

Url	Juice	G!PR	DomKW	DFS	aFD	whoisdate	Y!BL	Y!edu	Y!ac.uk	Y!uniDE	Y!uniCA	Y!gov
http://www.utoronto.ca/	Juice/9	8	38,201	01.08.1995	01.12.1997	01.08.1995	232,000	10,100	746	489	243,734	225
http://www.toronto.edu/	Dupe/22	8	5,713	01.08.1995	01.06.1997	08.05.1986	2,910	1,310	5	16	357	10

I had to split the table as even these few columns break the layout here

 

 

Ok, lot’s of weird headings – I know – what you see is a subset of the 55 different search engine marketing related detail parameters we track historically – time to explain our abbreviations for those common ones we use here explained briefly

Juice	Juice Indicator (cemper.com)
G!PR	Google Page Rank (PR)
Domkw	Keywords driving traffic to the domain (SemRush)
DFS	date first seen using netcraft.com
aFD	First time a domain was cached by archive.org
whoisdate	Domain Creation Date
Y!BL	Yahoo Backlinks
Y!edu	Yahoo .edu domain backlinks
Y!gov	Yahoo .gov domain backlinks
Y!uniDE	Yahoo Backlinks from german universities
Y!uniCA	Yahoo Backlinks from canadian universities
G!BL	Google backlinks (attention: crippled!)
G!idx	Google indexed pages
G!cache	Google page cache date (attention: often crippled!)
Y!BLint	Yahoo internal Backlinks (from a page to its home-site)
Y!LD	Yahoo domain backlinks
Alexa	Alexa traffic rank
Cuqv	Unique Visitors per Month (Compete.com)
M!FL	MSN forward links

Interpretation

Now let’s see what we can read from these few datapoints already

• Both domains seem to exist since 1986 – altough the .ca registry was messed up in 1995. Yes, they are very old & crusty.
• Both domains maintain a PR of 8. No sign of a pagerank removal to signal a penalty
• utoronto.ca ranks for 38,201 unique keyword traffic phrases according to SemRush (an awesome organic keyword tracker!)
• toronto.edu ranks for 5,713 unique phrases according to SemRush
• There are ZERO internal links from toronto.edu going to toronto.edu … Broken? No – this just shows that they just serve the same content made for utoronto.ca user the toronto.edu domain.
• There’s a fraction of backlinks going to the toronto.edu
  • 3000 accidential links you would LOVE to have
  • 1310 accidential .edu links!
  • 357 accidential .ca uni links (not to mention those 16 german and 5 british university links!)
  • 10 accidential .gov links – at least give me those
• Our Juice Indicator tool correctly identifies the .edu as duplicate (“Dupe/22“) – at least the homepage doesn’t rank as it should for unique phrases (well yes, it’s the same stuff as on utoronto.ca which is JUICy – what do you expect)
• The number of Google backlinks reported are EQUAL for both domains. This fact again shows that the Google backlink command was crippled 5 years ago and changed it’s rules internally several times since (as to which weird numbers to show)) it appears that they somehow understand that the two sites are equal, yet let both of them rank
• The Google cache command shows something accurate for the homepage – but note that the Google cache results had been crazy for 2+ years now
  
• Also Live/MSN show 5 million links going OUT of the toronto.edu as opposed to 38.4M of the utoronto.ca – we can assume that those would count in Live.

To sum up, it’s quite interesting to see that BOTH domains rank pretty well, for thousands of phrases – despite the duplicate content.

It’s still about Age & Trust

What’s the double standard here? The duplicate domain toronto.edu simply has

• TONS of great links
• is OLD & CRUSTY
• therefore seems to be TRUSTED ENOUGH by Google to serve 100% dupe content

So again, it’s still a matter of AGE & TRUST - large sites get away with dupe content as we see. Or as Todd Malicoat said years ago already (think it was him) – big brands get away with a lot more than small webmasters!

What would you do if you were the SEO for the University of Toronto?

Redirect the .edu to the utoronto.ca with a 301?
Probably, but what if according to Patrick Altoft the anchor text passing via 301 stops anytime – they would lose rankings from those 3000 links then… I’m sure.

What’s your take?

According to this blog post Google allows you to add a NOFOLLOW Cass class to your webmaster console and tell google (in secret) that a block that might be labelled “white-navigation-links” are in fact sold links and that Google should ignore them.

Well, as of today I haven’t seen that “No Follow Class” option in any of the webmaster console accounts I manage for my own stuff and clients.

What about you?

Google Bowling via Proxy

By Marketingfan.com Internet Markting Blog

Dan calls this “Google Proxy Hacking”, but frankly, we are not hacking any of Google’s proxies – so I’m talking about Google Bowling via Proxy Sites – related to the older black hat term “Google Bowling” for buying too many / bad links for competitor sites to knock them off the serps. Yes, it IS possible to knock a competitor site off the SERPs, altought Google says= there is nothing almost nothing a competitor can do to harm you (yeah, right – the Google folks weakened this message some months ago, because the “nothing” was plain wrong – and they knew it).

If you read thru Dan’s post you might get headaches just like this guy from all those details and the partly wrong promises for a cure for it with two solutions that BOTH address only the outdated part of the problem.

So I though I have to illustrate to you what’s going on and how Google Bowling via Proxies actually looks like

The above results are returned if you search for the unique phrase

related details is the CEMPER.COM expertise that you can order

which is was only found on my company site cemper.com ... (ok – now it’s also found on this marketingfan.com
blog and on marketingfan.at as soon as we translate it)

But WTH is Proxy Dust ???

As you can see this unique phrase which should id if my page is healthy does not show my own site but “one of those PITA sites”: run by a guy called Matt Twine from the UK (if that IS his real name…)

and as you can image the url http://www.proxydust.com/index.php?q=aHR0cDovL3d3dy5jZW1wZXIuY29t has an EXACT copy of my company site’s home page there…

Did I hear Spam Report? yadda yadda – don’t bother – the Googlers don’t seem to care, because I submitted that 2 weeks ago…

But it get’s worse

Now clicking that “filter=0” to reveal all search results we see this HUGE list of pages – cemper.com coming second…. as a filtered result right after that proxy site used for google bowling…

[... pages cut out here … ]

But also we have a couple more scumbags stealing my content and trying to hijack my site…

In fact only the ProxyDust copy wins big time over CEMPER.COM because … believe it or not…

that fricking domain registered in January 2007 got a Wikipedia backlink

And my site does not.

I currently think that’s the main reason why Google chose them over my own site – which is from 2000, not heavily SEOed, but I bet a handful more trusted than this Mark “Thief” Twine’s site.

Well, it might well be that Mark has NO CLUE about what he does, but all those ads plastered around my site indicate different.

In fact it appears the whole strategy of running those proxy sites is to earn money from the ads placed on other’s content and cashing in on their work…

What we (legit webmasters) can do…

Frankly, I love Dan’s general post as an introduction to this post, because I would have hated to explain it in all length as he did. But what he points out as “solutions” are somewhat old school methods to identify bots that pretend to be Google, Yahoo or MSNbot….

Dan’s post ALSO contains the 2nd method for sending ALL visitors a “noindex, nofollow” that do NOT

1) Identify as spiders
2) Pass a “valid IP address” test

Pretty cool – I think that might work – and will test this ASAP, in addition to my own method of blocking those scumbags.

Further readings:

I discussed this with IncrediBill last week who has a great post up on identifying fake bots – but his comment is also just

PROXYDUST appears to just pass thru the user agent as-is, hard to say without seeing an actual hijacking if they do something special with Googlebot.

Anyway, they operate out of uk2net and the easiest way to make sure you’ve got all their IPs is to just block the entire data center.

inetnum: 83.170.96.0 – 83.170.111.255
netname: UK2-NET
route: 83.170.96.0/20

and then

Automating it is sometimes proxy and behavior specific, nothing I could tell you how to do in a quick post.

Some of them actually slip through the cracks for a while until they reveal themselves so it’s not 100% bulletproof.

The only way to get most of them is to simply block all hosting centers.

I actually blocked a TON of IP ranges,including those of a rogue bot called Twiceler in the last 2 weeks…

but the “noindex” hack mentioned above is the next countermeasure…

I REALLY hope I can generalize this to protect ALL my sites without having to change all of them…

And then we got some more cool posts on

10 Ways to protect your site from negative SEO” where hamlet refers to “negative SEO” for all kinds of actions a competitor could take against you … frightening …. and Never Ending SERP Hijacking where he correctly states that the REAL problem are those sites like proxydust that DO NOT pretend to be Google….

What about you?

Has YOUR site been hijacked? Do you know?

How you could know? Just follow Jim’s post to find if a page is in supplemental … but actually make sure you look at the results closely… because what you might find is that somebody is stealing your content….

You should do that for EVERY PAGE of your site – best case – if you got the right tools for it…. but it costs a lot of resources either way – by hand or by machine tool.

Let me know about YOUR hijack experiences !

(and I’m sure people should talk about this at the SES in San Jose , however I fear they won’t too much…)

Update: You could of course get around the initial problem of having too less trust in Google by getting real juicy authority links using MSN’s linkfromdomain command by effectively even letting your competitor link indirect to you ... obviously you still want to make sure you get only the juicy pages and not spend your time with dead meat.

In fact EVERY website that uses affiliate link parameters would be affected

e.g. if these 4 urls would be indexed by MSN (or mabye a bit more)

mysite.com/?affil=test1

mysite.com/?affil=test2

mysite.com/?affil=test3

mysite.com/?affil=test4

THEN msn could already remove your site mysite.com from the search results for a duplicate content penalty.

The post also gives some clues about how to get rid of the problem, but this largely depends on what’s going on with the GET parameters… e.g. some affiliate sofware does NOT allow removing these parameters (for obvious reasons of tracking!!)

It’s time for an algo-update for MSN and all webmasters to have a close look at their most valuable sites.

Preparing for the SES tomorrow, I want to share my observations and interpretations of that WWW2005 paper

Goal

obviously search engines need a good and automatic way to identifying link networks and link farms – sites that are built soley for inflating page rank or a group of webmasters that join together for a link rink or a link exchange network

Method

key assumption is that
supporters of an honest page should not be overly dependent on one another, i.e. they should be spread
across sources of different quality

This means having too many high ranking sites is bad, while having only low ranking sites is bad too – emulate natural web

as in the case of the entire Web, the PageRank distribution of an
honest set of supporters should be power law.

You want a natural linking pattern for your sites

The two key observations in detecting link farms

Portions of the Web are self-similar; an honest set of supporter pages arise by independent actions of individuals and organizations that build a structure with properties similar to the entire Web. In particular, the PageRank of the supporters (ed: the linking sites) follows a power law distribution just as the case for the entire Web.

Again, you want a natural linking pattern for your sites

Link spammers have a limited budget; when boosting the PageRank of a target page, “unimportant” structures are not replicated.

If you need “unimportant” structures for your site, then go for low-value links like PR0, PR1, PR2, links from uncached pages, links from new sites, hell – any link – just make sure you build a GOOD, NATURAL (again) mix

So how can I lower my SpamRank ?

These are my interpretations & recommendations only:

1. Get a good mix of backlinks, where the PR follows a power-principle, say a power of 3 have 3^5 = 243 PR0, 81 PR1, 27x PR2, 9 PR3, 3 PR4 and 1 PR5 … you get the idea?
2. THEREFORE: Don’t bother buying links on High PR pages like PR9,PR8 that sure will stick out of your link profile immediately
3. Look at your competitors and their backlink profiles and try to emulate that
4. Get links from within content (presell pages), not only sidebar, navigational or footer links
5. Add “useless” pages like About, Sitemap, Contact us etc to your sites
6. Add “useless” links to your sites – some nofollow links, some PR0, some uncached

Hell – and don’t count on every single link… make your site evolve naturally, the days where you bought 3 PR7 and got a PR6 next month are over for YEARS… (I still get questions from people who believe those 2003ish myths that are burried in some old abandoned webmaster forums and minds)

Two other algorithms are cited and I’m sure they are in development or production already

B. Wu and B. D. Davison. Identifying link farm pages.

and with 22 scientific citations and only via payables from Springs this here

H. Zhang, A. Goel, R. Govindan, K. Mason, and B. V. Roy. Making eigenvector-based reputation systems robust to collusion.

More to read on the topic of link farms and link spam detection is listed here

Yahoo’s Link Farm detection patent

Using Rank Propagation and Probabilistic Counting for Link-Based Spam Detection

Undue Influence: Eliminating the Impact of Link Plagiarism on Web Search Rankings

Sidenote: Barry and others already reported from last years’ Chicago show (I think it was Chicago) that Matt Cutts from Google was playing with
his notebook to point out rented links and links from link farms.

While it’s not clear that thes algos are in effect for all search queries, all websites, all countries or languages, it’s sure Google & Co have implemented MANY of them – as the pure calculation is pretty simple once you have the huge database of web site vectors (interlink data) as Google & Co have.

The Google Accessible Search adds a small twist to how search results are returned. the most relevant results are sorted based on the simplicity of their page layouts.

So today, when users search from the Accessible search "site":http://labs.google.com/accessible, they’ll receive results that are prioritized based on their usability. WOW… did you read THAT? search results based on usability… that’s cool!

They look at a number of "usability signals" by examining the HTML markup found on a web page.

It favors

• pages with few visual distractions
• pages that are likely to render well with images turned off

So this is a great new feature – and a LOT to rethink when doing search engine optimization. Takes this idea vice-versa you can think of a search result penalty if your site is designed poorly or overloaded and cluttered… way time to get some really cool web-designers, eh?

and, as you can see from the links to the Google post, it already got a lot buzz

  New! Google "Accessible Search" 

  Google Accessibility Search 

  Google Accessible Search

  google accessible search 

  Google making it even easier to search 

  A ‘New’ Google Search – Are You a Simpleton? 

  Google Accessible Search 

  Google’s accessible search

  Finding easy-to-read web content 

  Find easy-to-read web sites with Google Accessible Search 

  Google steps forward

[via Official Google Blog: Finding easy-to-read web content]: