As I wrote in Trustrank algorithm – human reviewers confirmed Google employs manual reviewers for quality assurance of their results and possibly create human signals of quality in their system.

GoogleGuy commented and clarifyed on this:

The system that was up at eval.google.com was a console to evaluate quality passively, not to tweak our results actively. But when Henk van Ess submitted his own blog to Slashdot, he asserted "Real people, from all over the world, are paid to finetune the index of Google," and that made it sound like people were reaching in via this console to tweak results directly, which just isn't true at all.

I have serious reservations about Henk van Ess taking information from one of his own students (who presumably signed a non-disclosure agreement when the student agreed to help rate the quality of our results) and posting that information online. I also believe these web pages said things like "Google Proprietary and Confidential," but it appears that the screenshots have been cropped to exclude that information. Those are the two things that really made me sad, not the "breaking news" the Google evaluates its own results quality. It shouldn't be a surprise that Google evaluates the quality of its results in lots of ways–the fact is that every major search engine evaluates its relevance in many ways.

Andrew from SEM2 put it another way:

But surely, search quality evaluators (human) are akin to a normal part of any software testing/debugging/evaluation process in a modern engineering company. That's not a "discovery," or "news," as far as I can see.

And I must agree with him… it's just been too long that folks have been telling us or letting us percept that Google is a 100% machine run search engine without humans in there – except for manual removals. And that's not what we have seen in these algo A/B tests…