At the recent Search Central Live NYC, Google’s John Mueller discussed the third-party quality raters they use to evaluate changes to Google’s search algorithms. Although it wasn’t stated explicitly, the nuance was implied: keeping a human in the loop remains an important factor in fine-tuning your SEO decisions.
Third Party Quality Raters
Hopefully by now everyone knows that Google employs third-party quality raters to review algorithm changes and provide feedback that can be used to judge the success of various algorithm updates and tests. They don’t actually affect the rankings of individual websites, their judgment are about the effectiveness of the algorithms which themselves affect hundreds of thousands and millions of sites across the Internet.
Ordinarily such judgment calls of whether a site is useful or not are highly subjective (a matter of personal opinion). That’s why Google created a set of guidelines for the quality raters to use so as to standardize the criteria the raters use and make their judgments more objective (like considering facts that are either true or false).
Here is, according to my notes, how John Mueller explained it:
“And one of the ways that we do that is we work together with so-called quality raters. These are external people who review the quality of search results, who review the quality of web pages to let us know, are we in a good place? Are we going in the right direction? Are the changes that we are working on actually making sense and acceptable for you?”
What’s notable about that exchange is that the whole point of judging the algorithms is whether or not they are acceptable to humans.
Mueller next introduced the topic of the quality raters guidelines and how it’s important for SEOs and publishers to read. In fact, he calls it important and encourages anyone concerned about ranking better to at least give it a scan for topics that may be important to the individual.
He continued:
“So we have a set of guidelines that we published for these quality raters, which I think is actually surprisingly important. It’s a gigantic book, something I don’t know, 180 pages long. But it’s a lot of guidelines where we kind of draw out what we think makes sense for quality raters to review with regards to the content. And this is publicly available. You can look at it yourself as well.
I think for most websites it makes sense at least have gone through it, or maybe control F and search through it for keywords that you care about just so that you have a sense of what Google is thinking when they’re making changes.”
The Quality Raters Guidelines Is Not A Handbook Of Ranking Factors
Three are many SEOs who have spread the misinformation that the quality raters guidelines offers a peek into what Google is using for ranking websites. That’s false.
Mueller continues (my paraphrase):
“Obviously quality rater guidelines is not a document that says like, this is how we do ranking, but more just, this is how we review things on the web when we ask for input from these quality raters.
They do a number of different tasks for us and so one of them is page quality where they tell us like, is this a high quality page or not? Another one is to evaluate whether the pages that we show in the search results meet the needs of a user. Which is highly subjective sometimes, but we give them information on what they can do there and the other one is A/B testing, side-by-side testing where we present quality raters with a set of pages before and a set of pages afterwards, and they tell us which one of these is actually better.”
Humans In The Loop
The important takeaway from Mueller’s discussion about the quality raters and the guidelines they use is that how humans react to the search results is at the heart of what Google is doing with their algorithms. Some people tend to think of Google’s algorithms as mechanical machines that are cranking out search results and that’s pretty much what they are but they’re also emulating human judgment about what is and is not spam, what is and is not a high quality search result.
Rote SEO is highly focused on feeding the machine but the machine itself is emulating human judgment. SEO today is more than ever about considering how every choice made about a site affects humans and less about worrying about whether you’ve got enough entity keywords on a page and if the H1 heading is missing.
Human Judgment Is Core to Google’s Algorithm Development
Quality raters are used to judge whether algorithm changes make search results better for people. Algorithms are adjusted based on human reactions, not by machine metrics.
Quality Raters Guidelines Reflect Google’s Values
Google’s Quality Rater Guidelines are not a ranking manual. They define what Google considers useful, high-quality content. They can serve as a mirror that business owners and SEOs can hold up to their own content to see how it aligns with Google’s criteria for high quality.
SEO Today Is About Human Experience
The deeper message buried in what Mueller was talking about is that Google’s algorithms are trying to emulate human judgment, so SEOs should focus on user experience and usefulness, not checklists or busy-work like adding author bios with superfluous information that does nothing for site visitors.