Inside search engines' war on bad results

ANYONE can publish on the web, but it would be better if some people didn't; the world does not need another site that provides advice on how to unlock an iPhone or find cheap car insurance. Now new evidence shows that search engines have upped their game to make sure their results are not dominated by such low-quality sites.

Search engines are meant to pick out high-quality sites amid the sea of knock-offs, but even they get overwhelmed. As recently as March, for example, the first 10 results from a Google search for "how to organise your desktop" contained nine links to pages churned out by "content farms" - websites that publish reams of articles, often of dubious quality, that aim simply to attract clicks and advertising dollars.

That prompted New Scientist to ask computer scientistRichard McCreadie at the University of Glasgow, UK, to look into the issue. The results show that Google and Microsoft have won a major victory in the fight against such content farms. In the process they may have inflicted serious pain on two organisations often cited as providers of content farm material: Seed, a project from AOL, appears to have stopped commissioning content from freelance writers, though the firm declined to comment. And Santa Monica-based Demand Media has seen its stock price fall by over 50 per cent since it went public earlier this year for $1.5 billion.

Most of the credit for such changes has been given to Google, which this February announced that it had updated its search algorithm in a bid to prioritise sites that publish original and well-researched material. It won't provide details, but many site owners noticed that the update detected and penalised sites that publish multiple near-identical articles, a favourite tactic of content farms. For example, the traffic flowing from search engines to eHow, a Demand Media site, dropped 20 per cent after the update. Google has also enlisted users to keep up the fight - it says that it now demotes sites that people choose to exclude from search results.

To test how successful Google and Microsoft's Bing have been at fending off content-farmed results, McCreadie ran 50 search queries known to be a target of content farmers, such as "how to train for a marathon", in March and August this year. Then he paid people to examine the results for links to low-quality sites, where "low quality" was defined as uninformative sites whose primary function appears to be displaying adverts.

The results are striking. In the case of the marathon query, sites that contained lists of generic tips, such as "invest in a good pair of running shoes", were present in the top 10 in March but had disappeared by August, while high-quality sources, such as Runner's World magazine, now appear near the top. Similar trends were found throughout the 50 queries.

The story is far from over. Search engines play such a prominent role in our lives - Google alone handles over a billion queries a day - that companies constantly vie for the top places in the results. Sites can rank highly by producing authoritative material, but this is expensive, so there will always be those looking for a cheap short cut. Content farms may be out for now, but with billions of dollars hanging in the balance, it's just a matter of time before the next battle commences.

If you would like to reuse any content from New Scientist, either in print or online, please contact the syndication department first for permission. New Scientist does not own rights to photos, but there are a variety of licensing options available for use of articles and graphics we own the copyright to.

Have your say

Only subscribers may leave comments on this article. Please log in.

Only personal subscribers may leave comments on this article

Subscribe now to comment.

All comments should respect the New Scientist House Rules. If you think a particular comment breaks these rules then please use the "Report" link in that comment to report it to us.

If you are having a technical problem posting a comment, please contact technical support.

Source: http://feeds.newscientist.com/c/749/f/10897/s/1afb2cc7/l/0L0Snewscientist0N0Carticle0Cmg212284360B20A0A0Einside0Esearch0Eengines0Ewar0Eon0Ebad0Eresults0Bhtml0DDCMP0FOTC0Erss0Gnsref0Fonline0Enews/story01.htm

coco rocha coco rocha al sharpton izon heart attack grill gaddafi dead steve wynn