Is Wikipedia More Biased Than Encyclopædia Britannica?

By identifying politically biased language in Encyclopædia Britannica and Wikipedia, Feng Zhu hopes to learn whether professional editors or open-sourced experts provide the most objective entries.
by Michael Blanding

For more than a century, the long, stately rows of Encyclopædia Britannica have been a fixture on the shelves of many an educated person's home—the smooshed-together diphthong in the first word a symbol of old-world erudition and gravitas. So it was a shock to many when, in 2012, the venerable institution announced it would no longer publish a print version of its multivolume compendium of knowledge.

Though the Britannica would still be available online, the writing on the virtual wall was clear: It had been supplanted by the Internet. And more specifically, by an upstart phenomenon Wikipedia, the free, crowd-sourced encyclopedia that since its inception in 2001 had rapidly become the new go-to source for knowledge.

“Most of the topics of content we are dealing with on a daily basis do not have a verifiable answer”

"It's sad to see the trajectory of Encyclopædia Britannica," says Feng Zhu, an assistant professor in the Technology and Operations Management unit at Harvard Business School, who details the rise and fall of the information giant in a new working paper. "There has been lots of research on the accuracy of Wikipedia, and the results are mixed—some studies show it is just as good as the experts, others show [that] Wikipedia is not accurate at all."

Complicating matters, however, many of the topics that we look up in the Britannica—any encyclopedia—aren't factually cut-and-dried. "Most of the topics of content we are dealing with on a daily basis do not have a verifiable answer," says Zhu. "They can be quite subjective or even controversial."

History, they say, is written by the victors, and can read very differently depending on who is telling the tale. Even modern-day issues such as immigration, gun control, abortion, and foreign policy are open to fervent debate depending on who is doing the opining. Over the years, Britannica has handled this uncertainty by seeking out the most distinguished experts in their fields in an attempt to provide a sober analysis on topics; while Wikipedia has urged its civilian editors to maintain what it calls a neutral point of view (NPOV).

Who Is More Objective?

But is objectivity better achieved by considering one viewpoint or thousands? Along with cowriter Shane Greenstein of Northwestern's Kellogg School of Management, Zhu asks that question in a new paper, Do Experts or Collective Intelligence Write with More Bias? Evidence from Encyclopædia Britannica and Wikipedia.

A new study examines biases between print and
online encyclopedias.©

Zhu and Greenstein have long been interested in the question of crowd bias, which itself has been hotly debated by scholars in many fields including psychology and politics over the centuries. Are two heads better than one, or do too many cooks spoil the broth? Does the collective will of the majority lead to democratic consensus or fundamentalist groupthink?

The massive, ongoing natural experiment of Wikipedia offers a unique view into these questions. "The Internet makes it so easy for people to aggregate; some scholars worry that people will self-select into groups with a similar ideology," says Zhu. As a result, the Internet may lead to more biased opinions, which only harden over time as users separate into rival virtual camps.

To test this theory, Zhu and Greenstein took a database of terms developed by University of Chicago economists Matthew Gentzkow and Jesse Shapiro to examine newspaper bias. Gentzkow and Shapiro studied speeches in the 2005 Congressional Record to scientifically identify the top 500 unique phrases used by Democrats (e.g., tax breaks, minimum wage, fuel efficiency) and Republicans (e.g., death tax, border security, war on terror), rating each according to political slant.

Zhu and Greenstein then identified some 4,000 articles that appeared in both Encyclopædia Britannica and Wikipedia, and determined how many of each of these code words were included, in an effort to determine overall bias and direction.

They found that in general, Wikipedia articles were more biased—with 73 percent of them containing code words, compared to just 34 percent in Britannica.

“We can only say [that] Wikipedia is more left. We can't say which is reflecting true reality”

In almost all cases, Wikipedia was more left-leaning than Britannica. Dividing articles into categories, the researchers found, for example, that stories on corporations were 11 percent more slanted toward Democrats, while observing similar leanings on topics such as government (9 percent), education (4 percent), immigration (4 percent), and civil rights (3 percent). Other categories did not have enough data to significantly identify bias.

Of course, those findings don't say which of the two sources is correct in its viewpoint—only how they compare to one another. "We can only say [that] Wikipedia is more left," says Zhu. "We can't say which is reflecting true reality."

What's more, much of Wikipedia's bias seems to be due to the longer article length of the online publication, where word count is less of an issue than the historically printed Britannica. When compared word to word, most (though not all) of Wikipedia's left-leaning proclivities come out in the wash. In other words, for articles of the same length, Wikipedia is as middle-of-the-road as Britannica.

"If you read 100 words of a Wikipedia article, and 100 words of a Britannica [article], you will find no significant difference in bias," says Zhu. "Longer articles are much more likely to include these code words."

Rinsing Out Bias

Perhaps the most interesting finding of Zhu and Greenstein's research is that the more times an article is revised on Wikipedia, the less bias it is likely to show—directly contradicting the theory that ideological groups might self-select over time into increasingly biased camps.

"The data suggests that people are engaging in conversation with each other online, even though they have different points of view," says Zhu. "The crowd does exhibit some wisdom, so to speak, to self-correct bias."

The number of revisions required to start showing this effect, however, is quite large—at least 2,000 edits—and the articles most read by users aren't necessarily those most revised by editors. "To some extent, we are not seeing the scenario where too many cooks spoil the broth, we are mostly seeing an insufficient number of cooks," says Zhu.

If Wikipedia would like to improve its objectivity, Zhu recommends that it encourage editors to revise the most-read stories first, as well as encouraging people with different political leanings to edit the same article.

"Wikipedia can easily do this," he says. "It has all the information about how many times people are reading and editing articles. They could easily direct the attention of editors in order to have the most impact."

Room For Both?

As for Britannica, though its experts may be somewhat vindicated by Zhu and Greenstein's findings overall, the editors are still not found to be more objective than the crowd in articles that are sufficiently revised. If the company would like to stay relevant, Zhu suggests, then perhaps it should focus on niche articles on topics not likely to be adequately covered by Wikipedia editors.

"When it comes to their capabilities, Britannica may be able to do a much better job of marketing itself as the expert on topics that Wikipedia can't cover well, such as obscure diseases where there may not be enough experts who have time to write a Wikipedia article."

Readers, meanwhile, should be conscious of the inherent bias found in Wikipedia, and seek out other sources to corroborate information on articles that lack a large number of revisions over time.

On today's virtual bookshelf, in other words, there may be a place for Wikipedia and Encyclopædia Britannica to sit side by side.

About the Author

Michael Blanding is a writer based in Brookline, Massachusetts

Post A Comment

In order to be published, comments must be on-topic and civil in tone, with no name calling or personal attacks. Your comment may be edited for clarity and length.
    • Steve Schwartz
    • Neither Side,
    I am obviously not studying this or a scholar in this topic. The information below is purely opinion with some supporting details.

    This is an extremely interesting article, I started to internally debate this during my Masters degree. Professors would continually explain that Wikipedia was not a sufficient source to utilize for accurate information. However, some of our textbooks utilized Wikipedia as a source.

    It really hit me when I met a PhD candidate at Princeton that was beginning to teach history undergrad courses. I engaged him in conversation a bit and he explained that they were working on determining the legitimacy of an aspect in history. At that moment I began to think "it's history how could it be debated". As I thought about it more I then realized that history is written by the same groups of people. "Experts" in the specific topic, however, as we all know everyone has a bias. If these experts are all cut from the same cloth "Ivy League Schools" or high profile families. Chances are they will have a bit more of a similar bias. Additionally, if you review a US history book on WWII I would imagine it will be different than a German history book on WWII (have not verified). Sure there will be known facts however, there will be a bias promoting nationalism at the minimum.

    Regardless, of the medium used it can be edited to invoke a feeling as we have seen many times used by the media. If you watch the same story on two different media channels there is a good possibility that there will be major differences in the subjectivity even possibly the video editing.

    Where the article states that Wikipedia is typically more left biased (even though it explains it may be a false positive), it would still make sense if it was. Looking at the population that utilizes the Wikipedia and the internet the demographic is going to be comprised mainly of millennials. Studies by Pew Research indicate that millennials are typically more liberal and tolerant of left side views. Additionally, there are typically multiple articles on Wikipedia for one topic with slightly different names. I would be interested to know if they took them into account in the "political slant word" counting.

    Just some opinions on the question. I generally go to Wikipedia to gain a broad understanding then move from there to more specific and professional resources if purely "factual" information is needed. Regardless, we are all subject to someone else's interpretation. Two people watching the same event will provide slightly or completely different episodes of what happened.
    • Tom Olin
    • Wikipedia Editor
    Yes. As a Wikipedia editor, from personal experience I can tell you that Wikipedia is excessively biased.
    • Susan Chipman
    • retired, none
    An interesting phenomenon I have discovered on Wikipedia is topics that have been shut down because of associated political controversies. The examples that I recall are related to education. One was information related to Howard Gardner, who is affiliated with Harvard. Another was Waldorf Schools, a type of private school pursuing a particular educational philosophy.
    • Bob Kosovsky
    • Librarian, New York Public Library
    I'll admit my bias: I've been a Wikipedia editor for 8 years.

    The problem with the procedure described is that it removes context. Though not on Wikipedia, I sometimes resort to using right-leaning phrases when I want to mock certain ideas or ideologies. How can you know that the articles in Wikipedia have not done the same on both sides of the ideological spectrum?

    I also feel that the authors research technique is faulty. The articles in Encyclopedia Britannica are original research written by experts. Since original research is not allowed on Wikipedia, each article represents a summary of existing writing. So if you find that Wikipedia leans left in some areas, that means existing writing leans left.

    A better test would be to compare articles purportedly *about the same topics* rather than the encyclopedias a whole. Since Wikipedia has been criticized for "recentism" it could be that many of the left-leaning articles are on topics that don't exist in Britannica, thus invalidating a comparison.
    • Christopher Beland
    The method used by Zhu and Greenstein does not seem particularly accurate. For example, the phrases "so-called death taxes are actually politically popular sources of revenue in many countries" and "death taxes have been heavily criticized in many countries" could be scored as having the same bias, when actually they have the opposite. To really determine bias, articles need to be read by knowledgeable people to see if they characterize facts and opinions fairly, and give due weight to significant points of view. It's also unclear that a vocabulary attempting to be halfway between American Democrats and Republicans reflects unbiased thinking; certainly politics in the UK do not have the same political center as in the US, and remember Wikipedia has a worldwide audience.

    The authors clearly haven't spent much time observing the Wikipedia community if they think it's easy for the project to direct editing efforts. Editors tend to work on articles because they are interested in the topic. The article didn't mention this, but Wikipedia has a backlog of articles that are actually tagged with neutrality problems going that's seven years long, mostly in low-profile articles:

    There are also many more aspects of bias than political left/right questions, which the Wikipedia community itself has identified and is trying to counter:

    In my experience, the most-read articles do tend to get a large amount of editorial attention, and certainly the most controversial topics are likely to get at least tagged with a dispute banner, if not ironed out to everyone's satisfaction. I'm not saying that everything always works out perfectly, and there may be a some articles that are relatively overlooked that would benefit from more awareness of readership stats, but it doesn't seem like lack of attention from random editors to high-profile articles is a big problem.

    A popularity metric could also be exactly the wrong thing to reduce systemic bias. Is it more important for us to focus on making a perfectly unbiased account of Britney Spears' career over the past year, or on adding articles with detailed coverage of non-English-speaking countries that aren't being read at all right now but might be in high demand during the next international crisis or might help promote cross-cultural understanding?
    • Adlai Englard
    • Retired
    Before the advent of the digital age, the Encyclopedia Britannica was the first source to which I turned when doing academic research, or for personal intellectual enrichment.
    I, too, lament the demise of this venerable print resource.
    One thing which I have noticed is that in its current digital version, judging only from those articles which I have read, the Britannica articles contain numerous errors of spelling and grammar, which, of course, was not the case in the print version. On this score, the Wikipedia articles on the same topics had no grammatical or spelling errors, and their content was as accurate as in the Britannica. I'm glad that I still have the Britannica on my shelves, but, at least in my reading experience, Wikipedia is the source to which I now turn first for more current information about a topic.
    • Malcolm McIntosh
    • Professor, Griffith University, Australia
    This research would have been really useful if it had not taken as its starting point speeches in the US Senate and decided that they represented left and right. Wikipedia is used throughout the English speaking world, as was the Encyclopaedia Britannia, so why not start again and use several non-US myopic filters for more useful global research outcomes?
    • Rich Farmbrough
    • Editor, Wikpedida
    Beland is correct in stating that the paper does not show a deep understanding of the volunteer community, though clearly some effort was made. (The sentence "At most there are gentle reminders or "stubs" which editors
    may attach to articles to suggest changes." shows how completely these efforts failed in some respects.)

    Nonetheless the analysis (which compares 3918 pairs of articles, contrary to the impression received by Kosovsky) is valuable.

    Firstly it is important to stress that one of the conclusions was that bias per word was less than Britannica. Thus if the Britannica articles had been longer, or the Wikipedia articles shorter, there would have been no significant difference.

    Secondly the authors concluded that bias on Wikipedia articles decreases over revisions (and hence over time). This,if true, is an important advantage over standard document production.

    The methodology uses vocabulary preferences of members of US congress. It assumes that these reflect the bias of the content, however this makes two assumptions, both of which need supporting evidence.

    The biggest flaw is the assumption that the vocabulary represents bias in content, rather than disposition. It is quite possible, indeed likely, that Wikipedia's contributors, especially to US articles are more left-leaning than right leaning. (There are certainly notable - and noble - exceptions.) Writing for Wikipedia is a very different matter than either exhorting through the US Congress, or through the columns of the daily press. It does not follow that those with a political (or other) leaning allow that to affect the content they create to the same extent that they continue to use their politically influenced idiolect.

    A second confound that could have been easily dealt with is to remove text not in Wikipedia's voice, for example, quotations and cited article titles.

    Finally, it might be worth offering my experience that the bias in Wikipedia is not usually introduced by experienced editors, where the community appears to fail is in allowing itself to be unconsciously influenced into challenging right-leaning statements more readily than left-leaning statements. On the whole, though important, this is a peripheral problem.
    • Aim
    • Drilling Supervisor, KOC
    History written collectively, will rarely have false facts (let alone deliberate fabrications). Thus the larger the sample size of inputs the more accurate (and actually unbiased) the history stays as opposed to being written by "experts" or "high profile families".

    With due respect to Wikipedia and History channel,

    • J. Payne
    • HUECU
    Myself being in the top 5,000 most active Wikipedia editors, I would have to vote for Wikipedia. :-p However, that is not to say that E.B. is not itself objective. I have seen some lively debates on Wikipedia which have delved into very accute splitting of hairs to make sure verbage, prose, and synopsis in message conveyance was accurate to an ad nausea level.
    • Peter Lor
    • Professor, University of Pretoria
    A question to the authors: did you consider only the English-language version of Wikipedia? And did you only look at issues of interest to Americans? I suggest that if you include a couple of other languages you'll find that Wikipedia offers a much more diverse and more internationally representative view of the world.
    • Douglas Scott
    • Wikipedian
    A good article and the authors raise some good concerns over bias on Wikipedia. As a long time Wikipedia editor I can confirm that bias definitely does exist on Wikipedia and my suspicion is that bias is likely worse, on average, on non-English language Wikipedia than on English Wikipedia.

    Having said that I find this article strangely ironic. It complains of bias in Wikipedia but from within a purely American context with its references to Democrats and Republicans and what they consider biased. This seems to forget that Wikipedia, although invented in America and its servers run from there, is not an American project. It is a global project with contributors and readers from all over the world. As such to analyse bias on Wikipedia from only an American point of view is in its self to promote the systemic bias within Wikipedia.

    This analysis forgets that the strongest systemic bias that likely exists on Wikipedia largely produced accidentally by the fact that the 'average' or typical Wikipedia editor is a white, highly educated, American or European, 20 to 40 something year old male with either a strong interest in or professional background in a technology related field. That's the real cause of serious bias in Wikipedia. And I say that as a guy that fits that description very well with the only exception being that I live in Africa.

    As such I would refer it to:


    However I do feel it is also true that Wikipedia would greatly benefit from having more experts in relevant article topic fields editing it. There is a consensus of sorts that that would be a good thing.
    • bowlweevils
    • insect-flavoured breakfast cereal, self-organized semi-ovoid pile
    Aside from what others have said, nearly all of which I agree with, my experience with the Wikipedia/Britannica debate is very different.

    Yes, I am a highly educated, white male, though now 41. Since Wikipedia has existed, I have found it useful. When I want to learn about something, or get a quick reference on something, Wikipedia provides it.

    But I have never in my adult life considered consulting a printed general encyclopedia. That is the kind of thing I found somewhat useful as a child, but swiftly outgrew.

    A large part of this is that I am not interested in the types of topics covered by print encyclopedias, or, if I am, I already know more about the topic than the encyclopedia knows.

    But Wikipedia is biased in a very important and useful manner: toward science and math. You will find very little about physics in a basic print encyclopedia. If you want to know about the cosmic background radiation, how it was measured, theories about why it exists, good luck with an encyclopedia.

    If you want to know the history of parts of the world that are smaller than countries in any detail, perhaps Bavaria or Brabant, you'll find it in Wikipedia.

    If you wish to know more than the very bare basics of astrophysics, European history, or even linguistics, you will find that advanced knowledge has been taken from subject matter textbooks. You will find that the information is not pre-masticated by experts who start with the assumption that the inquirer knows virtually nothing.

    The research here could only be done because topics that were present in both bodies of knowledge were selected. It was also only possible for articles containing currently common political phrases in one country by the two mainstream parties. Let us remember that the US has one of the lower voter turn-out rates in the democratic world. Many potential voters opt out because neither party represents them, or that there is too little difference between the offerings to bother.

    That itself is a partisan topic. But the voter turn-out rate is not. So question whether the code-words used by either party represent, respectively, more than 20-30% of the potential voters of that one nation.

    Most of the knowledge of the world has nothing much to do with death taxes or minimum wages. And on those things, Britannica has always been silent. But many of those subjects are very salient to people who use information technology frequently, people who grew up with it. I am 41, and was introduced to basic computer programming when I was 10. I was playing video games for several years before that.

    And I was a child with a drawer full of road maps and atlases. Wikipedia has always had the advantage when it comes to presenting non-text information, at least visually. There is a recognized concern about presenting video and audio content. I don't recall much concern about whether information about various genres of music relevant to particular communities who wouldn't buy encyclopedias being expressed in those encyclopedias.

    If you wanted to know much of anything about music, especially popular music, you weren't going to a printed encyclopedia - you knew it was biased against such things, just as much as you knew your parents or grandparents were.
    • Jean
    • KM Specialist, Canada
    There is a controversy in the law world, more specifically, if judges thoughtlessly quote from Wikipedia for definitions or descriptions in their written court case decisions. (instead from the Oxford English Dictionary, a more traditional tome where there are arbitrators on word use and etmology).

    And the controversy is: who wrote that definition or description summary in Wikipedia? It's a mistake to think that there's reliability when the information will change whom and when?

    We need stable reference / fixed point in time when citing Wikipedia for key documents.

    Wikipedia should not be the sole primary information source in such situations.

    I would agree strict analysis by purely statistical occurrence of words does not provide context nor reliable conceptual definition authority.

    See here this post on gaps in using Google Translator and the problem of using masculine vs. feminine nouns for certain actions/verbs. Reflects societal biases which doesn't reflect true reality.
    • Cheryl Branche
    • Retired medical doctor; new MLIS; writer, Self
    I remember my sadness and dismay when I learned that the Britannica printed its last edition. I still can see the proud volumes sitting on my parents shelves in the den at their home.

    I, also, remember when the professor in my reference
    class in the library and information studies program I was in, told us not to use Wikipedia for our fact checking assignment. While Wikipedia is useful for some things, it is not useful for others.

    Goodbye to the tomes of Britannica. Hello, cautiously, to Wikipedia.
    • bowlweevils
    • i am up to late, mostly harmless
    Jean, as an attorney, I share your concerns about thoughtless use of Wikipedia by judges. But I also have concerns with thoughtless use of Britannica, or the OED, or any other general source by judges.

    No general references sources should be used. There are enough specific reference sources about any topic that might arise that there is no need to rely on such things.

    Everything I said about the limits of Britannica in my previous comment is true of the OED but moreso. The OED, by its nature, is highly conservative in content. If you are concerned that Wikipedia content is too new to be trustworthy, you should be equally concerned that the OED entry on a word might be so old that it is no longer trustworthy.

    And if you are concerned about the history of Wikipedia content, you have access to it. All of it.

    Below is a link to the first 500 edits made to the article on Mount Vesuvius. You can see what changes have been made, when they did it, and who they were. The times recorded for changes are at the minute level.

    For any change, you can click the "prev" link and you will see exactly what change was made with reference to the immediately preceding content. If you click the "cur" link, you will get a side-by-side comparison of the changes made between the edit you have selected and the current version of the article.

    The OED authorities do not provide a report of every decision they make about content as soon as that decision has been made.

    Wikipedia does. Take a look. The first 500 changes date back to 14:13 21 October 2010.

    And, as others have said, Wikipedia has forums that detail the disagreements, gaps in coverage, and concerns about bias in content provision with regard to gender, age, geographical location, and so on.

    Wikipedia is the most open source of information that has existed up to today. And that openness includes its history, all of its history.

    If you don't know who did what and when, it is because you aren't looking.