AI Helps Detect Media Bias, Informing Voters and News Consumers

News media shapes how people understand the world around them. The stories we encounter can influence what we believe is important, how we understand political and social issues, which people and perspectives we trust and ultimately the choices we make. Understanding how news coverage is selected and framed can therefore offer insight into not just what people are reading, but how the information environment may shape public understanding and behavior, including who we vote for in the upcoming midterm general elections.

When researchers at the Computational Social Science (CSS) Lab, a joint a venture of Penn Engineering, the Annenberg School for Communication and the Wharton School, launched the Media Bias Detector in 2024, the goal was to give news consumers a more objective way to examine how different outlets cover the same events, and ultimately decide how they perceive and use information shared in the meda. The public-facing dashboard uses artificial intelligence to analyze news articles for characteristics including topic, tone, political lean, and the facts and people included in coverage.

Now, the team is taking the project a step further.

In a new paper published in Science Advances, a research team led by Duncan Watts, Stevens University Professor and Director of the CSS Lab, introduces the scientific framework behind the Media Bias Detector and releases a data set containing more than 140,000 articles published during the 2024 U.S. presidential election year.

Rather than asking whether a news outlet is simply “biased,” the framework breaks news coverage into measurable components, allowing researchers to study how editorial choices shape what readers see, and what they don’t.

“The initial launch of the Media Bias Detector was focused on the public-facing side of the project, while this paper describes the underlying scientific framework that powers the dashboard,” says Samar Haider, a Ph.D. student in Computer and Information Science (CIS) and the study’s lead researcher. “We present a rigorous evaluation of our data annotation pipeline, showcase illustrative findings from the underlying data that cannot yet be visualized on the dashboard.”

“The dashboard was pretty full-featured relative to what else is out there in the media bias space,” adds Watts. “But the data set underlying it is much richer than any dashboard can adequately capture. This paper provides just a taste of what is possible with this new class of large-scale but still very granular news data.”

The new data set is a research resource. Other scholars can use it to investigate news coverage of specific topics, time periods or publishers at scale or even combine it with other previously collected data on news consumption to study even more complex research questions. Learn more about the Media Bias Detector in the CSS explainer video.

Bias Is Not Black and White

Instead of assigning news organizations a single “bias score,” their framework measures dimensions such as political lean, tone, topics, events and which individuals are quoted. This makes it possible to compare how different publishers cover the same subject without first deciding what perfectly unbiased coverage would look like.

“The fundamental challenge with measuring bias is that, by definition, it requires one to know the ‘true’ value of the thing you’re measuring,” says Watts. “And, for most topics in the news, objective ground truth simply isn’t available.”

Instead, the researchers look for what Watts calls the signatures of bias: systematic differences in the way publishers select and frame information.

“Our overarching aim of the project is to provide the tools to help people understand how news publishers differ from each other without assigning normative labels to them,” says Watts. “By aggregating these signals, and comparing them across publishers, or even within publishers over time, our hope is that we can detect the signatures of bias without having to decide what a perfectly unbiased publisher would look like.”

Notably, this study found that a publisher’s political lean can vary considerably depending on the topic, a level of nuance not captured by a single bias score. Looking at coverage at the article and topic levels allows researchers to compare publishers covering the same story rather than treating each organization as having one fixed political position. That signature is integrated into the public-facing dashboard to better inform users on their own news consumption.

What Gets Covered and What Gets Left Out

The larger data set allows the researchers to ask questions that go beyond how an individual article is written.

“We’re now able to tackle questions such as which politicians are platformed over others in the media, which experts are brought in to contextualize and frame stories for readers, and how we can disentangle differences in selection and framing between publishers by comparing them at the individual event level,” says Haider.

One finding illustrates the potential of examining those questions systematically across hundreds of thousands of articles. During the 2024 election, news organizations devoted more than four times as much coverage to the presidential “horse race” as to all policy topics combined.

The tendency toward covering who is ahead, who is behind and what the polls say is not new. But the scale of the imbalance surprised the researchers.

“I think it’s important to see results like this because the news media often advertises themselves as informing their readers about the important issues of the day,” says Watts. “But, when it comes to politics, what we find is that across the board the news did not inform people about the issues. It was more like sports commentary.”

The finding demonstrates why selection itself matters. Before a reader encounters the framing or tone of a story, an editorial decision has already determined which story they are seeing. The Media Bias Detector and the further advancements as shown in this current paper can help people untangle the media information that will begin to pour in as we approach the midterm general elections in November, allowing them to make their own decisions on how to perceive, interact with and use the information.

A Warning on Headlines

In their work, the researchers found systematic differences between the political lean of headlines and the articles they accompany.

The headline is often the only part of a story many people encounter, particularly when news is consumed through social media. Even readers who continue to read the article encounter the rest of the story through the lens established by its headline.

“Headlines are arguably the most important part of any news story,” says Haider. “If a headline is not faithful to the actual article, it can leave readers with a very different impression from what they might take away from reading the full story.”

While the data cannot establish whether the headline or the article text is itself “biased,” it can show when the two components differ systematically even though they describe the same story.

“We can’t measure bias directly, but we can see its imprints,” says Watts. “In this case, we can’t know if it’s the headline that is biased or the main text. Potentially both are in different ways. But what we can say is that bias is present, because they are ostensibly about the exact same thing and yet they tell different stories.”

The Media Bias Detector dashboard gives users a way to explore patterns in news coverage and make their own decisions about how they use information shared in the media. The underlying dataset outlined in this paper and integrated into the dashboard gives researchers the ability to ask entirely new questions, recombine different measurements and develop analyses that the dashboard itself cannot display while improving the user experience for better informed news consumption.

Learn more about the Computational Social Science Lab’s work here. This research was developed with funding from Richard Jay Mack and the Knight Foundation (Grant ID G-2023-67859).

This story is from Penn Engineering. Read more stories from Penn Engineering here.