Researchers: Using Videos in Web-Based Surveys Can Introduce Bias

Dina Shapiro Luft and Joe Cappella at Annenberg School for Communication ASC UPenn
Research Area: 

Researchers from the University of Pennsylvania’s Annenberg School for Communication say that using videos in online research studies may be easier than ever, but the use of video should be approached with caution. In fact, the Annenberg researchers argue, videos should be used only when they are integral to the study’s purposes.

In an article, “Video Content in Web Surveys: Effects on Selection Bias and Validity,” appearing in the Winter 2013 issue of Public Opinion Quarterly by Annenberg doctoral candidate Dina Shapiro-Luft and Professor Joseph N. Cappella, Annenberg’s Gerald R. Miller Professor of Communication re-examined the results of two previous web-based experiments that asked smokers to evaluate the persuasive qualities of antismoking public service announcements (PSAs). They sought to determine whether the ability to view videos differs between demographic groups, whether respondents who do not view videos for their full duration differ from those who view videos for an appropriate amount of time, and whether video characteristics influenced this relationship.

“Videos may pose problems for web-based research for several reasons,” the study’s authors write. “First, individuals who cannot view videos cannot participate fully in the study. Second, it requires that participants expose themselves to the material for an adequate amount of time.”

If surveys are completed without watching a video or only partially watching, they say, the study sample can become biased, thus threatening the validity of experiment.

Shapiro-Luft and Cappella investigated two primary factors thought to influence a respondent’s ability to view videos: technological restrictions and demographics. They note that 35 percent of adults in the U.S. do not have home broadband connections, which can affect their ability to view online videos. In other cases, the configuration of a respondent’s computer – especially in workplaces that restrict online activity -- may affect their ability to successfully watch an online video. They therefore predicted that respondents with slow internet connections and those who work full-time for private employers would be more likely to report video failure (i.e., not be able to download and view the video).

Based on previous research, they further predicted that older respondents, females, individuals with less education, those with lower income, and racial/ethnic minorities would also be more likely to report video failure.

The authors also investigated video viewing time as a function of the participant’s interest in the video topic and as a function demographics, predicting that participants for whom the video topic was less congenial and salient would be less likely to watch videos in their entirety and that younger individuals, males, employed individuals and racial/ethnic minorities would be less likely to watch videos in their entirety.

Finally, they looked at how the video content itself might impact video viewing times, predicting that those videos judged on average as less persuasive would have shorter viewing times.

Shapiro-Luft and Cappella found that almost all survey breakoff in the anti-smoking studies occurred due to technical issues with viewing the videos. Individuals without broadband connections were 44 percent more likely to report video failure, and privately employed individuals were more likely to report video failure.

Age, education, and gender were also found to play a role in reports of video failure: each additional year of education was associated with a 10 percent reduction in the probability of reporting video failure; females were 61 percent more likely to report video failure compared with males and older participants were less likely than younger participants to report video failure.

“The majority of video failures occur at the first video shown,” the authors write. “This supports the observation that video viewing ability is driven by respondent characteristics rather than by features of the videos themselves. As long as portions of the population access the internet via connections with slow data speeds, obtaining a representative sample for web-based surveys with video content will pose a challenge.”

The authors also found that when it comes to viewing the videos for their full duration, those respondents for whom the videos were more salient were less likely to skip through them. In fact, for each step along the Ladder of Contemplation toward readiness to quit smoking, participants are 31 percent less likely to skip through the videos. They also found that with each year of age, participants were 19 percent less likely to watch videos for less than their entire duration.

“Our findings show that video viewing ability is not random, but is systematically related to respondent characteristics,” write Shapiro-Luft and Cappella. “Inclusion of videos biases the sample by reducing it to types of respondents able to accommodate such content. In our study, these types of respondents were older and more educated, and more likely to be male and self-employed.”

Based on the study findings, Shapiro-Luft and Cappella recommend that web surveys be conducted using only the most widely available technology and that participants should not be allowed to move on to subsequent pages in a survey until a video has completed playing.

“The inclusion of videos is a complex decision that may lead to undesirable effects on the survey sample and results,” they conclude. “Videos should be used sparingly in web-based studies and researchers should be aware than in some cases, using videos may decrease the quality of data.”

Media Contact: Julie Sloane,, 215-746-1798