Testing AI Against Public Health’s Existing Tools

A new study by Penn researchers investigated whether conversations with AI-powered chatbots can change vaccine-hesitant parents intentions to vaccinate their children.

By Ian Scheffler, Penn Engineering

on a table is a vial of an HPV vaccine, a needle, and a cellphone. The cellphone screen shows a chat with a LLM about the HPV vaccine.

A new Penn-led randomized controlled trial has found that AI-powered chatbots can make vaccine-hesitant parents more likely to say they will immunize their children against human papillomavirus (HPV), but no more than standard written public health materials.

The findings raise questions about when, how and to what extent AI enhances public health communications. “Comparing a chatbot to nothing isn’t really a fair test. The interesting question is whether it does better than what public health agencies already have out there. In our study, it didn’t,” says senior author Sharath Chandra Guntuku, Research Associate Professor in Computer and Information Science (CIS) at Penn and secondary faculty at the Annenberg School for Communication.

Described in a new paper in JAMA Network Open that was co-authored by Gerald R. Miller Professor Emeritus of Communication Joseph Cappella, the trial — which included nearly 1,300 participants in the United States, the United Kingdom and Canada — found that skeptical parents who interacted with the chatbots were more likely than those who received no intervention to say they intended to immunize their children.

But spending a few minutes reading standard written materials provided online about the benefits of the HPV vaccine from governmental health agencies like the Centers for Disease Control and Prevention (CDC) produced essentially the same effect.

“It would have been easy to compare an AI chatbot with no intervention or a very weak control condition and find a positive result,” says Neil Sehgal, a doctoral student in CIS and the study’s first author. “But we wanted to know whether the chatbot added value beyond what public health agencies already provide.”

Three men stand side by side smiling at the camera. The one in the middle wears a pink shirt. — From left: co-author Lyle Ungar, first author Neil Sehgal and senior author Sharath Chandra Guntuku, in Amy Gutmann Hall, Penn’s hub for AI research. (Credit: Sylvia Zhang)

The Promise and Challenge of AI for Public Health

As chatbots have become more powerful and widely used, researchers around the world have begun to explore the extent to which AI can change people’s minds, a question made all the more urgent by the rising tide of vaccine hesitancy.

Two years ago, one study found that an AI-powered chatbot could reduce beliefs in conspiracy theories. Last year, a randomized controlled trial in China found that giving parents access to a chatbot improved the odds that they either vaccinated their daughters against HPV or scheduled an appointment to do so.

But the designs of those studies and others like them often make it difficult to assess the effectiveness of the chatbots. In the Chinese study, for instance, two weeks of access to the chatbot was compared with no intervention, making it unclear if other persuasive content, like standard public health materials, would have had a similar effect.

“What is new here is the comparison against a strong, realistic control,” says Alison M. Buttenheim, Professor of Nursing and Health Policy in Family and Community Health and a co-author of the study.

A chatbot interface shows text and an entry box for text. — A sample of the kind of interaction the chatbot had with participants in the randomized controlled trial. (Credit: Neil Sehgal)

Testing AI Against a Stronger Standard

In contrast, the Penn team set strict guidelines to make the chatbot as comparable as possible with existing public health materials, and to assess its effects over time: participants assigned to the chatbot or written materials groups were exposed to each for the same minimum time of 3 minutes, and all participants were assessed for their intention to vaccinate their children 15 and 45 days after the interventions.

Under these stricter conditions, the innovative features of the chatbots — including their ability to converse with parents in real time — essentially provided no additional benefit, even though parents spent more time engaging with the chatbots than the written materials.

“While chatbot conversations can move intentions immediately,” says Lyle Ungar, Professor in CIS and a co-author of the study, “their advantage disappeared when compared with well-designed public health materials and did not persist over time.”

Indeed, at 45 days, participants who had been assigned to read the public health materials expressed a higher intent to vaccinate their children than either those who interacted with the chatbots or received no intervention.

“AI chatbots are promising, but they should not be assumed to outperform existing tools simply because they are newer or more interactive,” adds Guntuku. “A short read of a CDC webpage held up at least as well as a chatbot conversation, and the effect actually lasted longer.”

This study was conducted at the University of Pennsylvania’s School of Engineering and Applied Science (Penn Engineering), School of Nursing (Penn Nursing), Leonard Davis Institute of Health Economics (LDI), Perelman School of Medicine (PSOM), Center for Health Incentives and Behavioral Economics and the Annenberg School for Communication (ASC), and was supported by the Penn Medical Communication Research Institute, Penn Global Research and Engagement Fund and the National Institutes of Health (NIH-NIMHD:R01MD018340, NIH-NIMH:R01MH132401 and NIH-NCI:R37CA259210).