Chatbots reduce health-related conspiracy beliefs not because of but despite being perceived as AI

Beliefs in conspiracy theories and their resistance to correction pose critical challenges to public health. A potential countermeasure to this is the use of Large Language Model (LLM) driven debates about conspiracy theories. Yet while this approach yields promising results for general conspiracies, its generalizability to the health domain as well as the underlying mechanism for its effectiveness remain unclear. This study investigates whether LLM-driven debates can reduce health-related conspiracy theories in the context of COVID-19. Furthermore, it examines attribution to artificial rather than human sources as a potential explanation. In an online experiment, 554 participants were randomly assigned to either a control condition or to debate an individual COVID-19 conspiracy theory with an AI-labelled LLM or a human-labelled LLM. Compared to the control, participants aware of their artificial conversation partner reported 7.88% points less confidence in the conspiracy after the intervention (d = 0.63, p < .001). Contrary to our expectation, however, this effect was stronger for those assuming a human conversation partner: They indicated a 13.76% points larger confidence drop compared to the control (d = 0.87, p < .001). This difference in effectiveness appears to be mediated by the perceived neutrality of the source.