Ain’t no party like a GPT party: assessing OpenAI’s GPT political alignment classification capabilities

This research investigates the potential limitations and biases in political party classification by OpenAI’s GPT-4o, a prominent generative AI model. With the growing influence of large language models (LLMs) in public discourse, understanding their ability to accurately interpret and reflect complex political landscapes becomes crucial. This study juxtaposes GPT-4o’s classifications of 405 political parties against the Chapel-Hill Expert Survey (CHES), assessing parties on two scales: economic left-right and social liberal-conservative. Methodologically, GPT-4o was prompted to rate each party, changing only party names and countries, to create a comparison dataset. Although GPT-4o’s classifications are overall very accurate, the findings reveal statistically significant disparities between GPT-4o’s outputs and expert classifications. Most notably, the models perform better when classifying left-leaning, Western-European, and larger political parties. The study highlights GPT-4o’s enhanced accuracy in classifying some parties more than others, underscoring the influence of training data and inherent biases in the model. These insights contribute to the emerging discourse on the limitations and biases of LLMs, particularly in the field of political science. The study not only underscores the importance of critically assessing AI-generated content in political contexts but also proposes a methodology for comparative analysis of machine and human expert outputs, paving the way for further research in evaluating and refining AI models in politically sensitive applications.