The 2017 German sample
Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz built persona prompts from respondents in the German Longitudinal Election Study and asked GPT-3.5 text-davinci-003 to predict each person's vote. In Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion, this case study tests whether a method explored on U.S. samples transfers to another national election.
The model's predicted vote matched the respondents' reported choice an average of 46% of the time. It overestimated Green, Left, and non-voter shares and underestimated FDP, AfD, and small-party shares; aggregate estimates for CDU/CSU and SPD showed no significant difference. A multinomial model fitted to GLES responses using the same prompt variables performed better overall, with F1 0.62. GPT also leaned heavily on party identification and captured fewer of the more complex relationships visible in the survey.
The paper's scope is one election, one survey, and one model snapshot. The authors flag untested variable ordering, ambiguity about the reference year for age, and changing model APIs as replication concerns. They specifically recommend anchoring time-sensitive prompt details and testing prompt construction; those checks would help explain the 46% agreement in this case.
Bibliography
- Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz (2024). Vox Populi, Vox AI? Using Language Models to Estimate German Public Opinion. arXiv:2407.08563v1. https://arxiv.org/abs/2407.08563v1.