Comparing Text Scaling Methods for Measuring Party Cohesion - A case study of Swedish parliamentary debates

Abstract

Only a few previous studies have utilised text scaling algorithms in the context of party cohesion. These efforts have lacked robust evaluation strategies and comparisons between different methods in terms of their effectiveness of measuring differences in political rhetoric. This thesis compares two computational text scaling algorithms, Wordfish and SemScale, with the aim to analyse their effectiveness as methods for measuring party cohesion, and provides an evaluation framework to facilitate this comparison. The aim is to distinguish whether text scaling algorithms provide a reliable technique for studying party cohesion through rhetorical cohesion in political texts. Additionally, this thesis explores the differences of two distinct algorithms when they are used to measure rhetorical cohesion within political parties. The data used to conduct this comparison are Swedish parliamentary debates from the term 2018-2022. While Wordfish is a rather simple statistical algorithm based on word frequencies and has been commonly used for political text scaling before, SemScale is a newer algorithm, which has been claimed to outperform Wordfish as it incorporates semantics through pretrained word embeddings into its process. The evaluation framework used to assess and compare the algorithms’ performance in measuring rhetorical cohesion consists of four parts. Parametric reasonability measures how reasonable the patterns found in the word-level parameters of each algorithm are in terms of political rhetoric, whereas cohesion similarity helps to uncover the similarities and differences of the cohesion results produced by the algorithms. Rhetorical validity evaluates the cohesion results from the standpoint of political rhetoric, while hypothesis testing is used to compare the results to preexisting theory and studies on party cohesion in the context of the case study. The results of this thesis show that while Wordfish is found to have a more transparent scaling process and its results are therefore easier to evaluate, the word embeddings help SemScale produce more consistently reliable results on both parameter-level and in terms of the validity of the measured cohesion levels. Further, the results highlight the importance of considering the quality of the dataset when choosing the appropriate text scaling method.