Evaluating and Improving Item Response Theory Models for Cross-National Expert Surveys

Abstract

The data produced by the Varieties of Democracy (V-Dem) project contains ordinal ratings of a multitude of country-level indicators across space and time, with multiple experts providing judgments for each country-year observation. We use an ordinal item response theory (O-IRT) model to aggregate multiple experts' ratings. The V-Dem data provide a challenging domain for such models because they exhibit little cross-national bridging. That is, few coders provide ratings for multiple countries, making it difficult to calibrate the scales of estimates cross-nationally. In this paper, we provide a systematic analysis of the issue of bridging. We first use simulations to explore how much bridging one needs to achieve scale identification when coders' thresholds vary across countries and when the latent traits of some countries lack variation. We then examine how posterior predictive checks can be used to check cases of extent of scale non-comparability. Finally, we develop and evaluate search algorithms designed to select bridges that are most likely allow one to correct scale incompatibility problems.