Are Large Language Models gatekeepers of information? A mixed method study comparing assessments of news article’s reliability by Large Language Models

Abstract

Contemporary research shows that there is an increase in usage of AI and Large Language Models (LLMs). LLMs are already widely implemented in society - in the workplace and in private, and it is not without consequences. Mis- and disinformation is today considered a major challenge as it may act as a root cause for division between groups of people. When LLMs are used as a source of knowledge and information by many people, it becomes a relevant target for research. This study focuses on assessments of news article’s reliability by different LLMs, using the lens of Gatekeeping theory combined with Institutional theory, conceptualizing LLMs as secondary gatekeepers of information. This study adopts a mixed method approach, initially conducting a qualitative content analysis to reveal which criteria LLMs report to use while assessing reliability. Secondly, a quantitative content analysis is performed to examine if reliability ratings of news articles differ between LLMs. Third, a laboratory experiment is conducted, manipulating elements of news articles to determine if and how LLMs change their assessments, acting as secondary gatekeepers of information and whether it differs amongst LLMs. Results show that there are only slight differences in which criteria are used and prioritized by LLMs, but they make different assessments analyzing the same news stories. Ideological orientation was not an element that had impact on the reliability ratings, but sources of attribution had significant impact, strengthening the idea of LLMs as technical regulators of information which is in line with previous research on this topic (Potnis et al., 2025; Simon, 2025). Ultimately, results are in line with the conceptualization of LLMs as secondary gatekeepers when manipulated cues change previous assessments in different ways.