UNIVERSITY OF GOTHENBURG DEPARTMENT OF PSYCHOLOGY Drawing Conclusions from Politically Charged Information: A Case of Scientific Understanding or Identity Protection? Simon Karlsson Individual paper, 15 credits Bachelor thesis in Psychology PX1500 Spring term 2019 Supervisor: Karl Ask 2 Drawing Conclusions from Politically Charged Information: A Case of Scientific Understanding or Identity Protection? Simon Karlsson Abstract. Previously conducted research suggest that people may use their cognitive capacity in a biased manner when they process politically relevant information. This study let 280 US adults draw inferences based on statistics concerning two politically charged questions; gun control and climate change. To see if the expected political biases occur, the statistical content presented was manipulated to either support or oppose the “official” view of the participants’ own party. Further, the potential moderating effects of numeracy and epistemological curiosity was investigated. Contrary to predictions, no political bias could be found among participants. The potential explanations for the lack of results supporting the study’s hypotheses, as well as concrete suggestions for future research in the field, is given. Across ten political attitudes and values that has been tracked since 1994, the average gap between American republicans and democrats has increased by 21 percentage-points (Pew Research Center, 2017).1 This seems to be an indicator of what is usually referred to as political polarization, which denotes the formation of political groups with distinctive, irreconcilable policy preferences (DiMaggio, Evans & Bryson, 1996). Noteworthy is that this phenomenon does not seem to be limited to political values. People are polarized in their non-normative stands, which should not be influenced by values, as well. For example, in a survey conducted in 2016, about 79% of liberal democrats thought that climate change was mostly due to human activity whereas only 15% of conservative republicans thought that was the case (Pew Research Center, 2016). Results like this brings into question humans’ ability to handle important factual information in a sensible way. The case for human rationality – or the lack thereof – has been a recurring issue in the study of political psychology. Today, it is less a question of whether people live up to the theoretical notion of a fully rational actor who is capable of updating prior beliefs in the light of new information. Empirical studies show that people, in many situations, tend to process politically relevant information in a biased manner (e.g., Fishle, 2000; Bisgaard, 2015; Lebo & Cassino, 2007). However, there is less of a consensus concerning why people tend to deviate from the ideal type of information processing. In this study, I investigated people’s ability to draw inferences based on statistics relating to issues in the middle of party conflict. By manipulating the political implications of the statistics presented, and testing the influence of the participants’ own political affiliation, it was possible to find out if any political bias arose in their conclusions. Moreover, the potential 1 The survey, which has been conducted seven times since 1994, asks Americans whether they agree on ten specific political claims. Examples of these claims are: “Government regulation of business usually does more harm than good”, “Poor people have it easy because they can get government benefits without doing anything in return”, and “Stricter environmental laws and regulations cost too many jobs and hurt the economy”. 3 moderating effects of participants’ numeracy and epistemological curiosity were examined. Here, numeracy refers to the ability to understand, manipulate, and use numerical information, whereas epistemological curiosity refers to the inclination and willingness to seek out new knowledge. Different Takes on Political Polarization An intuitive and appealing explanation of why inaccurate perceptions over policy relevant facts occur, such as in the climate issue, is that some part of the public simply have trouble understanding and interpreting scientific information. This theoretical account has been referred to as the “Science Comprehension Thesis” (SCT) in previous research (Kahan, Peters, Dawson & Slovic, 2013). According to this notion, an important source of scientific ignorance is that people are being adversely affected by their fast and effortless thinking, as being proposed by dual-process-theories (e.g. Kahneman, 2003; Stanovich & West, 1998). Following this logic, a reasonable strategy to tackle problems with political polarization is to make people more capable of processing scientific information in a systematic way. Yet a problem with the SCT and its implications is that it seems to neglect the social and motivational aspects of human cognition. For instance, an experimental study has provided evidence that people may use their cognitive capacity in a selective manner in favor of their own political affiliation group: American participants who scored high on a numeracy test turned out to mainly use this capability to interpret political data correctly when the data presented was congenial with their own political outlook (Kahan et al., 2013). Thus, opposed to the predictions by SCT, high-numeracy republicans and democrats were actually more polarized in their conclusions – based on the same data – than their low-numeracy counterparts. This result tells us that it may be deceptive to view reasoning skills as a trait that necessarily converts into a more truthful perception, at least in questions that have an expressive function of group membership. A proposed explanation for the increased polarization among high-numeracy individuals is that cultural and political conflict disables the public’s capacity to handle scientific information, and is referred to as the “Identity-Protective Cognition Thesis” (IPC) by Kahan et al. (2013). This theoretical account highlights the fact that people have high stakes in maintaining their roles in affinity groups whose members are bound by their commitment to shared moral values. The fact that some political topics – like gun control and climate change – becomes associated with membership in such affinity groups, is therefore expected to result in what is called identity-protective cognition. With this logic, it also seems to be a natural consequence that members of these affinity groups with a high cognitive capacity use this capability selectively, to mainly draw correct inferences when the data support conclusions that is being embraced by other group members. When a stance in a societal issue becomes associated with group membership, it can even be considered more rational – from the perspective of a person’s self-interest – to form beliefs that are congruent to the group, rather than to form beliefs based on the best empirical evidence. For instance, the abstract threats of climate change or gun control may be more peripheral to an individual’s well-being than the concrete threat of being detached from a group on which a person depends. A considerable amount of research insists that social influences have a strong impact in the forming of political attitudes (e.g., Slothuus & Vreese, 2010; Druckman, Peterson & Slothuus, 2013). For example, an experimental study (Cohen, 2003) 4 manipulated if policy proposals were embraced by representatives from the participants’ own party or the opposing party. After taking note of the specific policy proposal and the representatives’ stand towards it, the participants were asked to indicate their support for the policy. Surprisingly, the effect of policy content was totally eliminated when a policy- proposal was socially defined as either liberal or conservative; liberals were even willing to support a harsh welfare program, and conservatives an expensive one, as long as representatives from their own party endorsed it. The Potential Mitigating Effects of Accuracy and Curiosity Although political biases occur, they do not seem to always occur. A study similar to the one conducted by Cohen (2003) let participants report their support for an energy law under different conditions (Bolsen, Druckman & Cook, 2013). Here, both democrats and republicans were more supportive of the energy law when it was endorsed by representatives from their own party, and less supportive when it was endorsed by representatives from the opposing party – as being proposed by previous research. Remarkable was that when participants were told that they later would have to justify the reasons for their judgement, and were encouraged to consider multiple perspectives, this effect completely disappeared. But how is it that the participants’ political bias could disappear as a consequence of a simple instruction? A well-established subfield in psychology regards motivational goals and their impact on beliefs, attitudes, decisions, and the evaluation of evidence (Kunda, 1990). A core distinction in this area of research is the one between accuracy goals and directional goals, where the former refers to the motive to arrive at an accurate conclusion – whatever it may be – whereas the latter refers to the motive to arrive at a particular conclusion. With these different motivational accounts in focus, the study by Bolsen et al. (2013) intended to give participants instructions with the purpose of inducing accuracy goals. Accordingly, there are reasons to believe that temporary manipulations of this kind may affect how people assess political information. Beyond the situational inductions of accuracy goals, research has also charted an individual trait that may mitigate the occurrence of bias in the processing of political information, which is science curiosity (Kahan, Landrum, Carpenter, Helft & Jamieson, 2017). Briefly summarized, the concept aims to reflect people’s motivation to seek out and consume scientific information for personal pleasure. Unlike cognitive capacity and numeracy – which have been shown to even aggravate political polarization among those who score high on these properties – participants with opposing political affiliations did not diverge in their opinions as their scores on science curiosity increased. This finding challenge the traditional views of traits important for the processing of political information and tell us that, in fact, some people may be better than others at resisting identity-protective cognition in their assessments. The Current Study To sum up, one of the insights given by previous research is that people’s political reasoning, and thus also their conclusions, can be influenced by attitudes among other group members. When objects are given a social meaning by members of a reference group, this may affect how individuals form their opinions and attitudes towards the same object (Cohen, 2003). Furthermore, research suggests that this tendency is not limited to 5 moral considerations, but also affects the processing of factual information (Kahan, Peters, Dawson & Slovic, 2013). With these research findings as an outset, the current study investigated people’s ability to draw conclusions, based on some simpler forms of statistics, in two politically charged topics; gun control and climate change. Since the stance in these political issues have strong connection to party affiliation in American society, these two topics constitutes a good foundation for testing the idea of identity- protective cognition. The first prediction being proposed was that the participants would base their conclusions on the desirability of the conclusion itself – as indicated by the conclusion desired by other ingroup members – without further consideration of the statistical content which was being presented. To see if this holds true, the following hypothesis was formulated: Hypothesis 1 (H1): Participants will be more likely to answer a politically charged calculation task correctly if the correct answer is in line with their own party’s official standpoint, compared to when the data is in line with the opposing party’s official standpoint. Moreover, results in previous research indicate that cognitive capacity, or numeracy, do not necessarily lead to more accurate conclusions (Kahan, Peters, Dawson & Slovic, 2013; Kahan, 2013). In fact, high-numeracy participants may mainly use their skills to get the conclusion correct if the conclusion is in favor of the standpoint promoted by their own party. In an attempt to replicate this finding, the following hypothesis was formulated: Hypothesis 2 (H2): Democratic and republican participants with high numeracy skills will use this capability in a selective manner in favor of their own party’s agenda, which – as a result – will lead to high- numeracy democrats and republicans being more polarized in their conclusions, based on the same data, than their low-numeracy counterparts. Finally, since research has indicated that the individual disposition science curiosity could mitigate the political bias that numeracy and cognitive capacity could not (Kahan et al., 2017), this study also tested the moderating effect of epistemological curiosity (Litman, 2008). Worth noting is that science curiosity and epistemological curiosity are two different measures of curiosity, but may, nevertheless, have important properties in common. If this is shown to be the case, and epistemological curiosity mitigates political bias equally as well as science curiosity, this could ease further research in the field since it relies on a simpler form of measurement. To find out if this is the case, the following prediction was made: Hypothesis 3 (H3): Democratic and republican participants with high epistemological curiosity will be less polarized in their conclusions, based on the same data, than their low-curious counterparts. These hypotheses, as well as other study-decisions, were pre-registered and can be found via the following link: https://osf.io/h7ysa/. 6 Method Participants The participants in this study were recruited using Amazon's MTurk service. Initially, 338 participants were recruited. Of the original sample, 58 persons were excluded as a consequence fulfilling one or more of the exclusion criteria which were set up in advance. The four possible reasons for the exclusion of participants were; if they completed the entire survey in less than 100 seconds, if they completed any of the tasks that made up the study’s experimental conditions in less than 10 seconds (as it is unlikely that they would have processed the materials carefully in such a short time), if they correctly guessed the purpose of the study, or if they did not live in the US. Of the remaining 280 US adults, 58.9% were male (the rest female), and the mean age was 35.34 years (SD = 10.4). The modal value of the participants’ educational level was “Bachelor degree” (50.4%). Concerning the participants’ political outlooks, 55% identified themselves as at least somewhat affiliated to the Democratic Party, 17.9% as independents, and 27.1% as at least somewhat affiliated to the Republican Party. As for their ideological position, 48.6% defined themselves somewhere on the liberal end of the scale, 26% as moderates, and 25.4% somewhere on the conservative end of the scale. Participants were paid 1.2 USD if they finished the survey successfully, which, on average, took them 9 minutes and 40 seconds to do. Materials The study’s materials were embedded in an online survey which could be answered via computer or mobile device. In this section, I will outline the content of the survey (attached in Appendix I) and describe how it was transformed and coded into the variables used in the analyses. Numeracy. Numeracy was measured using an abbreviated numeracy scale by Wellers et al. (2013). The eight-item scale measures people’s ability to understand, manipulate, and use numerical information in a way that have been shown to favorably predict both decision-making and risk judgments (e.g., “Imagine that we roll a fair, six- sided die 1,000 times. Out of 1,000 rolls, how many times do you think the die would come up as an even number?”; Wellers et al., 2013). Regarding the coding, a correct answer to a task was coded as 1, while wrong answers were coded as 0. A composite variable consisting of the participants’ accumulated score on the eight numeracy tasks was then created. All the tasks that constitutes the numeracy-scale can be found in Appendix I. Political affiliation. The participants’ political affiliation was measured using their responses to two survey questions. First, participants were asked which of the statements, presented in a 7-point Likert scale, described their political party affiliation best (1 = strong democrat, 2 = democrat, 3 = independent lean democrat, 4 = independent, 5 = independent lean republican, 6 = republican, 7 = strong republican). They were then asked which of the statements, presented in a 5-point Likert scale, was most consistent with their ideological position (1 = very liberal, 2 = liberal, 3 = moderate, 4 = conservative, 5 = very conservative). For the analysis, the participants’ responses to the question about their ideological position was converted to its equivalent scores on a 7- 7 point Likert scale. This, in turn, made it possible to create an aggregated Likert scale by averaging the responses to these two questions ( = .918). In line with the study by Kahan et al. (2013), and to facilitate the visual representation of the data, political affiliation was recoded into a binary variable for the graphs (Figure 3 & 4) shown in the Results section. For this purpose, an aggregated score higher than 4 was recoded to 1 (denoting conservative republican), and an aggregated score lower than 4 was recoded to 0 (denoting liberal democrat). The participants who could not be placed in any of these categories (i.e. answered “moderate” on the ideology- question and “independent” on the party affiliation-question) were excluded from the data that underlie the visualization (n = 43). Epistemological curiosity. The participants’ epistemological curiosity was measured using the I/D-scale by Litman (2008), consisting of two subscales. One of the subscales is referred to as the interest-type curiosity scale (I-type), and aims to measure to what degree a person associates the discovering of new ideas with a feeling of pleasure (Litman, 2008). The I-type scale consists of five statements (e.g., “I enjoy exploring new ideas” and “I enjoy learning about subjects that are unfamiliar to me”) to which the respondents are asked to rank their agreement on a 4-point Likert scale (1 = almost never, 4 = almost always). The other subscale is referred to as the deprivation-type curiosity scale (D-type), and intends to measure a person’s tendency to spend time and effort to acquire a specific answer or solution (Litman, 2008). The D-type scale is measured in the same way as the I-type scale, although it contains different statements (e.g. “I can spend hours on a single problem because I just can’t rest without knowing the answer” and “I brood for a long time in an attempt to solve some fundamental problem”). Since the study’s hypothesis is not focused on any particular form of curiosity, but instead focuses on epistemological curiosity in general, a composite variable consisting of participants’ average score on all ten statements (both I-type and D-type) was calculated. A reliability analysis indicated that these ten items, used as a unidimensional scale, measured the same psychological concept in a reliable way ( = .891). Manipulation of statistical scenarios. Each participant was asked to interpret (fictive) data concerning two politically charged questions; gun control and climate change. Further, each of these questions existed in two versions, which from now on will be referred to as a pro-republican scenario and a pro-democratic scenario. The different scenarios reflect the political implication of the data presented. Thus, a pro-republican scenario denotes a task in which the data presented is in favor of the “official” standpoint of the Republican Party, whereas a pro-democratic scenario denotes a task in which the data supports the “official” standpoint of the Democratic Party. An illustration of the statistical content in each scenario is given in Figure 1, and the exact manipulations taking place in the different scenarios are shown in Appendix I. When assigned to the task about gun control, participants were told that a city government was trying to decide whether to pass a law banning private citizens from carrying concealed handguns in public, but that government officials was not sure whether the law would be more likely to “decrease crime by reducing the number of people carrying weapons” or “increase crime by making it harder for law-abiding citizens to defend themselves from violent criminals”. In order to address this question, participants were informed, researchers had divided cities into two groups: “one consisting of cities that had recently enacted bans on concealed weapons and another that had no such bans”. The participants’ task was then to take note of the statistics compiled 8 by researcher – presented in a 2 x 2 contingency table (as shown in Appendix II) – in order to determine whether cities that enacted the ban were more likely to have an increase or decrease in crime. These instructions, and the content of the statistics presented, was adopted from the study by Kahan et al. (2013). When assigned to the task concerning climate change, on the other hand, participants were told that the US Congress was trying to decide whether to implement the Climate Change Action Plan on a national level, but that the members of congress did not know how this would affect the employment rate. As a basis for decision, participants were informed, researchers had collected data where a random sample of American companies were divided into two groups: one consisting of companies that had recently implemented the climate plan, and another where companies had not implemented the climate plan. The participants were then exposed to the statistics that showed how the implementation of the climate plan affected employment, before they were asked to decide whether implementation was more likely to lead to a decrease or increase in employment. Although this task was made up for the current study, it was designed to have the same properties as the gun control-task used by Kahan et al. (2013). The exact design of these two tasks is found in the survey attached in Appendix I. In the data analyses, no distinction was made between the task about gun-control and the task about climate change. Accordingly, a pro-republican scenario could, for example, mean that the data presented supported the republican standpoint in either the gun-control task or the climate-change task. However, since the assignment was randomized, one would not expect any systematic differences in the assignment of these tasks. Figure 1. An illustration of which conclusions, supported by the statistical content presented, belong to which scenario. Outcome measure. The outcome variable measured whether participants solved the politically charged calculation task correctly or not. Accordingly, the dependent measure was a binary variable that could either hold the value of 1 (if correct), or the value of 0 (if incorrect). Since each participant drew conclusions in two separate tasks, one in a pro-democratic scenario and one in a pro-republican scenario, there were two of these outcome variables measured per participant. 9 Procedure The participants were first introduced to the study and its terms and condition, and was then asked to provide informed consent. It was made clear that the purpose of the research was to “collect information about reasoning skills and factors that affect this ability”. After accepting the terms and conditions, participants were passed on to the numeracy tasks. When the numeracy tasks had been completed, participants were assigned to the two “politically charged” calculation tasks – one at a time, displayed on separate pages. The assignment of the tasks was conditioned in a way that made every participant, as long as he or she was classified as a republican or a democrat, exposed to one calculation task where the data supported their own party’s position, and one version where the data supported the opposing party’s position. The assignment procedure is summarized and illustrated in Figure 2. After processing the information in each task, the participants were requested to assess which conclusion the data, that was being presented, supported. They were also asked to judge, on a scale ranging from 0 to 100, to what extent the data provided support for their conclusion. After completing the calculation tasks the participants completed the I/D-scale by Litman (2008). Next, they were asked to fill in some demographic data (age, gender, education, country of residence, political party-affiliation, and ideological position). Finally, the survey ended with some follow-up questions concerning the participants’ prior knowledge about similar studies. Specifically, they were asked to report their beliefs about the purpose of the study, so that participants who were aware of the study’s hypothesis could be excluded. Figure 2. Illustration of how the participants were assigned to the calculation tasks. Note: after being randomized into condition 1 or 2, participants were assigned to both tasks connected to the given condition. However, the order in which these two tasks were presented were randomized to prevent order effects. 10 Results The first hypothesis (H1) proposed that participants should be more likely to draw a correct conclusion when the data – on which the conclusion was based – supported their own party’s official standpoint, compared to when it did not. To evaluate this claim, two hierarchical logistic regressions were conducted; one in the pro-democratic scenario, and one in pro-republican scenario. The participants’ ability to draw correct conclusions where then analyzed using their political affiliation as the main predictor, while controlling for their epistemological curiosity and numeracy (as shown in Table 1, Step 1). Table 1. Hierarchical binary logistic regressions based on answers in the pro-democratic scenario (top panel) and in the pro-republican scenario (bottom panel). Predictor B SE Wald p Exp(B) 95% CI Pro-Democratic Scenario Step 1 Political affiliation (PA) -.033 .070 .220 .639 .968 [0.843, 1.110] Numeracy (N) .143 .069 4.342 .037 1.154 [1.009, 1.320] Curiosity (C) .345 .201 2.963 .085 1.412 [0.953, 2.093] Constant -1.780 .746 5.696 .017 .169 Step 2 Political affiliation (PA) .144 .401 .130 .718 1.155 [0.527, 2.533] Numeracy (N) .254 .167 2.314 .128 1.290 [0.929, 1.790] Curiosity (C) .357 .432 .682 .409 1.429 [0.612, 3.336] PA × N -.030 .041 .540 .462 .970 [0.869, 1.051] PA × C -.004 .112 .001 .972 .996 [0.801, 1.239] Constant -2.434 1.585 2.357 .125 .088 Pro-Republican Scenario Step 1 Political affiliation (PA) -.036 .070 .263 .608 .965 [0.840, 1.345] Numeracy (N) .159 .070 5.206 .023 1.172 [1.023, 1.344] Curiosity (C) .117 .199 .347 .556 1.124 [0.761, 1.662] Constant -1.392 .743 3.506 .061 .249 Step 2 Political affiliation (PA) .848 .419 4.099 .043 2.336 [1.027, 5.311] Numeracy (N) .635 .182 12.12 .000 1.888 [1.320, 2.700] Curiosity (C) .344 .446 .596 .440 1.411 [0.589, 3.378] PA × N -.126 .044 8.425 .004 .881 [0.809, 0.960] PA × C -.067 .114 .343 .558 .935 [0.748, 1.170] Constant -4.688 1.700 7.606 .006 .009 Note. N = 280 in the pro-republican scenario. N = 278 in the pro-democratic scenario. CI = confidence interval for Exp(B). Degrees of freedom = 1 for all Wald χ2-tests. There was no significant relationship to be found between the participants’ political affiliation and their ability to draw the correct conclusion, in either of the two 11 scenarios. In the pro-democratic scenario, the variable political affiliation showed an odds ratio less than one (OR = .968, p = .639, 95% CI [.843, 1.110]), indicating that the odds of getting the answer correct decreased as participants got higher values on the variable (i.e. moved towards the conservative/republican end of the scale), but the effect was non-significant. The results in the pro-republican scenario displayed a similar trend in that the variable political affiliation had a negative impact on the odds of getting the answer correct (OR = .965, p = .608, 95% CI [.840, 1.345]), but the effect was small and non-significant also in this case. Thus, there was no support for H1. The second hypothesis (H2) anticipated that participants would use their numeracy skills in a selective manner in favor of their own party’s political agenda, which – as a result – would lead to high-numeracy participants being more polarized in their conclusions, based on the same data, than their low-numeracy counterparts. To investigate this hypothesis, two interaction terms were added in Step 2 of the hierarchical logistic regression analysis; one consisting of the interaction between political affiliation and numeracy, and one consisting of the interaction between political affiliation and epistemological curiosity (as shown in Table 1, Step 2). The interaction term between numeracy and political affiliation gave the opportunity to examine if the effect of numeracy (on participants’ ability to get the answer correct) varied depending on the participant’s political affiliation, which was proposed by the hypothesis. In the pro-democratic scenario, the interaction term between numeracy and political affiliation was not significant (OR = .970, p = .462, 95% CI [.869, 1.051]), which meant that the effect of numeracy could not be said to vary depending on participants’ political affiliation. The predicted probabilities for conservative republicans and liberal democrats getting the answer correct in the pro-democratic scenario, given each score on the numeracy test, is illustrated in Figure 3. Figure 3. Predicted probability of getting the answer correct in the pro-democratic scenario. The vertical line connected to each prediction represents a 95% confidence interval. N = 235. 12 In the pro-republican scenario, there was a statistically significant interaction between numeracy and political affiliation (p = .004), which implied that the impact of numeracy varied depending on participants’ political affiliation. To explore the relationship closer, we start off by looking at the odds ratio of the variable political affiliation, representing the predicted change in odds for every unit change in political affiliation when participants’ score on the numeracy test (and epistemological curiosity) is equal to zero. Here, we see that the odds of getting the answer correct is predicted to change by a factor of 2.336 for every unit increase in the variable political affiliation (OR = 2.336, p = .043, 95% CI [1.027, 5.311]). However, this exact prediction may lack practical value because there were no participants who actually scored zero points on these properties, but can, nevertheless, been seen as an indicator of the predicted change in odds due to political affiliation when numeracy and epistemological curiosity is low. In short, under these circumstances, we notice that the odds of getting the answer correct increases as a consequence of participants moving towards the higher values on the political affiliation-scale (i.e. moving towards the conservative/republican end of the scale). Figure 4. Predicted probability of getting the answer correct in the pro-republican scenario. The vertical line connected to each prediction represents a 95% confidence interval. N = 237. Finally, to get a comprehension of how the effect of numeracy differs depending on the participants’ political affiliation, we observe the interaction term (OR = .881, p = .004, 95% CI [.809, .960]). The odds ratio can be interpreted as the difference in odds corresponding to every unit increase in numeracy between participants who differ by one unit in their party affiliation-score. Thus, since the odds ratio is smaller than one, we draw the conclusion that when participants’ numeracy scores increased, the advantage in performance displayed by participants with higher political-affiliation values (i.e. more conservative/republican) diminished. The effect is statistically significant, and the 13 confidence interval does not overlap the value of one, indicating that the effect is non- random. To get a more intuitive understanding of how liberal democrats and conservative republicans differ, we look at the predicted probabilities of these two categories getting the answer correct given each possible score on the numeracy test, which is illustrated in Figure 4. For instance, we can observe that the predicted probability for a liberal democrat with the highest possible score on the numeracy test to get the answer correct is about 64 percent (95% CI [.514, .772]), whereas the probability for their conservative republican counterparts to get the answer correct is predicted to be about 29 percent (95% CI [.128, .447]. Although there was a statistically significant result in the pro-republican scenario, the direction of the effect was not predicted by H2, leading to a failure to support the hypothesis in both scenarios. The last hypothesis (H3) predicted that democratic and republican participants with high epistemological curiosity would be less polarized in their conclusions, based on the same data, than their low-curious counterparts. Accordingly, for the hypothesis to be strengthened, the partisan’s ability to get the answer correct must vary as a function of their epistemological curiosity. To find out if this prediction is supported by the data, we look at the interaction term between political affiliation and epistemological curiosity (as shown in Table 1, Step 2). In the pro-democratic scenario, we could observe that the interaction term between political affiliation and epistemological curiosity was not significant (OR = 996, p = .972, 95% CI [.801, 1.239]). Nor was the interaction between political affiliation and epistemological curiosity significant in the pro-republican scenario (OR = 935, p = .558, 95% CI [.748, 1.170]). Thus, no support was given for H3. A visual representation of the predicted probabilities for conservative republicans and liberal democrats getting the answer correct in the different scenarios, given various scores on the epistemological curiosity-scale, is illustrated in Appendix III (Figure 9 & 10). Discussion The purpose of the present study was to get a better understanding of how people process politically charged information. This was done by testing three main hypotheses, all inspired by previous research findings in the area of study. Starting off with the first two hypotheses, there was no support to be found for the predicted outcomes, despite this study’s attempt to mimic the set-up of previous research that suggested otherwise (Kahan et al, 2013). First of all, it was anticipated that participants would be more likely to answer a politically charged calculation task correctly if the correct answer was congenial to their own party’s official standpoint. However, there were no results supporting this claim in the analyses conducted. The lack of predicted results also emerged in the evaluation of the second hypothesis, where high- numeracy individuals were expected to be more polarized in their conclusions, based on the same data, that their low-numeracy counterparts. Even more surprisingly in this case – based on what was expected from earlier studies – was the statistically significant result in the opposite direction. In the pro-republican scenario, high-numeracy liberal democrats were significantly more likely to get the answer correct then their conservative republican counterparts. 14 Concerning the absence of predicted results in the first two hypotheses, it is important to emphasize in what ways the current study differs from previous research and, in addition, consider whether it seems reasonable that these differences underlie the diverse outcomes. Here, I will account for two differences that may have had such an impact. Primarily, the current study may differ from previous research in how, and in which order, the different parts of the survey were presented. Although not made perfectly clear in the study’s method section, it seems like the study by Kahan et al. (2013) let participants answer demographic questions (including questions about political affiliation) before they were exposed to the experimental part of the study, whereas the current study’s demographic section took place after all manipulations had taken place. This may seem like a negligible difference, but may, however, have had an impact. In support of this claim, the study by Bolsen et al. (2013) shows that the induction of accuracy goals – by encouraging participants to consider multiple perspectives and telling them that they later have to justify the reasons for their judgement – could eliminate political bias that would otherwise exist. Moreover, another study indicates that the highlighting of participants’ political identity can have an impact on their assessments in politically charged questions (Unsworth & Fielding 2014). For instance, right-wing participants – who were skeptical to humans’ contribution to climate change to begin with – became significantly more skeptical if their political identity had been highlighted before their assessment. Similar results have appeared in several other studies (e.g. Kim, Han, Duhacheck & Tormala, 2018; Hart & Nisbet, 2012; Petersen, Skov, Serritzlew & Ramsøy, 2013). With this being said, motivated reasoning or identity-protective cognition does not only seem to be a fixed characteristic within individuals, but also a behavior sensitive to cues in the environment. Accordingly, it likely matters which type of information the participants process before they are asked to assess the political information, and it cannot be ruled out that questions about political affiliation triggers responses in line with the own party agenda. It is impossible to know if effects like this influenced the results in the present study, but the possibility can, nevertheless, guide future research in the field. By being explicit when it comes to the exact procedure in which the study’s materials are presented, and even describe things such as the name of the survey available to participants and important instructions in the introduction, these potential sources of errors can be eliminated, and the possibility of replicating results could become more favorable. Another difference of importance is that the current study measured two outcomes per participant whereas the study by Kahan et al. (2013) only measured one. However, since each politically charged task was presented on separate pages, it must have been the presence of the first task that affected participants’ assessment in the latter task. Yet we would not expect any systematic differences in participants’ assessment in the different scenarios, since the order in which they were presented was randomized. What is possible, on the other hand, is that an overall decrease in political bias arose as a consequence of participants’ assessment in the second task presented to them – it could be harder to get away with processing political information in a biased manner if one already has been exposed to a similar task right before. Finally, the third hypothesis predicted that individuals with a high epistemological curiosity would be less polarized in their conclusions, based on the same data, than their counterparts with a low epistemological curiosity. Nevertheless, the results did not 15 support this prediction, which could have several potential explanations. First and foremost, given the lack of results indicating that political bias actually took place in this study, one would also expect it to be hard to find factors that counteract this effect. This means that the absence of predicted results concerning epistemological curiosity and its moderating effects on political bias could be a consequence of the lack of political bias among participants in general, rather than the lack of predictive power by the I/D-scale. Yet it is worth noting that the previous study by Kahan et al. (2017) – in which curiosity was shown to reduce political bias – did not use the same measure of curiosity as in the present study. Unlike the I/D-scale (Litman, 2008), the measure of curiosity used by Kahan et al. (2017) did not exclusively rely on self-reported assessments, but also measured actual behavior among participants. That may be a more favorable way to measure curiosity, since it may, for instance, overcome the shortcomings connected to self-assessments. Taken together, the result in the current study is not in line with previous research. Before jumping to conclusions regarding this inconsistency, however, it seems reasonable to first investigate the potential explanations for the different results. Here, results in previous research stresses the importance of taking situational factors in consideration when conducting a study about politically motivated reasoning. It seems possible that the occurrence of politically motivated reasoning is dependent on an interaction between dispositional accounts on the one hand, such as numeracy and curiosity, and situational accounts on the other hand, such as the salience of social identity or the temporary induction of motivational goals. Hence, it is important in future replication attempts to hold one of these factors constant when testing the effect of the other. 16 References Bisgaard, M. (2015). Bias will find a way: Economic perceptions, attributions of blame, and partisan-motivated reasoning during crisis. The Journal of Politics, 77(3), 849-860. http://dx.doi.org/10.1086/681591 Bolsen, T., Druckman, J. N., & Cook, F. L. (2014). The influence of partisan motivated reasoning on public opinion. Political Behavior, 36(2), 235-262. http://dx.doi.org/10.1007/s11109-013-9238-0 Cohen, G. L. (2003). Party over policy: The dominating impact of group influence on political beliefs. Journal of Personality and Social Psychology, 85(5), 808-822. http://dx.doi.org/10.1037/0022-3514.85.5.808 DiMaggio, P., Evans, J., & Bryson, B. (1996). Have Americans’ social attitudes become more polarized? American Journal of Sociology, 102(3), 690-755. http://dx.doi.org/10.1086/230995 Druckman, J., Peterson, E., & Slothuus, R. (2013). How Elite Partisan Polarization Affects Public Opinion Formation. American Political Science Review, 107(1), 57-79. 10.1017/S0003055412000500 Fischle, M. (2000). Mass response to the Lewinsky scandal: Motivated reasoning or Bayesian updating? Political Psychology, 21(1), 135-159. http://dx.doi.org/10.1111/0162-895X.00181 Hart, P. S., & Nisbet, E. C. (2012). Boomerang effects in science communication: How motivated reasoning and identity cues amplify opinion polarization about climate mitigation policies. Communication Research, 39(6), 701-723. http://dx.doi.org/10.1177/0093650211416646 Kahan, D. M. (2013). Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making, 8(4), 407-424. Retrieved from https://search- proquest-com/docview/1449308024?accountid=11162 Kahan, D. M., Landrum, A., Carpenter, K., Helft, L., & Jamieson, K. H. (2017). Science curiosity and political information processing. Political Psychology, 38, 179- 199. http://dx.doi.org/10.1111/pops.12396 Kahan, Dan & Peters, Ellen & Cantrell Dawson, Erica & Slovic, Paul. (2013). Motivated Numeracy and Enlightened Self-Government. SSRN Electronic Journal. 10.2139/ssrn.2319992 Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58(9), 697-720. http://dx.doi.org/10.1037/0003-066X.58.9.697 17 Lebo, M. J., & Cassino, D. (2007). The aggregated consequences of motivated reasoning and the dynamics of partisan presidential approval. Political Psychology, 28(6), 719-746. http://dx.doi.org/10.1111/j.1467- 9221.2007.00601.x Litman, J. A. (2008). Interest and deprivation factors of epistemic curiosity. Personality and Individual Differences, 44(7), 1585-1595. http://dx.doi.org/10.1016/j.paid.2008.01.014 Kim, C. H., Han, D., Duhachek, A., & Tormala, Z. L. (2018). Political identity, preference, and persuasion. Social Influence, 13:14, 177-191. http://dx.doi.org/10.1080/15534510.2018.1518786 Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480-498. http://dx.doi.org/10.1037/0033-2909.108.3.480 Petersen, M. B., Skov, M., Serritzlew, S., & Ramsøy, T. (2013). Motivated reasoning and political parties: Evidence for increased processing in the face of party cues. Political Behavior, 35(4), 831-854. http://dx.doi.org/10.1007/s11109-012- 9213-1 Pew Research Center. (2017, October 5). The Partisan Divide on Political Values Grows Even Wider [article]. Retrieved from: https://www.people- press.org/2017/10/05/the-partisan-divide-on-political-values-grows-even-wider/ Pew Research Center. (2016, October 4). The Politics of Climate [article]. Retrieved from: https://www.pewresearch.org/science/2016/10/04/the-politics-of-climate/ Slothuus, R., & de Vreese, C. H. (2010). Political parties, motivated reasoning, and issue framing effects. The Journal of Politics, 72(3), 630-645. http://dx.doi.org/10.1017/S002238161000006X Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought.Journal of Experimental Psychology: General, 127(2), 161-188. http://dx.doi.org/10.1037/0096-3445.127.2.161 Unsworth, K. L., & Fielding, K. S. (2014). It's political: How the salience of one's political identity changes climate change beliefs and policy support. Global Environmental Change, 27, 131-137 Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E. (2013). Development and testing of an abbreviated numeracy scale: A rasch analysis approach. Journal of Behavioral Decision Making, 26(2), 198-212. http://dx.doi.org/10.1002/bdm.1751 18 Appendix I A survey about reasoning Welcome! Purpose of research study: The purpose of this study is to collect information about reasoning skills and factors that affect this ability. Procedures: First of all we want you to try to solve some calculation tasks. When this is done, we want you to answer some questions about your character traits. Thereafter you will be asked to fill in some demographic data and, finally, answer a couple of follow-up questions about the survey. Your MTurk confirmation code will be presented on the last page of the survey. Risk/discomforts: There are no risks for participating in this study beyond those associated with normal computer use. Benefits: Although it may not directly benefit you (except your compensation), this study may benefit society by increasing our understanding of peoples reasoning skills. Voluntary participation and right to withdraw: Participation in this study is voluntary, and you can stop at any time without any penalty. To stop, click on the "Return HIT"-button, our close your browser window. Circumstances that could lead us to end your participation: We may decide to end your participation if we determine that you do not answer the questions seriously, or if you don't follow the instructions closely enough. Confidentiality: The only identifying information based on your participation in this study will be your Amazon Mechanical Turk serial number. We note that this could be linked to your public profile page, so you might consider what information you choose to share on your public profile. These serial numbers will not be shared with anyone outside the research team and will only be used by Amazon to handle financial transactions. We note again that the serial numbers will only remain with Amazon; we will code your transcriptions with a new random number for our data collection purposes. Compensation: If you satisfactorily complete the study, you will receive $1.2 per HIT to compensate you for your participation. Payments are made via Amazon's payment system. The survey is estimated to take about 10 minutes to complete. Contact information: 19 If you have any questions about this research, you may contact: Simon Karlsson at guskarsil@student.gu.se. If you have any questions about your right as a participant in this study or any concerns or complaints, please contact the University of Gothenburg. Clicking accept: By clicking on the "I accept"-button, you indicate that you are 18 years of age or older, that you voluntarily agree to participate in this study and that you understand the information in this consent form. You have not waived any legal rights you otherwise would have as a participant in a research study. o I accept o I don't accept End of Block: Introduction Start of Block: Numeracy Please answer the following questions as best you can. Don't spend too much time on each question; if you don't know the answer, make a guess or skip the question. Imagine that we roll a fair, six-sided die 1,000 times. Out of 1,000 rolls, how many times do you think the die would come up as an even number? ________________________________________________________________ In the BIG BUCKS LOTTERY, the chances of winning a $10.00 prize are 1%. What is your best guess about how many people would win a $10.00 prize if 1,000 people each buy a single ticket from BIG BUCKS? ________________________________________________________________ In the ACME PUBLISHING SWEEPSTAKES, the chance of winning a car is 1 in 1,000. What percentage of tickets to ACME PUBLISHING SWEEPSTAKES win a car? ________________________________________________________________ 20 If the chance of getting a disease is 10%, how many people would be expected to get the disease out of 1,000? ________________________________________________________________ If the chance of getting a disease is 20 out of 100, this would be the same as having a ______ % chance of getting the disease. ________________________________________________________________ A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? ________________________________________________________________ If it takes five machines 5 minutes to make five widgets, how long would it take 100 machines to make 100 widgets? ________________________________________________________________ Suppose you have a close friend who has a lump in her breast and must have a mammogram. Of 100 women like her, 10 of them actually have a malignant tumor and 90 of them do not. Of the 10 women who actually have a tumor, the mammogram indicates correctly that 9 of them have a tumor and indicates incorrectly that 1 of them does not. Of the 90 women who do not have a tumor, the mammogram indicates correctly that 81 of them do not have a tumor and indicates incorrectly that 9 of them do have a tumor. The following table summarizes the information provided. Imagine that your friend tests positive (as if she has a tumor). What is the likelihood that she actually has a tumor? _______________________________________________________________ End of Block: Numeracy 21 Start of Block: Political block 1 (gun-republican) A city government is trying to decide whether to pass a law banning private citizens from carrying concealed handguns in public. Government officials are unsure whether the law will be more likely to decrease crime by reducing the number of people carrying weapons or increase crime by making it harder for law-abiding citizens to defend themselves from violent criminals. To address this question, researchers has divided cities into two groups: one consisting of cities that had recently enacted bans on concealed weapons and another that had no such bans. Please indicate whether cities that enacted a ban on carrying concealed handguns were more likely to have a decrease or increase in crime: What conclusion does the data support? o Cities that enacted a ban on carrying concealed handguns were more likely to have a decrease in crime. o Cities that enacted a ban on carrying concealed handguns were more likely to have an increase in crime. To what extent does the data presented in the previous question provide support for your conclusion? Weak support Strong support 0 10 20 30 40 50 60 70 80 90 100 1 End of Block: Political block 1 (gun-republican) 22 Start of Block: Political block 4 (clmt-democrat) The US Congress is trying to decide whether to implement the Climate Change Action Plan on a national level. However, they do not know how this will affect the employment rate. Some Members of Congress believe that the employment rate will fall as a result of companies having to implement expensive climate-friendly actions instead of being able to pay employees, others believe that the employment rate will rise as a result of new job opportunities. To get a better understanding of which decision they should make they use data collected by researchers, where a random sample of American companies are divided into two groups: one consisting of companies that had recently implemented the Climate Change Action Plan, and another where the companies had not implemented the Climate Change Action Plan. Please indicate whether the Climate Change Action Plan were more likely to decrease or increase the employment rate: What conclusion does the data support? o Companies that enacted a Climate Change Action Plan were more likely to have a decrease in employment. o Companies that enacted a Climate Change Action Plan were more likely to have an increase in employment. To what extent does the data presented in the previous question provide support for your conclusion? Weak support Strong support 0 10 20 30 40 50 60 70 80 90 100 1 End of Block: Political block 4 (clmt-democrat) 23 Start of Block: Political block 2 (gun-democrat) A city government is trying to decide whether to pass a law banning private citizens from carrying concealed handguns in public. Government officials are unsure whether the law will be more likely to decrease crime by reducing the number of people carrying weapons or increase crime by making it harder for law-abiding citizens to defend themselves from violent criminals. To address this question, researchers has divided cities into two groups: one consisting of cities that had recently enacted bans on concealed weapons and another that had no such bans. Please indicate whether cities that enacted a ban on carrying concealed handguns were more likely to have a decrease or increase in crime: What conclusion does the data support? o Cities that enacted a ban on carrying concealed handguns were more likely to have a decrease in crime. o Cities that enacted a ban on carrying concealed handguns were more likely to have an increase in crime. To what extent does the data presented in the previous question provide support for your conclusion? Weak support Strong support 0 10 20 30 40 50 60 70 80 90 100 1 End of Block: Political block 2 (gun-democrat) 24 Start of Block: Political block 3 (clmt-republican) The US Congress is trying to decide whether to implement the Climate Change Action Plan on a national level. However, they do not know how this will affect the employment rate. Some Members of Congress believe that the employment rate will fall as a result of companies having to implement expensive climate-friendly actions instead of being able to pay employees, others believe that the employment rate will rise as a result of new job opportunities. To get a better understanding of which decision they should make they use data collected by researchers, where a random sample of American companies are divided into two groups: one consisting of companies that had recently implemented the Climate Change Action Plan, and another where the companies had not implemented the Climate Change Action Plan. Please indicate whether the Climate Change Action Plan were more likely to decrease or increase the employment rate: What conclusion does the data support? o Companies that enacted a Climate Change Action Plan were more likely to have a decrease in employment. o Companies that enacted a Climate Change Action Plan were more likely to have an increase in employment. To what extent does the data presented in the previous question provide support for your conclusion? Weak support Strong support 0 10 20 30 40 50 60 70 80 90 100 1 End of Block: Political block 3 (clmt-republican) 25 Start of Block: I/D Scale A number of statements that people use to describe themselves are given below. Read each statement and then select the appropriate response using the scale below to indicate how you generally feel. There are no right or wrong answers. Do not spend too much time on any one statement but give the answer that seems to describe how you generally feel. I enjoy exploring new ideas. o Almost Never o Sometimes o Often o Almost Always Difficult conceptual problems can keep me awake all night thinking about solutions. o Almost Never o Sometimes o Often o Almost Always I enjoy learning about subjects that are unfamiliar to me. o Almost Never o Sometimes o Often o Almost Always 26 I can spend hours on a single problem because I just can’t rest without knowing the answer. o Almost Never o Sometimes o Often o Almost Always I find it fascinating to learn new information. o Almost Never o Sometimes o Often o Almost Always I feel frustrated if I can’t figure out the solution to a problem, so I work even harder to solve it. o Almost Never o Sometimes o Often o Almost Always 27 When I learn something new, I would like to find out more about it. o Almost Never o Sometimes o Often o Almost Always I brood for a long time in an attempt to solve some fundamental problem. o Almost Never o Sometimes o Often o Almost Always I enjoy discussing abstract concepts. o Almost Never o Sometimes o Often o Almost Always 28 I work like a fiend at problems that I feel must be solved. o Almost Never o Sometimes o Often o Almost Always End of Block: I/D Scale Start of Block: Demographic data Please fill in the following demographic data. This will only take about a minute. What is your age? ________________________________________________________________ What is your gender? o Male o Female o Other 29 What is the highest degree or level of school you have completed? If currently enrolled, highest degree received. o Less than high school diploma o High school degree or equivalent (e.g. GED) o Some college, no degree o Associate degree (e.g. AA, AS) o Bachelor's degree (e.g. AA, AS) o Master's degree (e.g. MA, MS, MEd) o Professional degree (e.g. MD, DDS, DVM) o Doctorate (e.g. PhD, EdD) Do you currently live in the United States of America? o Yes o No 30 Which of the following statements describe your political party affiliation best? o Strong Democrat o Democrat o Independent Lean Democrat o Independent o Independent Lean Republican o Republican o Strong Republican Which of the following statements is most consistent with your ideological position? o Very Liberal o Liberal o Moderate o Conservative o Very Conservative End of Block: Demographic data Start of Block: Control questions Finally, we will ask you some questions regarding your prior knowledge about studies like this. Please answer these questions truthfully. 31 What do you think the purpose of this study was? ________________________________________________________________ Have you participated in a study where you were asked to solve the same kind of reasoning tasks, as in this study, before? o Yes o No If your answer was "Yes" in the previous question, please describe which reasoning tasks you recognized and how this may have affected your response. ________________________________________________________________ End of Block: Control questions Start of Block: End of survey Thank you for participating in this survey! Your MTurk confirmation code: ${e://Field/ResponseID} (Copy and paste this code into the corresponding field in the MTurk window.) Simon Karlsson University of Gothenburg Sweden End of Block: End of survey 32 Appendix II Figure 5. Statistics presented in the pro-republican scenario regarding gun control. Figure 6. Statistics presented in the pro-democratic scenario regarding climate change. Figure 7. Statistics presented in the pro-democratic scenario regarding climate change. Figure 8. Statistics presented in the pro-republican scenario regarding gun control. 33 Appendix III Figure 9. Predicted probability of getting the answer correct in the pro-democratic scenario. The vertical line connected to each prediction represents a 95% confidence interval. N = 235. Figure 10. Predicted probability of getting the answer correct in the pro-republican scenario. The vertical line connected to each prediction represents a 95% confidence interval. N = 237.