The Presence and Impact of Order Effects in Airline Pilot Competency Assessments

Abstract

This study investigated whether competency assessments of professional Airline Pilots are subject to order effects such as primacy or recency effects. 18 examiners participated in a web-based experiment where they evaluated crew performance in three short video-recordings of a flight crew performing in a simulator. The scenarios, depicting three levels of competency, were randomly presented in either an improving or deteriorating order. A final, overall evaluation was made following the three scenarios. Results indicate no recency effects, although weak statistical signs of systematic differences in gradings between presentation orders exists. A low number of participants combined with high variations stipulates a cautious conclusion that order effects are likely to have been present and may skew the appraisal of true performance.