EMERGENCE OF REFERRING EXPRESSIONS THROUGH LANGUAGE GAMES

Abstract

There has been a recent focus on how neural agents in language games ground referring expressions in visual 3D-scenes. This thesis explores when referring expressions emerge and if they align with referring expression found in natural languages like English. For this, multiple new artificial datasets based on the CLEVR dataset are generated to control precisely for the bias included in the visual scenes, namely the attributes of the target object and distractors. The datasets and their controlled biases are validated in a series of reference expression generation and comprehension tasks. A sender and a receiver are playing language games in which they need communicate referring expressions to solve the same tasks. For many tasks, they are able to successfully ground referring expressions in their own emerged language. An analysis of the emerged languages shows that the emerged referring expressions are aligned very few with natural language referring expressions. However, they share certain features like an incremental approach in which some attributes are consistently used more often than others