Presenting ENBYS The Europarl Non-BinarY Sentences: an English-Polish-French Dataset for Machine Translation Evaluation of Non-Binary Gender Inclusion

Abstract

Investigations of gender bias within natural language processing (NLP), and more specifically machine translation (MT), tend to adopt a binary view of gender, upholding a masculine/feminine dichotomy. The present thesis aims to widen that research area by shedding light on how machine translation handles non-binary genders and the nonstandard linguistic forms that express them. We focus here on singular forms in three languages that each have a different grammatical system regarding gender: English, in which grammatical gender is almost nonexistent; French, which has a binary masculine/feminine grammatical gender; and Polish, which, in the singular, has three main grammatical genders: masculine (with further subdivisions), feminine and neuter. In all three languages, non-binary communities have developed different strategies to express themselves in ways that reflect their identities, through the use of existing forms (such as using third-person plural pronouns, neuter pronouns, neuter agreements), the practice of splitting (including both masculine and feminine word endings separated by typographical symbols), or the creation of completely new words forms. We thus present the Europarl Non-BinarY Sentences (ENBYS), an English-Polish- French dataset of parallel sentences varying on the gender of a main actor within the sentence, on which we evaluate three widely used machine translation tools (DeepL Translator, Google Translate and Microsoft Translator) on their ability to recognize and generate correctly gendered forms between each language pair (EN ↔ FR, EN ↔ PL, FR ↔ PL). We find that while the translation models transfer global semantic information adequately, they struggle to generate non-binary forms when expected, and instead revert to binary forms.