How well can ChatGPT create user stories compared to humans?

Abstract

User stories are widely used in Agile software development yet in practice they are usually poorly written. Researchers have been conducting studies to explore ways to improve user stories and evaluate their quality, but none of them focuses on the newly released ChatGPT’s abilities to create user stories yet. INVEST grid method suggests using a 0-3 rating to evaluate user stories based on each INVEST attribute. This study seeks to identify ChatGPT’s (GPT-3.5) abilities to create user stories, its strengths and weaknesses. We conducted an experiment to gather user stories from six participants and handed out a survey (survey 1) to gain insights from their experiences. 60 user stories were created during the experiment and rated by 27 respondents in survey 2. Based on survey 1 responses, we identified that ease of use, time efficiency and providing a good starting point are some of ChatGPT’s strengths, and lack of creativity and creating incomplete user stories are some of its weaknesses. Ratings gathered from survey 2 showed that ChatGPT can generate user stories of similar quality to those created by humans while taking less time. Our findings suggest that ChatGPT has the potential to improve the user story creation process.