Understanding How Developers Use LLMGenerated Refactoring Suggestions

Abstract

Large language models (LLMs) have gained widespread popularity and have steadily improved over time, enabling software developers to use them for coding tasks. One common task is code refactoring, where the chatbot suggests changes for the developer to apply. Although current research focuses on evaluating LLM-generated refactoring suggestions, there is a limited understanding of how developers implement these suggestions in practice. To explore this, we analyze GitHub commits where developers use LLMs along with their linked ChatGPT (GPT-3.5 or GPT-4) conversations. We found that most developers use ChatGPT’s suggestions without modification, indicating their trust in the technology. This study presents methods that future research can use to explore LLM-developer interactions and evaluate the effectiveness of other LLMs in refactoring and other code-related tasks. For evaluation we have implementations in our Github repository. Index Terms—Large Language Models (LLMs), ChatGPT, Code Refactoring, GitHub, Commit, AI-Assisted Development