ClariTest: An LLM-Based Tool for Context-Aware Test Case Readability Improvement - Utilizing Prompt Structure and RAG-Augmented Large Language Models to Improve the Readability of Generated Test Suites While Preserving Test Semantics

Abstract

Automated software test generation tools are highly effective at achieving high code coverage. However, these tests usually have poor readability, generic identifiers and unclear logic. This lack of readability hinders developers from properly understanding the code, reducing developers trust and increasing long-term maintainability costs. Large Language Models (LLMs) offer a promising solution to automatically refactor and document code, but their usage is often hindered by “semantic drift”. Semantic drift refers to instance where the model hallucinates and unintentionally alters the tests intended behavior. To bridge this gap between machine generated code and human comprehension, this study introduces ClariTest, an automated refactoring tool that leverages Retrieval Augmented Generation (RAG) and prompting to enhance readability of Python test suites without altering their intended behavior. ClariTest was evaluated using a subset of Pynguin-generated test cases. This study evaluates four LLMs, specifically, GPT-5-Mini, Claude Sonnet 4.6, Deepseek-Coder-33b-Instruct and Deepseek-V3.2, and two prompting strategies, Base prompt including the Direct-Instruction prompting technique and a Combined Prompt composed of Direct Instruction, Chain-of-Thought and Self-Critique. Additionally, the study investigated the impact of model temperature (by comparing 1.0 and 0.2) to evaluate the trade off between allowing higher creativity and enforcing a more deterministic behavior. To evaluate the performance of these con figurations, this study employed a systematic quantitative assessment by measuring the change in semantic drift indicators (code coverage, test passing/failing rates and total tests) and code naturalness via the Dependency-Aware Naturalness (DAN) metric. Additionally, a qualitative survey of software developers was conducted to assess the perceived readability of the refactored test suites. Regarding the temperature evaluation, the results revealed a trade off, while a lower temperature (0.2) generally saw enhanced improvements to code naturalness compared to temperature 1.0, however, it also led to decreased stability across most models, illustrated by an increasing standard deviation in semantic drift metrics. In comparison to this, a higher temperature (1.0) avoided this decrease in stability, resulting in more consistent semantic preservation across the test suites. The quantitative results identified GPT-5-Mini with the Base Prompt (Direct-Instruction) as the most optimal configuration, as this configuration successfully maximized code naturalness while minimizing semantic drift. The qualitative survey illustrated that readability improved according to 96.88% of the participants. Furthermore, the data from the survey highlighted that the introduction of meaningful identifiers and explanatory inline comments were the most critical enhancements. Ultimately, the study demonstrates that when properly constrained by RAG and prompting, LLMs can reliably transform machine generated tests into maintainable, human readable code.