Automated Prompt Optimization for LLM-based Test Update Localization

Abstract

Maintaining consistency between production code and its associated test cases is a critical, yet costly, task in software development. As systems evolve, identifying which tests require updates becomes increasingly complex. Recently, Large Language Models (LLMs) have shown great potential for automatically localizing test cases. Yet the performance of existing LLM-based automated approaches is strongly affected by how prompts are formulated. While prior work has shown that prompt engineering can significantly influence LLM behavior, many existing approaches rely on static, manually crafted prompts that do not adapt to varying contexts. Our thesis proposes an automated prompt optimization approach, named Adaptive Reasoning-guided Gradient-based Optimization (ARGO), for improving LLM based test update localization in a multi-agent pipeline. Although several automated prompt optimization paradigms exist, our study focuses specifically on text-gradient based optimization as a way to automatically revise prompts based on feedback from localization errors. As part of this investigation, it also examines whether incorporating configurable prompt strategies, such as structural prompt modifications, can further guide the optimization process. The approach is integrated into an existing test localization framework and evaluated using a ground-truth dataset of real-world code changes and corresponding test updates. Our evaluation compares automatically optimized prompts against manually crafted baseline prompts across multiple real-world repositories, while accounting for the non-deterministic nature of LLM outputs. The analysis considers not only the direct effect of prompt optimization on the target module, but also how changes in that module influence downstream collaborating modules in the multi-agent pipeline. The results demonstrate that adaptive and structurally-aware prompt optimization can improve the performance and practical applicability of LLM-based test update localization, contributing to more reliable automation in software maintenance work flows.