TableAnalyst: an LLM-agent for tabular data analysis - Implementation and evaluation on tasks of varying complexity

Abstract

The automotive industry relies on manual analysis of tabular data, particularly for software test result evaluation. This time-consuming process diverts engineers from their core expertise. This study explores the application of a large language model (LLM) agent to address this challenge. We present a prototype LLM agent specifically designed for analyzing software test data in the automotive industry. Our agent employs an iterative feedback approach, utilizing two key modules: a planner and a code interpreter (CI). The planner decomposes user queries into step-by-step plans for data manipulation, while the CI translates these steps into executable code. To evaluate the agent’s output quality, we introduce a novel dataframe similarity metric validated by a domain expert. This metric demonstrates promise as a valuable tool for evaluating table analysis agents. Furthermore, we explore task complexity metrics through correlation analysis. The results suggest that LLMs can identify complexity within the analysis query. However, the results also suggest that LLMs may struggle to translate this query into actionable plan steps. Overall, this work demonstrates the potential of LLM agents for automating software test data analysis tasks in the automotive industry. The development of complexity metrics and a novel evaluation metric contributes to further research and improvement of such agents.