Intermediate Representations for LLM-Based Test Generation: A Comparative Study in Automotive Software Verification

Abstract

Large Language Models (LLMs) can generate test scripts from natural-language requirements, but direct generation often misses boundary behaviors, fault handling, and requirement-specific signal details. We investigate whether intermediate representations (IRs)—artifacts inserted between requirements and executable test scripts—can improve the completeness, traceability, and practical utility of LLM based test generation in an industrial automotive context. We design a unified four-path framework: three generation paths (A–C) that vary only the IR type while holding constant the LLM family, discovery infrastructure, and output format—no IR (direct generation), a structured JSON reasoning document, and an executable Python state-machine specification (xSpec)—and one augmentation path (D) that uses the executable IR in a separate enhancement mode to identify gaps in existing human-authored tests rather than generating test scripts from scratch. The framework is evaluated on 31 functional requirements from Volvo Group’s body/cabin electronics domain, with three runs per configuration to account for LLM nondeterminism. Results show that introducing IR improves static test-script quality—as measured by behavioral completeness and requirement traceability—over direct generation. All quality assessments are based on static analysis of test-script content. Comparing IR types, the executable IR outperforms the structured IR on traceability, but the two are statistically indistinguishable on completeness, signal correctness, and value correctness—indicating that structuring the LLM’s reasoning provides most of the measurable benefit, with executability adding a targeted advantage. The augmentation path preserves 94.7% of existing human-authored behaviors while identifying coverage gaps with 87.5% precision and 63.2% recall, and a median coverage improvement of +0.302. Most identified gaps (62.5%) are fault-condition behaviors, the category where existing test suites are weakest. These findings support a staged adoption strategy: structured IRs for immediate quality gains, executable IRs where traceability matters most, and IR-guided augmentation to extend existing trusted tests rather than replace them.