A Requirements-Aware Prompting Framework for LLM-Based Code Generation - Design and Empirical Evaluation

Abstract

Large language models (LLMs) are increasingly used to generate software code from natural-language descriptions, yet the quality of the input requirement is rarely treated as an independent variable in such systems. In practice, software requirements are often ambiguous, incomplete, inconsistent, or difficult to verify, creating uncertainty for LLM-based code generation. This thesis presents a requirements aware prompting framework grounded in the ISO/IEC/IEEE 29148 requirements quality standard and empirically evaluates how requirement quality and prompt structure influence LLM behaviour during code generation. The proposed framework operationalises four requirement-quality attributes: un ambiguity, completeness, verifiability, and consistency by incorporating them into structured prompts through quality-aware constraints and behavioural routing policies. Six complementary empirical studies were conducted across three large language models (GPT-4o-mini, GPT-4.1, and Claude Sonnet 4.6) using both bench mark and real-world software engineering datasets. Benchmark evaluation included HumanEval, a collection of 164 Python programming tasks with hidden unit tests, and MBPP Sanitized, a benchmark of 427 short programming problems, out of which 413 were evaluated under synthetically degraded requirement-quality conditions intended as a methodological stress test. Real-world evaluation used the PURE corpus, comprising 2,503 natural-language software requirements extracted from publicly available Software Requirements Specification (SRS) documents, with 1,000 requirements selected for behavioural analysis, alongside a corroborative SRS dataset of 47 industrial-style requirements derived from the FarmBot system. Results indicate that prompt structure has limited influence on functional correctness of the LLM generated code. Across HumanEval and MBPP, compiled constraints prompting produced no statistically significant improvement in pass rates. However, the MBPP quality degradation procedure was methodologically limited, as benchmark tasks retained intact behavioural specifications and embedded examples despite degradation annotations. Consequently, benchmark findings are interpreted cautiously and do not constitute evidence that requirement quality is irrelevant to code generation correctness. Nonetheless, substantial behavioural shifts emerged in LLM output behaviour when the framework was applied to real world software requirements. On the PURE dataset, compiled prompts increased diagnostic outputs from near-zero levels under baseline prompting to between 63% and 73% across evaluated models, with Claude Sonnet 4.6 producing the strongest diagnostic response (73.4%). Similar behavioural patterns were observed on the in dependent FarmBot SRS dataset, providing preliminary cross-domain corroboration of the PURE findings. Requirement-level behavioural correlations and consistency analyses indicate systematic relationships between requirement quality and LLM output behaviour, particularly the transition between implementation and diagnostic responses, with un ambiguity showing the strongest observed association and verifiability appearing theoretically important but less reliably assessed in the automated grading pipeline. Because the framework used pre-defined quality thresholds to determine whether requirements should be implemented or diagnosed, logistic regression is interpreted primarily as a validation of the framework’s internal logic routing consistency, while cross-model behavioural comparisons provide the main evidence for how requirement quality influences LLM output behaviour. The findings establish a clear distinction between correctness and behaviour in LLM based code generation. While prompt design does not significantly improve correct ness for well-specified benchmark tasks, it provides a powerful mechanism for con trolling model behaviour when requirements are quality-defective. The proposed framework enables quality-aware interaction with LLMs by identifying defective requirements before implementation proceeds, thereby supporting safer and more reliable integration of LLMs into software engineering workflows.