Human-AI Runtime Requirements Validation: A Feedback Driven Framework - An investigation into the design, implementation and evaluation of a Human-AI framework for runtime requirements validation with a feedback driven approach.

Abstract

Requirements engineering is a vital process to ensure all stakeholder needs are met appropriately. However, stakeholder requirements are prone to constant change due to changing stakeholder needs, lack of domain knowledge, and long feedback loops. Many companies struggle to capture continuously evolving requirements; this is also true at the case company. The case company has developed Tool X, which requires extensive requirements engineering to be performed before usage. However, the requirements at the company evolve often, with each user having a unique set of requirements that could be unclear due to the highly interdependent nature of the data. To capture these ever-evolving requirements at runtime, this research applied a Design Science Research (DSR) approach to investigate the problem, and to design, implement, and evaluate a solution framework at the case company. Based on insights extracted from 10 stakeholder interviews, the study designed a multi-agent Human-AI runtime requirements validation framework. This framework utilizes a Knowledge Graph and three distinct Data Processing Pipelines (DPPs) to create a continuous feedback-driven loop, ensuring the human-in-the-loop remains firmly in charge. The evaluation of the implemented framework demonstrated robust technical results: the system achieved a 78.4% error detection rate with zero false positives, and 88.6% of its technical suggestions were accurate. Furthermore, user testing confirmed the AI-assisted process was highly usable, useful for achieving deployment goals, and significantly less taxing than manual validation. These findings highlight that shifting to continuous runtime validation drastically speeds up feedback loops, provides stakeholders with domain knowledge needed to validate as human in charge, practitioners prefer AI as a collaborative assistant rather than an autonomous actor.