Exploring LLM Performance For Vulnerability Mitigation

Abstract

With software systems becoming larger and more complex, traditional manual approaches to detecting and correcting security flaws are no longer efficient. On the other hand, LLMs have emerged as a promising solution for automated vulnerability repair; their reliability in security-critical workflows remains a significant concern due to their non-deterministic nature and potential to introduce new vulnerabilities. Current research is often limited by a lack of standardized benchmarking and systematic comparisons across multiple models using real-world datasets. This thesis addresses these gaps by methodically evaluating the effectiveness of nine modern LLMs, including code-specialized models like StarCoder2-15B and general purpose models like Phi-2 in generating secure and functionally correct code fixes. Utilizing the CrossVul dataset, which provides paired samples of vulnerable and fixed code across different languages: C, C++, Java, and Python, we investigate how model performance varies across different programming languages and vulnerability types by four different zero-shot prompt categories: Basic, Role-based, Constraint-Based, and Comprehensive. To evaluate the performance of the selected models, our methodology combines several quantitative metrics, like CodeBLEU, AST Edit Distance, and ROUGE to compare the model-generated code with the ground truth, and further supports these results through manual analysis of the generated code. By conducting repeated executions to measure consistency, this study aims to provide an evidence-based framework for choosing reliable LLM-based agents. Ultimately, the findings will offer practical guidance for software security professionals and contribute to the academic discourse on leveraging AI for safer software development.