Investigating Text-only and Multimodal Retrieval Augmented Generation frameworks for Visual Question Answering - A study on the impact of modality and parameter optimization

Abstract

Verifying the correctness of product assembly processes in a manufacturing setting is a crucial part of ensuring the quality of products. Automating this procedure can help improve both the security and the efficiency of the routines. Utilizing a machine learning model for automating these kinds of procedures would require fine-tuning and a lot of computational resources for the algorithm to adapt to the specific domain. An alternative approach that has been shown to enhance the performance to the same extent as fine-tuning while not requiring additional computational resources, is utilizing Retrieval-Augmented Generation (RAG) frameworks. In this project, two RAG frameworks – a Text-only and a Multimodal RAG framework, are developed. The main goal of the frameworks is to accurately answer user queries about products where the ground-truth answer is located in a PDF user manual. The frameworks are developed by integrating a retrieval component and a generative component, either LLaMA2-7B or LLaVA-7B. The retrieval component retrieves relevant to the user query context from manuals, which the generative component uses to base the response on. In addition to exploring how the performances are affected by the modalities of the frameworks, parameter tuning is explored. Evaluating how different values of chunk size and top-k parameters affect the performances allows for optimizing the RAG frameworks. The evaluation is performed by using BERTScore metrics and LangSmith metrics that provide complementary human-like judgment. The most crucial metrics are B-recall and contextual accuracy, which both evaluate how well the generated response captures the information embedded in the ground-truth answer. The results show that the Text-only RAG framework is more stable across changes in the parameters than the Multimodal RAG framework, leading to generating more coherent and rational responses. However, finding the most optimal parameters for the Multimodal RAG framework could lead to it outperforming the Textonly RAG framework. Overall, moderate chunk sizes 128 and 256 and top-k values 4 or 6 led to the best-produced performances for both of the RAG frameworks.