Feasibility evaluation of Business Intelligence tools as measurement systems: an industrial case study

Abstract

Context: Collecting, analyzing and presenting software metrics are common practices in the software engineering industry. These practices can be done by using either dedicated tools like Rational Insight or tools which are proprietary. The trade-offs between these two types of tools are, for example, suitability for purpose vs. cost of maintenance. Objectives: The objective of this research is to compare the existing proprietary measurement systems framework with business intelligence tools represented by IBM/Rational Insight with respect to functionality and performance. The research also aims to find features which need to be considered while adopting a business intelligence tool as a measurement system in software development organizations. Methods: The research method is a case study and data was collected by conducting interviews with managers, designers and developers of the existing proprietary measurement systems. In order to illustrate the workflow of business intelligence tools, ten scenarios were created. Questionnaires and surveys were conducted to gather stakeholders’ ideas about the evaluation of the scenarios. Results: The usability of Rational Insight was evaluated using the ten scenarios and the realization of these scenarios were shown and accepted in workshops and interviews. Conclusions: The study has shown that business intelligence tools bring integrity into measurement systems and perform better compared to custom measurement system that is used at Ericsson. The capabilities of business intelligence tools are higher than studied proprietary measurement systems.