Mining and comparing software testing metrics and evaluating the use of different test quality metric tools
Mining and comparing software testing metrics and evaluating the use of different test quality metric tools
Abstract
Software test metrics are often compared
with one another to find the most accurate predictor of test
quality, however little is written about the connections between
types of metrics. Moreover, open-source software is more commonly
used to extract these test metrics, but the issues researchers
face when attempting to use them is under-reported. [Objective]
This paper aims to investigate the correlation between coverage,
mutation and diversity-based test metrics, as well as to describe
the challenges involved in automating the use of test quality
metric extraction tools. [Method] We conducted an MSR study
on 22 open-source software repositories from Github which were
mined in order to extract test quality metrics. To do this we
developed a tool which combined several metric extraction tools
to provide code coverage, mutation coverage and average test
suite diversity values. [Results] The challenges we faced when
conducting this MSR study stemmed mostly from trying to
automate the test quality metric extraction process, such as
finding repositories that were compatible with all of our metric
extraction tools, Maven plugins interfering, costly execution time
and tools not working out of the box.We then applied Spearman’s
Rho to calculate the correlation between the metrics. [Conclusion]
Our results show that code coverage, mutation coverage and test
suite diversity all have a positive correlation with each other. We
also found that automating the extraction of test metrics limited
the availability of repositories due to stricter requirements.
Degree
Student essay
Collections
View/ Open
Date
2022-09-28Author
Ekdahl, Mattias
Engström, Simon
Keywords
test quality metrics
mutation testing
mining software repositories
code coverage
test diversity
Language
eng