Benchmarking Framework - Evaluating Corpora Search Systems Using a Spectrum of Complex Search Queries

Abstract

This thesis presents a benchmark designed to evaluate corpora search systems in digital humanities, focusing on handling complex search queries. By analyzing usergenerated queries from Korp server logs, the research examines various query structures and computational challenges. The study develops a two-level abstraction process to simplify query patterns, enabling clustering and feature extraction techniques. The Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) algorithm is used to segment queries into clusters representing different user behaviors. The performance of these clusters is evaluated by executing queries on selected corpora using Corpus Workbench (CWB), with execution times as the main metric. The results indicate variability in query execution efficiency, influenced by corpus size and query complexity. To improve benchmark clarity, clusters with similar structures were grouped to create a more focused evaluation framework. The findings show how query complexity and data volume affect system performance, highlighting the importance of scalability in corpora search systems. This benchmark provides a framework for evaluating the performance of corpora search systems in digital humanities, contributing to the development of digital research tools in this field.