Analysing Test-Driven Development Adherence in Open-Source Projects Using Test-to-Code Traceability Links

Abstract

Several studies have been conducted to determine the impact of Test-Driven Develop- ment (TDD) on software quality. Many studies utilise test-to-code traceability tools and strategies to detect TDD adherence as part of their methodology. However, most test-to-code traceability tools rely on filename-based matching algorithms, which suffer from low recall; therefore, most TDD detection methods are not accurate. This study aims to assess the effectiveness of a statement-based matching algorithm over a filename-based one and whether it can be used to detect the TDD adherence of a software project. The filename-based and statement-based matching algorithms are implemented in Python. To evaluate these algorithms, 500 tests from sixteen Java projects (encompassing frameworks, libraries, and tools for data processing, testing, and web services) are used. These projects range in size from 5,115 lines of code (LoC) to 378,167 LoC. This evaluation helps in understanding the performance of the algorithms through their weighted F1-scores. A mathematical function is created from first principles to correlate method coverage and time deltas with the test-with-development (TWD) adherence of a project. Thereafter, 100 Java projects are used to demonstrate the utility of this function. The results show that the statement-based matching algorithm, with a weighted F1-score of 0.771 and a 95% confidence interval of [0.741, 0.786], is more accurate than the filename-based matching algorithm, with a weighted F1-score of 0.218 and a 95% confidence interval of [0.193, 0.246]. Additionally, the results show that the relative TWD adherence of eleven projects is found to be highly correlated with TDD scores from surveys (rs = 0.723 and p-value = 0.012), and the method coverage of eighteen projects is found to be highly correlated with code coverage obtained from Codecov (rs = 0.744 and p-value = 0.0003). The literature review reveals that no studies have explored the correlation between TDD adherence and the development step size in software projects. An investigation is conducted using a statement-based test-to-code traceability tool and relative TWD adherence values to determine if TDD projects have smaller development steps compared to non-TDD projects. The study finds no significant correlation between median production method churn and relative TWD adherence of projects in all but one out of eight cases.