News

Test flakiness is a huge issue in large-scale industrial software systems. We asked ourselves: Do test and environmental complexity increase flakiness? Our starting point were hypotheses we formulated based on our experience, previous papers, and blog posts. We then validated those hypotheses in the context of SAP HANA. With our study, we confirmed some hypotheses (e.g., longer-running tests are more likely to be flaky) but also had some unexpected results (e.g., load on the testing infrastructure was not correlated with higher flakiness). The paper is going to be presented at the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2024) in Barcelona, and a preprint is already available.

Update 2024-10-25: We received a best paper award for this paper!