Variational debugging : understandig differences among executions

Abstract

Interactions among multiple program inputs or options can lead to undesired or wrong behavior, such as system crashes and security vulnerabilities. This is especially challenging for large numbers of inputs such as for highly configurable systems, as the number of configurations to test grows exponentially with the number of boolean options. A lot of research has focused on systematically covering this con- figuration space, however there is little knowledge on how inputs interact as they can only be observed from their effects (e.g., from bug reports). A better understanding of interactions is needed to improve quality assurance. In this thesis, we developed a dynamic analysis based on variational execution, that monitors data and control ow interactions among all options simultaneously. We analyzed the program traces of multiple medium sized highly-configurable systems to characterize and identify where and how interactions occur. We found that the essential configuration complexity (i.e., the degree of interactions occurring during executions) is indeed much lower than the combinatorial explosion, but that the pattern of how options interact are more nuanced than what state of the art analysis techniques exploit. Interaction characteristics can inform analysis techniques, however, understanding of a specific interaction (e.g., a system crash), is challenging as this requires understanding how multiple inputs interact with each other to cause this undesired behavior while the program succeeds otherwise. Debugging such faults requires understanding the individual effects of inputs and how they interact to cause the fault. Contrasting traces of failing and the succeeding executions can reveal the interactions in their differences, and thus the information needed to understand the fault. We propose to align the execution traces of all configurations to discover and explain interactions. As complete traces are too large to be used for debugging, we present variational traces that concisely represent the differences on data and control flow caused by interactions. We again use variational executions to scale the generation of variational traces to exponential configuration spaces. To enable programmers interacting with such variational traces, we provide our debugging tool Varviz. We have shown that variational traces improve the performance of debugging variability faults by a factor of two compared to standard debuggers.