Does Great Power Come with Great Explainability? Comparing Explanation Strategies for Automated Program Diagnosis

Zusammenfassung

Debugging is an important activity in software development, yet providing actionable and comprehensible explanations for program failures remains challenging. Automated tools such as ALHAZEN and AVICENNA address this by using distinct strategies: ALHAZEN employs binary decision trees to show failure-inducing conditions, while AVICENNA uses a specification language to model complex input dependencies. To assess their impact on usability and user efficiency, we conducted a controlled within-between-subjects user study with 18 participants tasked with resolving four software bugs using either tool or no support. Quantitative results showed that both tools improved debugging efficiency compared to manual methods, with AVICENNA offering more precise diagnostics but requiring higher cognitive effort. Qualitative feedback revealed a preference for AVICENNA’s expressiveness despite its complexity. Our findings show that effective debugging tools have tradeoffs between accuracy and interpretability to support developers’ decision-making in increasingly complex software environments.