Towards Explainable Embodied AI

Abstract

The performance of autonomous agents has improved with the advancements in learning and planning algorithms, but the applicability of such agents in the human-inhabited world is limited. One of the factors is that humans find it difficult to interpret the model's decision-making and thus, do not trust it as a teammate. The goal of explainable embodied AI is to provide predictions with explanations that clarify the internal logic and are human-understandable.
This work approaches explainability in AI agent's policies by taking inspiration from how humans explain. First, humans are known to associate a few dominant factors while explaining their decision. We visualize such vital features of the trained policies for control and navigation tasks by gradient-based attribution methods.
Second, having shared knowledge for representing concepts often helps to understand and explain the decision-making. We utilize language priors in navigation algorithms for robots assisting in simulated urban households and search-and-rescue settings.
Third, explainability comes with structured reasoning. To bring explainability in architecture design, we learn modular and hierarchical navigation policies to maximize area coverage in unseen environments.
We conclude that embodied AI policies can be understood with feature attributions to explain how input state features influence the predicted actions. However, feature attributions are not human intelligible in all cases, and attributions for the same policy are sensitive to `reference' or `baseline' design choice. A complementary direction is to develop inherently explainable policies by incorporating common knowledge priors and modular hierarchical components, allowing humans to understand high-level information flow and influence AI decisions. We hope that the proposed explainability methods for embodied AI facilitate the analysis of policy failure cases in different out-of-distribution scenarios.