Next, this thesis explores how systems can detect subtle emotional responses to unexpected system behaviors, by designing and collecting data from a human study of participants interacting with a driving simulation system to perform non-critical tasks. Findings shown that participants’ emotional responses to different stimuli such as surprise, confusion, and frustration, could be distinguished based on facial action units, providing important insights to building a real-time autonomous emotional response detector.

Finally, this thesis studies a few post-hoc expectation mismatch mitigation strategies. Specifically, this thesis focuses on acknowledgment, apology, and explanation to mitigate negative emotional responses by humans using verbal, visual, and nonverbal modalities. Analyses shown that acknowledgment and apology as sole mitigation strategies are not sufficient to influence people’s emotional responses in driving simulator settings, and further actions from the system are preferred. Moreover, different modalities of explanation, such as verbal and visual, are comparably effective in reducing unexpectedness in robot behaviors, and such modes of explanations could retain their effectiveness cross-platform.