Pause the Video: Quick but quantitative expert evaluation of tutorial choices in a Reading Tutor that listens - Robotics Institute Carnegie Mellon University

Pause the Video: Quick but quantitative expert evaluation of tutorial choices in a Reading Tutor that listens

Jack Mostow, Cathy Huang, and Brian Tobin
Conference Paper, Proceedings of International Conference on Artificial Intelligence in Education (AIED '01), May, 2001

Abstract

To critique Project LISTEN's automated Reading Tutor, we adapted a panel-of-judges methodology for evaluating expert systems. Three professional elementary educators watched 15 video clips of the Reading Tutor listening to second and third graders read aloud. Each expert chose which of 10 interventions to make in each situation. To keep the Reading Tutor's choice from influencing the expert, we paused each video clip just before the Reading Tutor intervened. After the expert responded, we played back what the Reading Tutor had actually done. The expert then rated its intervention compared to hers. Although the experts seldom agreed, they rated the Reading Tutor's choices as better than their own in 5% of the cases, equally good in 36%, worse but OK in 41%, and inappropriate in only 19%. The lack of agreement and the surprisingly favorable ratings together suggest that either the Reading Tutor's choices were better than we thought, the experts knew less than we hoped, or the clips showed less than they should.

Notes
to appear

BibTeX

@conference{Mostow-2001-8211,
author = {Jack Mostow and Cathy Huang and Brian Tobin},
title = {Pause the Video: Quick but quantitative expert evaluation of tutorial choices in a Reading Tutor that listens},
booktitle = {Proceedings of International Conference on Artificial Intelligence in Education (AIED '01)},
year = {2001},
month = {May},
}