The Robotics Institute

Carnegie Mellon Robotics Institute

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Abstract

To critique Project LISTEN's automated Reading Tutor, we adapted a panel-of-judges methodology for evaluating expert systems. Three professional elementary educators watched 15 video clips of the Reading Tutor listening to second and third graders read aloud. Each expert chose which of 10 interventions to make in each situation. To keep the Reading Tutor's choice from influencing the expert, we paused each video clip just before the Reading Tutor intervened. After the expert responded, we played back what the Reading Tutor had actually done. The expert then rated its intervention compared to hers.

Although the experts seldom agreed, they rated the Reading Tutor's choices as better than their own in 5% of the cases, equally good in 36%, worse but OK in 41%, and inappropriate in only 19%. The lack of agreement and the surprisingly favorable ratings together suggest that either the Reading Tutor's choices were better than we thought, the experts knew less than we hoped, or the clips showed less than they should.

Notes

Associated Lab(s) / Group(s): Project LISTEN

Associated Project(s): Project LISTEN's Reading Tutor

Note: to appear

Text Reference

Jack Mostow, Cathy Huang, and Brian Tobin, "Pause the Video: Quick but quantitative expert evaluation of tutorial choices in a Reading Tutor that listens," Proceedings of the Tenth Artificial Intelligence in Education (AI-ED) Conference, May, 2001.

BibTeX Reference

@inproceedings{Mostow_2001_3668,
   author = "Jack Mostow and Cathy Huang and Brian Tobin",
   title = "Pause the Video: Quick but quantitative expert evaluation of tutorial choices in a Reading Tutor that listens",
   booktitle = "Proceedings of the Tenth Artificial Intelligence in Education (AI-ED) Conference",
   month = "May",
   year = "2001",
   Notes = "to appear"
}

The Robotics Institute

Menu