Graphics enhanced version of this site

Evaluation purposes, excuses, and methods: Experience from a Reading Tutor that listens
J. Mostow
Interactive Literacy Education: Facilitating Literacy Environments Through Technology, C. K. Kinzer, L. Verhoeven, ed., Erlbaum Publishers, Mahway, NJ, 2005.

Jump to: Abstract | Notes | Text Reference | BibTeX Reference

This chapter gives three good reasons to evaluate reading software, identifies three methods for doing so, and refutes three excuses for not evaluating - namely, that evaluation is premature, unnecessary, or will be done by others:

(1) Wizard of Oz experiments help test whether (and clarify how) a proposed approach might work, and refute the excuse that evaluation is premature because the approach has not yet been implemented in a proposed system that may take years to develop.

(2) Conventional controlled studies help determine whether an implemented system helps children gain more in reading than they would otherwise. This criterion is necessary to improve on the status quo, but the difficulty of meeting it refutes the excuse that evaluation is unnecessary due to the supposedly innate superiority of learning on computers, or of a proposed way to use them.

(3) Experiments embedded in an automated tutor help analyze which tutorial actions help which students and words, thereby guiding improvement of the tutor in ways that third party evaluation cannot, thus refuting the excuse that evaluation can be left to others.

The chapter details some practical lessons learned from designing, performing, and analyzing experiments embedded in Project LISTEN's school-deployed Reading Tutor, which uses speech recognition to listen to children read aloud, and is helping hundreds of children learn to read.

Associated lab/group: Project LISTEN
Associated project: Project LISTEN's Reading Tutor

Note: to appear

J. Mostow, "Evaluation purposes, excuses, and methods: Experience from a Reading Tutor that listens," Interactive Literacy Education: Facilitating Literacy Environments Through Technology, C. K. Kinzer, L. Verhoeven, ed., Erlbaum Publishers, Mahway, NJ, 2005.

@incollection{Mostow_2005_4646,
   author = "Jack Mostow",
   editor = "C. K. Kinzer, L. Verhoeven",
   title = "Evaluation purposes, excuses, and methods: Experience from a Reading Tutor that listens",
   booktitle = "Interactive Literacy Education: Facilitating Literacy Environments Through Technology",
   publisher = "Erlbaum Publishers",
   address = "Mahway, NJ",
   year = "2005",
   note = "to appear"
}