Does help help? Introducing the Bayesian Evaluation and Assessment methodology

Joseph E. Beck, K.-m. Chang, Jack Mostow, and Albert Corbett
9th International Conference on Intelligent Tutoring Systems, July, 2008, pp. 383 - 394.


Download
  • Adobe portable document format (pdf) (80KB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Abstract
Most ITS have a means of providing assistance to the student, either on student request or when the tutor determines it would be effective. Presumably, such assistance is included by the ITS designers since they feel it benefits the students. However, whether-and how-help helps students has not been a well studied problem in the ITS community. In this paper we present three approaches for evaluating the efficacy of the Reading Tutor's help: creating experimental trials from data, learning decomposition, and Bayesian Evaluation and Assessment, an approach that uses dynamic Bayesian networks. We have found that experimental trials and learning decomposition both find a negative benefit for help--that is, help hurts! However, the Bayesian Evaluation and Assessment framework finds that help both promotes student long-term learning and provides additional scaffolding on the current problem. We discuss why these approaches give divergent results, and suggest that the Bayesian Evaluation and Assessment framework is the strongest of the three. In addition to introducing Bayesian Evaluation and Assessment, a method for simultaneously assessing students and evaluating tutorial interventions, this paper describes how help can both scaffold the current problem attempt as well as teach the student knowledge that will transfer to later problems.

Notes
Associated Lab(s) / Group(s): Project LISTEN
Number of pages: 12
Note: ITS2008 Best Paper Award

Text Reference
Joseph E. Beck, K.-m. Chang, Jack Mostow, and Albert Corbett, "Does help help? Introducing the Bayesian Evaluation and Assessment methodology," 9th International Conference on Intelligent Tutoring Systems, July, 2008, pp. 383 - 394.

BibTeX Reference
@inproceedings{Beck_2008_6219,
   author = "Joseph E Beck and K.-m. Chang and Jack Mostow and Albert Corbett",
   title = "Does help help? Introducing the Bayesian Evaluation and Assessment methodology",
   booktitle = "9th International Conference on Intelligent Tutoring Systems",
   pages = "383 - 394",
   month = "July",
   year = "2008",
   number= "report_number",
   Notes = "ITS2008 Best Paper Award"
}