|
|
|
|
RI | Publications | The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop
|
|
Text only version of this site
The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop
A. Steinfeld, R. Bennett, K. Cunningham, M. Lahut, P. Quinones, D. Wexler, D. Siewiorek, P. Cohen, J. Fitzgerald, O. Hansson, J. Hayes, M. Pool, and M. Drummond
tech. report CMU-CS-06-125, Computer Science Department, Carnegie Mellon University, May, 2006.
Jump to: Download | Abstract | Notes | Text Reference | BibTeX Reference
| Download [Help] |
Adobe portable document format (pdf) [1530 KB]
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.
| Abstract |
The RADAR project involves a collection of machine learning research thrusts that are integrated into a cognitive personal assistant. Progress is examined with a test developed to measure the impact of learning when used by a human user. Three conditions (conventional tools, Radar without learning, and Radar with learning) are evaluated in a a large-scale, between-subjects study. This paper describes the activities of the RADAR Test with a focus on test design, test harness development, experiment execution, and analysis. Results for the 1.1 version of Radar illustrate the measurement and diagnostic capability of the test. General lessons on such efforts are also discussed.
| Notes |
Sponsor: Defense Advanced Research Projects Agency (DARPA)
Grant ID: Contract No. NBCHD030010
Associated centers: VASC, CIMDS, MRTC, SRI, and FRC
Associated labs/groups: Human-Robot Interaction Group and Intelligent Coordination and Logistics Laboratory
Associated project: Reflective Agents with Distributed Adaptive Reasoning
| Text Reference |
A. Steinfeld, R. Bennett, K. Cunningham, M. Lahut, P. Quinones, D. Wexler, D. Siewiorek, P. Cohen, J. Fitzgerald, O. Hansson, J. Hayes, M. Pool, and M. Drummond, The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop, tech. report CMU-CS-06-125, Computer Science Department, Carnegie Mellon University, May, 2006.
| BibTeX Reference |
@techreport{Steinfeld_2006_5578,
author = "Aaron Steinfeld and Rachael Bennett and Kyle Cunningham and Matt Lahut and Pablo-Alejandro Quinones and Django Wexler and Daniel Siewiorek and Paul Cohen and Julie Fitzgerald and Othar Hansson and Jordan Hayes and Mike Pool and Mark Drummond",
title = "The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop",
institution = "Computer Science Department, Carnegie Mellon University",
month = "May",
year = "2006",
number = "CMU-CS-06-125",
address = "Pittsburgh, PA"
}