Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems

O. Firschein, M. Fischler, and Takeo Kanade
Proceedings of the 1993 DARPA Image Understanding Workshop, May, 1993.


Download
  • Adobe portable document format (pdf) (168KB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Abstract
We discuss the need for a new series of benchmarks in the vision field. to provide a direct quantitative measure of progress understandable to sponsors of research as well as a guide to practitioners in the field. A first set of benchmarks in two categories is proposed (1) static scenes containing manmade objects, and (2) static naturalloutdoor scenes. The tests are "end-to-end" and involve determining how well a system can identify instances (an item or condition is present or absent) in selected regwns of an image. The scoring would be set up so that the automatic setting of adjustable parameters is rewarded and manual tuning is penalized. To show how far machine vision has yet to go, a Benchmark 2000 problem is also suggested using children's "what is wrong" puzzles in which defective objects in a line drawing of a scene must be found.

Notes
Associated Center(s) / Consortia: Vision and Autonomous Systems Center
Number of pages: 6

Text Reference
O. Firschein, M. Fischler, and Takeo Kanade, "Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems," Proceedings of the 1993 DARPA Image Understanding Workshop, May, 1993.

BibTeX Reference
@inproceedings{Kanade_1993_1351,
   author = "O. Firschein and M. Fischler and Takeo Kanade",
   title = "Creating Benchmarking Problems in Machine Vision: Scientific Challenge Problems",
   booktitle = "Proceedings of the 1993 DARPA Image Understanding Workshop",
   month = "May",
   year = "1993",
}