Carnegie Mellon University
The
Benchmarking in Manipulation Research: The YCB Object and Model Setand Benchmarking Protocols

Berk Calli, Aaron Walsman , Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron Dollar
IEEE Robotics and Automation Magazine (IEEE RAM), August, 2015.


Download
  • Adobe portable document format (pdf) (2MB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Abstract
In this paper we present the Yale-CMU-Berkeley (YCB) Object and Model set, intended to be used to facilitate benchmarking in robotic manipulation research. The objects in the set are designed to cover a wide range of aspects of the manipulation problem. The set includes objects of daily life with different shapes, sizes, textures, weight and rigidity, as well as some widely used manipulation tests. The associated database provides high-resolution RGBD scans, physical properties, and geometric models of the objects for easy incorporation into manipulation and planning software platforms. In addition to describing the objects and models in the set along with how they were chosen and derived, we provide a framework and a number of example task protocols, laying out how the set can be used to quantitatively evaluate a range of manipulation approaches including planning, learning, mechanical design, control, and many others. A comprehensive literature survey on existing benchmarks and object datasets is also presented and their scope and limitations are discussed. The set will be freely distributed to research groups worldwide at a series of tutorials at robotics conferences. Subsequent sets will be otherwise available to purchase at a reasonable cost. It is our hope that the ready availability of this set along with the ground laid in terms of protocol templates will enable the community of manipulation researchers to more easily compare approaches as well as continually evolve standardized benchmarking tests and metrics as the field matures.

Keywords
robotic manipulation, robotics, perception, benchmarks, motion planning, objects, manipulation, control, quality of life

Notes
Associated Center(s) / Consortia: Quality of Life Technology Center, National Robotics Engineering Center, and Center for the Foundations of Robotics
Associated Lab(s) / Group(s): Personal Robotics

Text Reference
Berk Calli, Aaron Walsman , Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron Dollar, "Benchmarking in Manipulation Research: The YCB Object and Model Setand Benchmarking Protocols," IEEE Robotics and Automation Magazine (IEEE RAM), August, 2015.

BibTeX Reference
@inproceedings{Walsman__2015_7970,
   author = "Berk Calli and Aaron {Walsman } and Arjun Singh and Siddhartha Srinivasa and Pieter Abbeel and Aaron Dollar",
   editor = "IEEE",
   title = "Benchmarking in Manipulation Research: The YCB Object and Model Setand Benchmarking Protocols",
   booktitle = "IEEE Robotics and Automation Magazine (IEEE RAM)",
   month = "August",
   year = "2015",
}