Leveraging learning techniques for agricultural robotics

Harjatin Baweja
Master's Thesis, Tech. Report, CMU-RI-TR-19-65, August, 2019

Download Publication

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


Effective plant breeding requires scientists to find correspondences between genetic markers and desirable physical traits of the genotype. While the cost for gene sequencing has gone down by several orders of magnitude in the last two decades, measuring physical traits of a plant at scale is still a labour intensive task making it borderline intractable. This creates an opportunity for robotics to fill this gap in order to accelerate the breeding pipeline. Aided by current progress in machine learning, in this thesis we elucidate methods to perform high throughput non contact and contact based plant phenotyping. We developed a general purpose computer vision pipeline for plant physical trait detection and semantic segmentation. We tested this pipeline for Sorghum stalk counting and stalk width measurement from images. We discuss the shortcomings of having separate perception and manipulation modules for contact based phenotyping and propose an end to end reinforcement learning framework for learning from RGBD observations. We propose a principled online learning approach to weight different auxiliary losses in order to accelerate reinforcement learning. Our approach gives a 3x speed up for reaching and manipulation tasks in three simulated environments.

author = {Harjatin Baweja},
title = {Leveraging learning techniques for agricultural robotics},
year = {2019},
month = {August},
school = {},
address = {Pittsburgh, PA},
number = {CMU-RI-TR-19-65},
keywords = {Plant Phenotyping; Reinforcement Learning;},
} 2019-08-15T11:15:04-04:00