Cooperative Coaching in Robot Learning

Abstract

Many closed loop learning algorithms perform gradient descent on a cost function with respect to the parameters of a learning controller. The authors observe that both local closed loop learners, which consider only the cost of the current time step, and optimal control based closed loop learners, which consider the future effects of control actions, can become stuck in sub-optimal local minima in the cost function. The authors propose the use of "cooperating coaches" to deal with this problem. Each coach attempts gradient descent based on its own cost function and they work together to avoid getting stuck in local minima. When one coach has achieved the best result it can (the gradient for its cost function is zero), another coach takes over to guide the search through the parameter space. The authors demonstrate cooperative coaching on the problem of curve tracking with an inverted pendulum and show that it yields faster, smoother tracking of target curves by combining the best aspects of two different coaches.