/Predictive-State Decoders: Encoding the Future Into Recurrent Neural Networks

Predictive-State Decoders: Encoding the Future Into Recurrent Neural Networks

Arun Venkatraman, Nicholas Rhinehart, Wen Sun, Lerrel Pinto, Martial Hebert, Byron Boots, Kris M. Kitani and J. Andrew (Drew) Bagnell
Conference Paper, Neural Information Processing Systems (NIPS), December, 2017

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Recurrent neural networks (RNNs) are a vital modeling technique that rely on internal states learned indirectly by optimization of a supervised, unsupervised, or reinforcement training loss. RNNs are used to model dynamic processes that are characterized by underlying latent states whose form is often unknown, precluding its analytic representation inside an RNN. In the Predictive-State Representation (PSR) literature, latent state processes are modeled by an internal state representation that directly models the distribution of future observations, and most recent work in this area has relied on explicitly representing and targeting sufficient statistics of this probability distribution. We seek to combine the advantages of RNNs and PSRs by augmenting existing state-of-the-art recurrent neural networks with Predictive-State Decoders (PSDs), which add supervision to the network’s internal state representation to target predicting future observations. Predictive-State Decoders are simple to implement and easily incorporated into existing training pipelines via additional loss regularization. We demonstrate the effectiveness of PSDs with experimental results in three different domains: probabilistic filtering, Imitation Learning, and Reinforcement Learning. In each, our method improves statistical performance of state-of-the-art recurrent baselines and does so with fewer iterations and less data.

BibTeX Reference
author = {Arun Venkatraman and Nicholas Rhinehart and Wen Sun and Lerrel Pinto and Martial Hebert and Byron Boots and Kris M. Kitani and J. Andrew (Drew) Bagnell},
title = {Predictive-State Decoders: Encoding the Future Into Recurrent Neural Networks},
booktitle = {Neural Information Processing Systems (NIPS)},
year = {2017},
month = {December},