/The Pose Knows: Video Forecasting by Generating Pose Futures

The Pose Knows: Video Forecasting by Generating Pose Futures

Jacob Walker, Kenneth Marino, Abhinav Gupta and Martial Hebert
Conference Paper, ICCV, October, 2017

Download Publication (PDF)

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Current approaches to video forecasting attempt to generate
videos directly in pixel space using Generative Adversarial
Networks (GANs) or Variational Autoencoders
(VAEs). However, since these approaches try to model all
the structure and scene dynamics at once, in unconstrained
settings they often generate uninterpretable results. Our insight
is that forecasting needs to be done first at a higher
level of abstraction. Specifically, we exploit human pose detectors
as a free source of supervision and break the video
forecasting problem into two discrete steps. First we explicitly
model the high level structure of active objects in the
scene (humans) and use a VAE to model the possible future
movements of humans in the pose space. We then use
the future poses generated as conditional information to a
GAN to predict the future frames of the video in pixel space.
By using the structured space of pose as an intermediate
representation, we sidestep the problems that GANs have in
generating video pixels directly. We show through quantitative
and qualitative evaluation that our method outperforms
state-of-the-art methods for video prediction.

Associated Lab: Computer Vision Lab

BibTeX Reference
author = {Jacob Walker, Kenneth Marino, Abhinav Gupta, Martial Hebert},
title = {The Pose Knows: Video Forecasting by Generating Pose Futures},
booktitle = {ICCV},
year = {2017},
month = {October},