Ego-Surfing: Person Localization in First-Person Videos Using Ego-Motion Signatures

Ryo Yonetani, Kris M. Kitani and Yoichi Sato
Journal Article, IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2018), November, 2018

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.


We envision a future time when wearable cameras are worn by the masses and recording first-person point-of-view videos of everyday life. While these cameras can enable new assistive technologies and novel research challenges, they also raise serious privacy concerns. For example, first-person videos passively recorded by wearable cameras will necessarily include anyone who comes into the view of a camera — with or without consent. Motivated by these benefits and risks, we developed a self-search technique tailored to first-person videos. The key observation of our work is that the egocentric head motion of a target person (ie, the self) is observed both in the point-of-view video of the target and observer. The motion correlation between the target person’s video and the observer’s video can then be used to identify instances of the self uniquely. We incorporate this feature into the proposed approach that computes the motion correlation over densely-sampled trajectories to search for a target individual in observer videos. Our approach significantly improves self-search performance over several well-known face detectors and recognizers. Furthermore, we show how our approach can enable several practical applications such as privacy filtering, target video retrieval, and social group clustering.

author = {Ryo Yonetani and Kris M. Kitani and Yoichi Sato},
title = {Ego-Surfing: Person Localization in First-Person Videos Using Ego-Motion Signatures},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI 2018)},
year = {2018},
month = {November},
} 2018-11-06T14:45:02-04:00