Home/Cross-stitch Networks for Multi-task Learning

Cross-stitch Networks for Multi-task Learning

Ishan Misra, Abhinav Shrivastava, Abhinav Gupta and Martial Hebert
Conference Paper, IEEE Conference on Computer Vision and Pattern Recognition, June, 2016

Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.


Multi-task learning in Convolutional Networks has displayed remarkable success in the field of recognition. This success can be largely attributed to learning shared representations from multiple supervisory tasks. However, existing multi-task approaches rely on enumerating multiple network architectures specific to the tasks at hand, that do not generalize. In this paper, we propose a principled approach to learn shared representations in ConvNets using multi-task learning. Specifically, we propose a new sharing unit: “cross-stitch” unit. These units combine the activations from multiple networks and can be trained end-to-end. A network with cross-stitch units can learn an optimal combination of shared and task-specific representations. Our proposed method generalizes across multiple tasks and shows dramatically improved performance over baseline methods for categories with few training examples

BibTeX Reference
title = {Cross-stitch Networks for Multi-task Learning},
author = {Ishan Misra and Abhinav Shrivastava and Abhinav Gupta and Martial Hebert},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
publisher = {IEEE},
month = {June},
year = {2016},