Carnegie Mellon University
Differentiable Sparse Coding

David Bradley and J. Andrew (Drew) Bagnell
Proceedings of Neural Information Processing Systems 22, December, 2008.

  • Adobe portable document format (pdf) (494KB)
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.

Prior work has shown that features which appear to be biologically plausible as well as empirically useful can be found by sparse coding with a prior such as a laplacian that promotes sparsity. We show how smoother priors can preserve the benefits of these sparse priors while adding stability to the Maximum A-Posteriori (MAP) estimate that makes it more useful for prediction problems. Additionally, we show how to calculate the derivative of the MAP estimate efficiently with implicit differentiation. One prior that can be differentiated this way is KL-regularization. We demonstrate its effectiveness on a wide variety of applications, and find that online optimization of the parameters of the KL-regularized model can significantly improve prediction performance.

sparse coding, sparse approximation, semi-supervised learning, machine learning

Sponsor: NDSEG Fellowship

Text Reference
David Bradley and J. Andrew (Drew) Bagnell, "Differentiable Sparse Coding," Proceedings of Neural Information Processing Systems 22, December, 2008.

BibTeX Reference
   author = "David {Bradley } and J. Andrew (Drew) Bagnell",
   title = "Differentiable Sparse Coding",
   booktitle = "Proceedings of Neural Information Processing Systems 22",
   month = "December",
   year = "2008",