Graphics enhanced version of this site

Simon Baker
Adjunct Faculty (Adjunct)

No longer a member of RI.

Jump to: Research interests | Keywords | Labs & Groups | Projects | Publications

I do computer vision. Within this field, I am particularly interested in the following areas:

Faces:

Real-Time Non-Rigid Face Tracking / Active Appearance Model Fitting: 2D and 3D Face Model Building: Resolution Enhancement / Hallucinating Faces: Face Databases / PIE: Face Recognition Across Pose / Eigen Light-Fields: Gaze Estimation

3D Reconstruction and Vision/Graphics:

Fundamental Theorem of 3D Vision: Shape-From-Silhouette Across Time: Human Kinematic Modeling: Human Articulated Tracking / Markerless Motion Capture: Markerless Motion Transfer: Scene Flow: Spatio-Temporal View Interpolation: Textureless Layers

Vision Theory:

Fundamental Theorem of 3D Vision: Efficient Image Alignment / Lucas-Kanade 20 Years On Unifying Framework: Fundamental Limits on Super-Resolution: Light-Fields / Theoretical Properties for Stereo and Face Recognition Textureless Layers

Vision for Safe Driving:

Danger Detection / Prediction and Planning: Robust Car Tracking: Bird's Eye View Generation: Driver Head Tracking: Driver Gaze Estimation

Super-Resolution:

Theoretical Limits: Hallucinating Faces: Super-Resolution Optical Flow

Other:

Projector-Camera Systems / Tele-Graffiti: Catadioptric Camera Design: Feature Detection: The Template Update Problem: Automatic Construction of Active Appearance Models: Setting Low-Level Vision Parameters

I work with a variety of different faculty, students, staff, visitors, and alumni including:

Adrian Broadhurst, Vijayakumar Bhagavatula, German Cheung, Jeff Cohn, Bob Collins, Fernando de la Torre, Ralph Gross, Jessica Hodgins, Changbo Hu, Takahiro Ishikawa, Takeo Kanade, Qifa Ke, Iain Matthews, Andreas Nowatzyk, Raju Patil, Jeff Schneider, Steve Seitz, Jianbo Shi, Terence Sim, Jake Sprouse, Naoya Takao, David Tolliver, Sundar Vedula, Jing Xiao

Please see the projects below for more details.

computer vision, object recognition, pattern recognition, sensors, stereo vision, and visual tracking

• Calibrated Imaging Lab - High precision vision and imaging lab
• Face Group - Robust detection, recognition, and tracking of human faces with automated analysis of expressions
• Human Identification at a Distance - We are developing and evaluating human identification technologies as part of the Defense Advanced Research Projects Agency (DARPA) sponsored program in Human Identification at a Distance (HumanID).
• Intelligent Desktop Group - We are developing vision technologies for intelligent desktops.
• Virtualized Reality^TM - Construct views of real events from nearly any viewpoint
• Vision for Safe Driving - Computer vision algorithms and systems for automotive safe driving applications.
• Vision for Virtual Environments - Using techniques from computer vision and robotics, we are developing novel sensing and display technologies to support practical, useful virtual environments.

• 2D->3D Face Model Construction - We develop a linear algorithm that uniquely recovers the 3D non-rigid shapes and poses of a human face from a 2D monocular video.
• AAM Fitting Algorithms - Many varieties of algorithms for fitting Cootes and Taylor's "Active Appearance Models" are developed.
• AAMs with Occlusion - We are developing algorithms to construct AAMs from occluded training images and to efficiently fit AAMs to faces containing occlusion.
• Automatic Construction of Active Appearance Models - An algorithm for the automatic (unsupervised) construction of an Active Appearance Model.
• Car Tracking - Algorithms for tracking cars and generating "bird's eye views" of the surrounding road scene.
• Coplanar Shadowgrams for Acquiring Visual Hulls of Intricate Objects - We present a practical approach to shape-from-silhouettes using a novel technique called coplanar shadowgram imaging that allows us to use dozens to even hundreds of views for visual hull reconstruction.
• Face and Facial Feature Tracking - Rigid Tracking of Faces and Non-Rigid Tracking of Facial Features
• Face Databases - Miscellaneous face databases collected at CMU.
• Face Model Building and Fitting - Techniques for building and fitting 2D and 3D models of human faces and heads.
• Face Recognition - Recognizing people from images and videos.
• Face Recognition Across Pose - Recognizing people from different poses.
• Facial Expression Analysis - Automatic facial expression encoding, extraction and recognition, and expression intensity estimation for the applications of MPEG4 application: teleconferencing, human-computer interaction/interface.
• Gaze Estimation - Algorithms for estimating where someone is looking
• Hallucinating Faces - A super-resolution algorithm with a strong face-specifc prior.
• Human Kinematic Modeling and Motion Capture - We are developing a system for building 3D kinematic models of humans and then using the models to track the person in new video sequences.
• Human Motion Transfer - We are developing a system for capturing the motion of one person and rendering a different person performing the same motion.
• Image Enhancement for Faces - Video enhancement techniques, specifically tailored for human faces.
• Light-fields - A variety of uses of light-fields in computer vision.
• Lucas-Kanade 20 Years On - A Unifying Framework for Image Alginment
• Photometric Limits on Computer Vision - An investigation into the fundamental limits imposed on computer vision algorithms by imperfect or incomplete photometric information.
• PIE Database - A database of 41,368 images of 68 people with Pose, Illumination, and Expression variation.
• Prediction & Planning - This project analyses the safety and interaction of moving objects in complex road scenes.
• Scene Flow - Methods of computing dense, non-rigid motion of 3D scenes.
• Setting Low-Level Vision Parameters - Techniques for feeding back information from high-level vision modules to low-level modules to improve the performance of the overall system.
• Spatio-Temporal View Interpolation - An image-based rendering algorithm for view interpolation across both space and time.
• Super-Resolution Optical Flow - A super-resolution algorithm for complex non-rigid scenes.
• Tele-Graffiti - A system that allows two or more users to communicate remotely via hand-drawn sketches.
• Template Update - We are developing an algorithm to update template tracking that avoids the "drifting" problem of the naive update algorithm.
• Temporal Shape-From-Silhouette - We are developing algorithms for the computation of 3D shape from multiple silhouette images captured across time.
• Textureless Layers - Techniques for the 3D reconstruction of scenes consisting of constant intensity piecewise planar regions (layers).

• Active Appearance Models with Occlusion
  R. Gross, I. Matthews, and S. Baker
  Image and Vision Computing, Vol. 24, No. 6, 2006, pp. 593-604. [Abstract]
  Download: pdf [3404 KB] copyrighted

• Resolution-Aware Fitting of Active Appearance Models to Low-Resolution Images
  G. Dedeoglu, S. Baker, and T. Kanade
  Proceedings of the 9th European Conference on Computer Vision, ECCV 2006, Springer-Verlag, Vol. II, No. 3952, May, 2006, pp. 83 - 97. [Abstract]
  Download: pdf [1908 KB] copyrighted

• Generic vs. person specific active appearance models
  R. Gross, I. Matthews, and S. Baker
  Image and Vision Computing, Vol. 23, No. 11, November, 2005, pp. 1080-1093. [Abstract]
  Download: pdf [2086 KB] copyrighted

• Multi-View AAM Fitting and Camera Calibration
  S.C. Koterba, S. Baker, I. Matthews, C. Hu, J. Xiao, J. Cohn, and T. Kanade
  Proc. International Conference on Computer Vision, Vol. 1, October, 2005, pp. 511 - 518. [Abstract]
  Download: pdf [2053 KB] copyrighted

• Shape-From-Silhouette Across Time: Part II: Applications to Human Modeling and Markerless Motion Tracking
  K.M. Cheung, S. Baker, and T. Kanade
  International Journal of Computer Vision, Vol. 63, No. 3, August, 2005, pp. 225 - 245.
  Download: pdf [2755 KB] copyrighted

• Shape-From-Silhouette Across Time Part I: Theory and Algorithms
  K.M. Cheung, S. Baker, and T. Kanade
  International Journal of Computer Vision, Vol. 62, No. 3, May, 2005, pp. 221 - 247. [Abstract]
  Download: pdf [2047 KB] copyrighted

• Image-Based Spatio-Temporal Modeling and View Interpolation of Dynamic Events
  S. Vedula, S. Baker, and T. Kanade
  ACM Transactions on Graphics, Vol. 24, No. 2, April, 2005, pp. 240 - 261. [Abstract]
  Download: pdf [2053 KB] copyrighted

• Three-Dimensional Scene Flow
  S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No. 3, March, 2005, pp. 475 - 480. [Abstract]
  Download: pdf [576 KB] copyrighted

• Active Appearance Models Revisited
  I. Matthews and S. Baker
  International Journal of Computer Vision, Vol. 60, No. 2, November, 2004, pp. 135 - 164. [Abstract]
  Download: pdf [759 KB] copyrighted

• Automatic Construction of Active Appearance Models as an Image Coding Problem
  S. Baker, I. Matthews, and J. Schneider
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 10, October, 2004, pp. 1380 - 1384. [Abstract]
  Download: pdf [212 KB] copyrighted

• Markerless Human Motion Transfer
  K.M. Cheung, S. Baker, J.K. Hodgins, and T. Kanade
  Proceedings of the 2nd International Symposium on 3D Data Processing, Visualization and Transmission, September, 2004. [Abstract]
  Download: pdf [1216 KB] copyrighted

• Face Recognition Across Pose and Illumination
  R. Gross, S. Baker, I. Matthews, and T. Kanade
  Handbook of Face Recognition, Stan Z. Li and Anil K. Jain, ed., Springer-Verlag, June, 2004. [Abstract]
  Download: pdf [617 KB] copyrighted

• Real-Time Combined 2D+3D Active Appearance Models
  J. Xiao, S. Baker, I. Matthews, and T. Kanade
  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2, June, 2004, pp. 535 - 542. [Abstract]
  Download: pdf [5306 KB] copyrighted

• The Template Update Problem
  I. Matthews, T. Ishikawa, and S. Baker
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 6, June, 2004, pp. 810 - 815. [Abstract]
  Download: pdf [340 KB] copyrighted

• Appearance-Based Face Recognition and Light-Fields
  R. Gross, I. Matthews, and S. Baker
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, No. 4, April, 2004, pp. 449 - 465. [Abstract]
  Download: pdf [775 KB] copyrighted

• Lucas-Kanade 20 Years On: A Unifying Framework
  S. Baker and I. Matthews
  International Journal of Computer Vision, Vol. 56, No. 3, March, 2004, pp. 221 - 255. [Abstract]
  Download: pdf [462 KB] copyrighted

• Textureless Layers
  Q. Ke, S. Baker, and T. Kanade
  tech. report CMU-RI-TR-04-17, Robotics Institute, Carnegie Mellon University, March, 2004. [Abstract]
  Download: pdf [412 KB] copyrighted

• The CMU Pose, Illumination, and Expression Database
  T. Sim, S. Baker, and M. Bsat
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No. 12, December, 2003, pp. 1615 - 1618. [Abstract]
  Download: pdf [369 KB] copyrighted

• Steady-State Feedback Analysis of Tele-Graffiti
  N. Takao, S. Baker, and J. Shi
  Proceedings of the IEEE International Workshop on Projector-Camera Systems, October, 2003. [Abstract]
  Download: pdf [391 KB] copyrighted

• Tele-Graffiti: A Camera-Projector Based Remote Sketching System with Hand-Based User Interface and Automatic Session Summarization
  N. Takao, J. Shi, and S. Baker
  International Journal of Computer Vision, Vol. 53, No. 2, July, 2003, pp. 115 - 133. [Abstract]
  Download: pdf [892 KB] copyrighted

• When is the Shape of a Scene Unique Given its Light-Field: A Fundamental Theorem of 3D Vision?
  S. Baker, T. Sim, and T. Kanade
  IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 25, No. 1, January, 2003, pp. 100 - 109. [Abstract]
  Download: pdf [155 KB] copyrighted

• Limits on Super-Resolution and How to Break Them
  S. Baker and T. Kanade
  IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 24, No. 9, September, 2002, pp. 1167 - 1183. [Abstract]
  Download: pdf [998 KB], ps.gz [1892 KB] copyrighted

• Single Viewpoint Catadioptric Cameras
  S. Baker and S.K. Nayar
  Panoramic Vision: Sensors, Theory, Applications, Ryad Benosman and Sing Bing Kang, ed., Springer-Verlag, 2001.
  Download: pdf [498 KB], ps.gz [4123 KB] copyrighted

• Super-Resolution: Limits and Beyond
  S. Baker and T. Kanade
  Super-Resolution Imaging, S. Chaudhuri, ed., Kluwer Academic Press, 2001.
  Download: pdf [783 KB], ps.gz [1267 KB] copyrighted

• Hallucinating Faces
  S. Baker and T. Kanade
  Fourth International Conference on Automatic Face and Gesture Recognition, March, 2000. [Abstract]
  Download: pdf [270 KB], ps.gz [322 KB] copyrighted

• A Theory of Single-Viewpoint Catadioptric Image Formation
  S. Baker and S.K. Nayar
  International Journal of Computer Vision, Vol. 35, No. 2, 1999, pp. 1 - 22. [Abstract]
  Download: pdf [1290 KB], ps.gz [568 KB] copyrighted

• Design and Evaluation of Feature Detectors
  S. Baker
  doctoral dissertation, Graduate School of Arts and Sciences, Columbia University, September, 1998.
  Download: pdf [3551 KB], ps.gz [5436 KB] copyrighted

• A Layered Approach to Stereo Reconstruction
  S. Baker, R. Szeliski, and P. Anandan
  Proceedings of the 1998 IEEE Conference on Computer Vision and Pattern Recognition, June, 1998, pp. 434 - 441. [Abstract]
  Download: pdf [434 KB], ps.gz [3272 KB] copyrighted

• Parametric Feature Detection
  S. Baker, S.K. Nayar, and H. Murase
  International Journal of Computer Vision, Vol. 27, No. 1, 1998, pp. 27 - 50.
  Download: pdf [2144 KB], ps.gz [2674 KB] copyrighted

• Pattern Rejection
  S. Baker and S.K. Nayar
  Proceedings of the 1996 IEEE Conference on Computer Vision and Pattern Recognition, June, 1996, pp. 544 - 549. [Abstract]
  Download: pdf [299 KB], ps.gz [269 KB] copyrighted