Abstract: For intelligent robots to become ubiquitous, we need to “solve” locomotion, navigation and manipulation at sufficient reliability in widely varying environments. In locomotion, we now have demonstrations of humanoid walking in a variety of challenging environments.  In navigation, we pursued the task of “Go to Any Thing” – a robot, on entering  a newly rented Airbnb, should be able to find objects such as TV sets or potted plants. The biggest challenges in robotics today lie in manipulation, particularly in dexterous manipulation with multi-fingered hands. Learning approaches have been responsible for recent advances, but they are held up by the lack of “big data” at the scale available in language and vision. I argue that this shortage can be circumvented by taking inspiration from how humans  acquire motor skills in childhood. For dexterous manipulation, multimodal perception is key – vision, touch and proprioception. In my view, visual imitation should be based on 3D/4D reconstruction – then a physics simulator provides a pre-trained world model. The core technology for reconstruction of human bodies, hands, and objects now exists with systems like HMR, HaMeR and SAM 3D.  Visual imitation, while essential, is not sufficient, as policies need to consider contact forces as well. RL in simulation and sim-to-real have been workhorse technologies for us, assisted by a few technical innovations. I will sketch promising directions for future work.

Bio: Jitendra Malik is Arthur J. Chick Professor of EECS at UC Berkeley, and VP and Distinguished Scientist at Amazon. His group has conducted research on many different topics in computer vision, computer graphics, machine learning and robotics resulting in concepts such as anisotropic diffusion, high dynamic range imaging, normalized cuts, R-CNN and rapid motor adaptation. His publications have received twelve best paper awards, including six test of time awards – the Longuet-Higgins Prize for papers published at CVPR (three times) and the Helmholtz Prize for papers published at ICCV (three times). He has mentored more than 80 PhD students and postdoctoral fellows.

Jitendra received the 2016 ACM/AAAI Allen Newell Award, 2018 IJCAI Award for Research Excellence in AI, and the 2019 IEEE Computer Society’s Computer Pioneer Award for “leading role in developing Computer Vision into a thriving discipline through pioneering research, leadership, and mentorship”. He is a member of the US National Academy of Sciences, the National Academy of Engineering and Fellow, American Academy of Arts and Sciences.