Abstract:   Transformers are ubiquitous. They influence nearly every aspect of modern AI. However, the mechanics of their training remain poorly understood. This poses a problem for the field due to the immense amounts of data, computational power, and energy being invested in the training of these networks. I highlight a recent intriguing empirical result from our group. Specifically, although self-attention catastrophically fails to train without a skip connection under standard conditions, deep transformers can in fact be trained successfully without them. In this talk, I explore what makes this possible and what it reveals about the fundamental training dynamics of modern transformers. I also speculate on why truly deep networks may be important for improving generalization and efficiency.

Bio:  Simon Lucey Ph.D. is the Director of the Australian Institute for Machine Learning (AIML) and a professor in the School of Computer Science, at Adelaide University. He is also Director of the CommBank Foundational AI Research Centre. Prior to this he was an associate research professor at Carnegie Mellon University’s Robotics Institute (RI) in Pittsburgh USA; where he spent over 10 years as an academic. He was also Principal Research Scientist at the autonomous vehicle company Argo AI from 2017-2022. He has received various career awards, notably the AmCham AI Scientist of the year in 2024. He was also a member of the Australian Government’s AI Expert Group, and their National Robotics Strategy committee. Simon’s research interests span AI, machine learning, computer vision and robotics.

The VASC seminar is generously sponsored by HeyGen, an all-in-one AI-powered video generation platform that leverages advances in computer vision, generative modeling, and multimodal learning to make high-quality video creation both scalable and accessible.