BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Robotics Institute Carnegie Mellon University - ECPv6.15.12.1//NONSGML v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Robotics Institute Carnegie Mellon University
X-ORIGINAL-URL:https://www.ri.cmu.edu
X-WR-CALDESC:Events for Robotics Institute Carnegie Mellon University
REFRESH-INTERVAL;VALUE=DURATION:PT1H
X-Robots-Tag:noindex
X-PUBLISHED-TTL:PT1H
BEGIN:VTIMEZONE
TZID:America/New_York
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20250309T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20251102T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20260308T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20261101T060000
END:STANDARD
BEGIN:DAYLIGHT
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
DTSTART:20270314T070000
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0400
TZOFFSETTO:-0500
TZNAME:EST
DTSTART:20271107T060000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTART;TZID=America/New_York:20260720T153000
DTEND;TZID=America/New_York:20260720T163000
DTSTAMP:20260911T194514
CREATED:20260714T141829Z
LAST-MODIFIED:20260720T165726Z
UID:151792-1784561400-1784565000@www.ri.cmu.edu
SUMMARY:Cutting the Skip: Training Residual-Free Transformers
DESCRIPTION:Abstract:   Transformers are ubiquitous. They influence nearly every aspect of modern AI. However\, the mechanics of their training remain poorly understood. This poses a problem for the field due to the immense amounts of data\, computational power\, and energy being invested in the training of these networks. I highlight a recent intriguing empirical result from our group. Specifically\, although self-attention catastrophically fails to train without a skip connection under standard conditions\, deep transformers can in fact be trained successfully without them. In this talk\, I explore what makes this possible and what it reveals about the fundamental training dynamics of modern transformers. I also speculate on why truly deep networks may be important for improving generalization and efficiency. \n \nBio:  Simon Lucey Ph.D. is the Director of the Australian Institute for Machine Learning (AIML) and a professor in the School of Computer Science\, at Adelaide University. He is also Director of the CommBank Foundational AI Research Centre. Prior to this he was an associate research professor at Carnegie Mellon University’s Robotics Institute (RI) in Pittsburgh USA; where he spent over 10 years as an academic. He was also Principal Research Scientist at the autonomous vehicle company Argo AI from 2017-2022. He has received various career awards\, notably the AmCham AI Scientist of the year in 2024. He was also a member of the Australian Government’s AI Expert Group\, and their National Robotics Strategy committee. Simon’s research interests span AI\, machine learning\, computer vision and robotics. \nSponsor:\nThe VASC seminar is generously sponsored by HeyGen\, an all-in-one AI-powered video generation platform that leverages advances in computer vision\, generative modeling\, and multimodal learning to make high-quality video creation both scalable and accessible.
URL:https://www.ri.cmu.edu/event/cutting-the-skip-training-residual-free-transformers/
LOCATION:Newell-Simon Hall 4305
CATEGORIES:Seminar,VASC Seminar
ATTACH;FMTTYPE=image/jpeg:https://www.ri.cmu.edu/app/uploads/2026/07/7-20-26-4.jpg
END:VEVENT
END:VCALENDAR