Visual-Tactile Synthesis for Texture Generation - Robotics Institute Carnegie Mellon University
Loading Events

PhD Thesis Proposal

November

7
Fri
Ruihan Gao PhD Student Robotics Institute,
Carnegie Mellon University
Friday, November 7
3:00 pm to 4:30 pm
Newell-Simon Hall 3305
Visual-Tactile Synthesis for Texture Generation
Abstract: Recent advances in generative models have enabled the creation of highly realistic visual content, yet they remain limited to visual perception alone. In contrast, human interaction with the physical world is inherently multimodal — we not only see textures but also feel them. This gap motivates the goal of my thesis: to build generative models that jointly synthesize visual and tactile modalities for material and texture generation. By unifying what we see and what we touch, such models can drive new forms of physically grounded content creation, from robotics and virtual reality to material design.

However, extending generative modeling to touch presents unique challenges: tactile data is scarce, noisy, and expensive to collect, and there is no large-scale paired dataset linking visual appearance with tactile response. To address these challenges, my research explores three synergistic directions.

Part I: I introduce controllable visual-tactile synthesis models that jointly generate aligned visual and tactile textures from shared latent representations, enabling explicit control over appearance and feel.

Part II: I propose tactile-aware 3D generation frameworks that integrate tactile sensing into 3D diffusion pipelines, allowing models to infer physically grounded material properties from visual cues and geometry.

Part III: Building on these insights, I aim to develop scalable multimodal generation systems that leverage large vision and language foundation models and physics priors to synthesize novel materials directly from text or image input, without relying on extensive paired tactile data.

 
Thesis Committee:
Jun-Yan Zhu (Co-chair)
Wenzhen Yuan (Co-chair)
Shubham Tulsiani
Andrew Owens (Cornell Tech)