Abstract: Inductive biases have proven effective and often essential in the design of performant deep learning systems. This thesis presents two contributions that target distinct facets of how inductive biases can improve modern deep models. In Part 1, we focus on controlling the diversity–likelihood trade-off at inference-time of generative models. In Part 2, we study the architectural inductive bias for 3D perception for multi-view transformer models.
Part 1 presents Temporal Score Rescaling (TSR), a mechanism to steer the sampling diversity of denoising diffusion and flow matching models, allowing users to sample from a sharper or broader distribution than the training distribution. We build on the observation that these models leverage (learned) score functions of noisy data distributions for sampling, and show that rescaling these allows one to effectively control a `local‘ sampling temperature. Notably, this approach does not require any finetuning or alterations to training strategy, and can be applied to any off-the-shelf model. We validate our framework across five disparate tasks — image generation, pose estimation, depth prediction, robot manipulation, and protein design. Across these tasks, our approach allows sampling from sharper (or flatter) distributions and yields consistent performance gains.
Part 2 presents RayRoPE, where we study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can adapt to the geometry of the underlying 3D scene. We find that prior encoding schemes for multi-view attention do not meet these desiderata. RayRoPE represents patch positions based on associated rays and computes query-frame projective coordinates to ensure $SE(3)$ invariance. To adapt to scene geometry, RayRoPE predicts a per-token depth to obtain its position along the corresponding ray, while also modeling uncertainty and analytically computing the expected positional encoding. We validate our method on the tasks of novel-view synthesis and stereo depth estimation. While remaining efficient, RayRoPE consistently improves over alternate position encoding schemes.
Committee:
Prof. Shubham Tulsiani (chair)
Prof. Deva Ramanan
Prof. David Held
Hanzhe Hu
