Reliable state and geometry estimation from limited observations is a fundamental challenge in robotics and perception. Observations are often noisy, partial, or ambiguous, making estimation ill-posed without additional structure. This thesis argues that robust estimation in these regimes is enabled by integrating structured knowledge into the inference process.

Estimation can be viewed as inferring latent state or geometry by combining three complementary forms of structure: constraints that restrict the feasible solution space, physical models that describe how observations are generated, and priors that regularize underdetermined solutions.

First, constraints provide structure in the solution space. We develop methods for imposing hard constraints in real-time estimation through incremental optimization, and for learning well-conditioned measurement noise models that shape the optimization landscape. We also study constraints learned from demonstrations and analyze when they can serve as general state constraints.

Second, physical models provide structure in the observation process. We study settings where sensing physics fundamentally limits observability, and show how incorporating physics-based forward models into neural rendering enables accurate 3D reconstruction and resolves ambiguities in acoustic and multimodal sensing.

Finally, priors provide structure when estimation remains underdetermined even after modeling constraints and physics. We demonstrate how priors learned from data-rich visual domains can be transferred to data-scarce sensing modalities, enabling pose-free acoustic 3D reconstruction and tactile 3D reconstruction from sparse robot touches.

Together, these contributions provide a unified perspective on estimation as the integration of constraints, physical models, and priors.

Thesis Committee Members: