Abstract:
Zero-shot vision-language models such as CLIP have demonstrated remarkable recognition capabilities without task-specific training. However, their text embeddings are derived from large-scale pretraining over vast domains, which can introduce spurious correlations that hurt performance on minority groups when sensitive attributes are entangled with target classes. Correcting this bias is challenging in realistic settings, where biases are numerous and overlapping, and explicit annotations of spurious attributes are unavailable.

This thesis presents SPOT (Spectral Preconditioning of Text Embeddings), a post-training calibration method that adapts CLIP’s text axes to a target domain through a single closed-form spectral transformation. SPOT estimates the covariance of unlabeled target-domain image embeddings, decomposes the text axis in the resulting eigenbasis, and applies an anisotropic shrinkage that preserves energy where class semantics concentrate while attenuating directions aligned with spurious correlations. I also present DP-SPOT, a differentially private extension that derives sensitivity bounds for the SPOT map and releases calibrated axes under formal guarantees, enabling deployment in privacy-constrained settings.

Experiments across multiple group-robustness benchmarks show that SPOT matches or approaches state-of-the-art debiasing methods on binary classification tasks and substantially outperforms them in multi-class settings. DP-SPOT achieves near-parity with its non-private counterpart under strict privacy budgets, while all competing baselines lack any privacy guarantee.

Committee:
Fernando De la Torre (advisor)
Artur Dubrawski
Yinong Wang