Abstract:

AI-generated content is inherently derived from training data, yet it remains a mystery which specific data points large generative models rely on for a given generation. To address this, my research focuses on training data attribution—identifying the training images that are most influential in synthesizing a specific output. The ideal objective is to find the exact subset of training data that, if removed, would prevent a retrained model from generating that content. However, the required combinatorial search and iterative retraining are computationally intractable.

To make this search tractable, we propose using model unlearning as a proxy for the counterfactual model. By forcing a model to unlearn a synthesized output, we can trace which training data points it effectively “removes” by observing changes in the training loss. We show that this is a highly effective method for finding influential data in text-to-image models.

Next, we interpret these attribution results by identifying which features best predict them. We show that joint text-image features, after light calibration, serve as strong predictors of attribution. We then analyze whether textual or visual features are more salient in the attribution process. We find that this salience shifts drastically across different settings.

To enable more fine-grained analysis of the attribution results, we introduce TPIPS, a similarity metric conditioned on specific visual aspects. We repurpose a vision-language model to compute text-conditioned, aspect-specific perceptual similarity grounded in human judgments.

Based on the analysis, I will conclude by discussing the next open challenges and potential next steps in the field.

Thesis Committee Members:

Jun-Yan Zhu, Chair

Deva Ramanan

Ruslan Salakhutdinov

Alexei A. Efros, UC Berkeley

David Bau, Northeastern