Abstract: 

Generative models have made rapid advancements in recent years, and text-conditioned diffusion models have become the standard paradigm for image and video synthesis.

However, controlling the output purely through text is not the ideal medium for many practical applications. In my thesis research, I have focused on making diffusion models more controllable and efficient beyond text-only interaction. To this end, I explore two complementary directions.

Part A:

I study methods for generating more expressive and diverse outputs.

In my first project, I explore how users can specify desired output images through visual prompts rather than text.

This enables compositional image generation where object identities and appearances are controlled through reference images.

In my second project, I address the issue of redundant generations when the task involves generating a group of images from the same text prompt.

Part B:

I study efficient, structure preserving translations with diffusion models.

In my first project, I explore zero-shot image editing method that shows how well-trained text-to-image models can be repurposed for editing real images.

However, such zero-shot methods are slow and struggle when the base text-to-image model is not trained for the target domain.

In my next project, I explore how we can fine-tune text-to-image models to perform image translation in both paired and unpaired settings.

In my final project, I will show how we can extend this to video-to-video translation, and the unique challenges involved.

Thesis Committee:

Jun-Yan Zhu (Co-Chair)

Srinivasa Narasimhan (Co-Chair)

Shubham Tulsiani

Daniel Cohen-Or (Tel Aviv University)