arxiv:2309.15818

Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation

Published on Sep 27, 2023

· Submitted by

akhaliq on Sep 28, 2023

#3 Paper of the day

Upvote

Authors:

David Junhao Zhang ,

Jay Zhangjie Wu ,

Jia-Wei Liu ,

Rui Zhao ,

Mike Zheng Shou

Abstract

Significant advancements have been achieved in the realm of large-scale pre-trained text-to-video Diffusion Models (VDMs). However, previous methods either rely solely on pixel-based VDMs, which come with high computational costs, or on latent-based VDMs, which often struggle with precise text-video alignment. In this paper, we are the first to propose a hybrid model, dubbed as Show-1, which marries pixel-based and latent-based VDMs for text-to-video generation. Our model first uses pixel-based VDMs to produce a low-resolution video of strong text-video correlation. After that, we propose a novel expert translation method that employs the latent-based VDMs to further upsample the low-resolution video to high resolution. Compared to latent VDMs, Show-1 can produce high-quality videos of precise text-video alignment; Compared to pixel VDMs, Show-1 is much more efficient (GPU memory usage during inference is 15G vs 72G). We also validate our model on standard video generation benchmarks. Our code and model weights are publicly available at https://github.com/showlab/Show-1.

View arXiv page View PDF Add to collection

Community

jayw

Paper author Sep 28, 2023

•

edited Sep 28, 2023

Project Page | Code | Paper

mikelabs

Sep 28, 2023

Love the project page - always great to have some video examples.

My summary: Show-1 is a hybrid model that combines pixel and latent diffusion for efficient high-quality text-to-video generation. Both of these approaches have tradeoffs, so researchers tried a hybrid approach combining both.

Highlights from the paper:

Pixel diffusion excels at low-res video generation precisely aligned with text
Latent diffusion acts as efficient upsampling expert from low to high res
Chaining the two techniques inherits benefits of both Show-1 achieves strong alignment, quality, and 15x less inference memory
The key is using pixel diffusion for the initial low-resolution stage. This retains alignment with text descriptions.
Latent diffusion then serves as a super-resolution expert, upsampling efficiently while preserving fidelity.

By blending complementary techniques, Show-1 moves past tradeoffs limiting the individual models.

More details here. Paper is here (includes links to example generations).