Guide
How diffusion models generate images and video
From DDPM to latent diffusion to diffusion transformers: the three papers that explain how modern AI image and video generators work.
The short answer
A diffusion model learns to undo noise. Training corrupts real images in many small steps and teaches a network to reverse each step; generation starts from pure noise and runs the reversal until an image appears. Later work changed where it runs (a compressed latent space), what runs it (transformers) and what steers it (text).
Most AI image and video tools you can buy today are descendants of one idea from 2020. Understanding it helps you judge what a vendor means by "our model" and which limits come with it. This guide follows three papers that explain it.
What is a diffusion model?
A diffusion model learns to undo noise. Training corrupts real images with noise in many small steps and teaches a network to reverse each step. Generation starts from pure noise and runs the learned reversal until an image appears. Everything since 2020 refines that loop: where it runs (pixels or a compressed latent space), what network runs it (a U-Net or a transformer), and what steers it (text, boxes or other conditioning).
How did DDPM make diffusion work?
Ho et al. define a diffusion model as a parameterized Markov chain, trained with variational inference, that produces samples matching the data after a finite number of steps 1. The idea predates the paper, but earlier versions had not shown they could produce high-quality samples. DDPM showed they could, with an unconditional CIFAR10 FID of 3.17 and LSUN 256x256 samples comparable to ProgressiveGAN 1.
- A specific training parameterization. The paper trains on a weighted variational bound and shows this is equivalent to denoising score matching over multiple noise levels in training, and to annealed Langevin dynamics in sampling 1.
- Many small steps. Generation is a long sequential chain of denoising steps, which is why pixel-space sampling is slow.
Mental model: the forward process is fixed and destroys the image; the network only has to learn the reverse.
How did latent diffusion make it affordable?
Pixel-space diffusion is expensive: optimizing powerful models can take hundreds of GPU-days, and inference needs many sequential evaluations. Rombach et al. move the process into the latent space of a powerful pretrained autoencoder, which they report reaches a near-optimal balance between complexity reduction and detail preservation 2.
They also add cross-attention layers so the network can be conditioned on general inputs such as text or bounding boxes 2. A compressed latent space plus flexible conditioning is the design behind Stable Diffusion-style text-to-image systems, and it cut compute substantially compared with pixel-based diffusion models 2.
What did diffusion transformers change?
Peebles and Xie keep the latent diffusion setup but replace the U-Net backbone with a transformer operating on latent patches. They analyze scalability by forward-pass compute (Gflops) and find that models with more Gflops, from more depth or width or from more input tokens, consistently reach lower FID 3. Their largest model, DiT-XL/2, reached an FID of 2.27 on class-conditional ImageNet 256x256, state of the art at publication 3.
The takeaway is architectural: transformers give diffusion a predictable scaling recipe, which is why they became a common backbone for large generators.
How does this extend to video?
The papers above are about images. Video generators extend the same denoising idea across time, so the network must keep frames consistent as well as realistic. Treat any claim about a specific current video model as time-sensitive and check the vendor's technical report.
What did each paper contribute?
| Paper | Contribution | Problem it addressed |
|---|---|---|
| DDPM 1 | Showed diffusion gives high-quality samples | Earlier diffusion models were not competitive |
| Latent diffusion 2 | Runs diffusion in an autoencoder's latent space and adds cross-attention conditioning | The cost of pixel-space diffusion |
| DiT 3 | Transformer backbone with clear compute scaling | The U-Net as an inherited default |
What does this mean when you choose an image or video tool?
The architecture explains the trade-offs you will test: many denoising steps cost time, conditioning decides how well you can direct the output, and results vary from run to run. Use the image and video category criteria (control and consistency, output quality, usage rights and provenance, production workflow) and run each tool on the same brief several times. See how to read an evaluation for what a trustworthy result looks like, and the glossary for terms such as diffusion model, latent space and FID.
Evidence in this article Third-party research Work published by researchers or institutions other than us or the vendor. We link it; we have not reproduced it.
Frequently asked questions
What is a diffusion model?
A generative model trained to reverse a gradual noising process. It starts from random noise and removes it step by step, guided by what it learned from real images, until a new image appears.
Why is latent diffusion cheaper than pixel-space diffusion?
It runs the denoising process in the compressed latent space of a pretrained autoencoder instead of on full-resolution pixels. Rombach et al. report that this cuts compute substantially while keeping detail.
What did diffusion transformers change?
Peebles and Xie replaced the U-Net backbone with a transformer working on latent patches. They found that more forward-pass compute consistently gave lower FID, which gave diffusion a predictable scaling recipe.
Do these papers explain current video generators?
Only the shared idea. They cover images. Video models extend denoising across time and must keep frames consistent, and claims about any specific video model are time-sensitive and belong in the vendor's technical report.
Sources
- Denoising Diffusion Probabilistic Models Ho, Jain, Abbeel. NeurIPS 2020. Third-party research paper, as supplied. The figures quoted here are from its abstract and were not re-checked by the site build.
- High-Resolution Image Synthesis with Latent Diffusion Models Rombach, Blattmann, Lorenz, Esser, Ommer. CVPR 2022. Third-party research paper, as supplied. The figures quoted here are from its abstract and were not re-checked by the site build.
- Scalable Diffusion Models with Transformers Peebles, Xie. 2022. Third-party research paper, as supplied. The figures quoted here are from its abstract and were not re-checked by the site build.