What Is a Diffusion Model?

A diffusion model is a specific type of generative AI model that can produce images, audio, or video, by learning to reverse the effects of translating these mediums to noise. Diffusion models are a core technology in generative AI, and are becoming common in enterprise workflows for media generation.

The unique denoising process in their training makes diffusion models more stable and capable of generating higher-quality, more diverse media compared to previous generative AI methods.

Before the rise of diffusion models, data scientists primarily used Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs).

Both of these earlier models came with engineering trade-offs. GANs pit two networks against each other to produce higher-quality outputs. This adversarial structure can make GANs unstable and suffer “mode collapse,” a scenario where the model stops learning and only produces a very narrow range of outputs. VAEs offer much more stable training, but their compression techniques typically produce less detail.

Diffusion models address the major issues of GANs and VAEs. Their training process is sequential and highly stable. These models can consistently generate outputs with a high degree of fidelity and compositional accuracy.

Diffusion models’ great improvement over earlier generative models resulted in their rapid adoption. For example, marketing agencies use diffusion models to automate media production. Pharmaceutical companies can even use them to generate novel molecular structures for drug discovery.

Diffusion models use several important neural network techniques to generate useful outputs.

Denoising network architecture

The reverse process relies on a neural network to predict the noise present in the input. Traditionally, developers used a U-Net architecture for this step. A U-Net compresses the image data for processing and then expands it back to its original resolution, using attention mechanisms to preserve specific structural details. More recently, the field has started adopting transformer-based models, such as Diffusion Transformers (DiT), because they scale more predictably as computational requirements increase.

Latent space (latent diffusion)

Diffusing high-resolution datasets is computationally expensive. Engineers can use a technique called latent diffusion for compression. Latent diffusion compresses data into a smaller area called a latent space rather than the full pixel space. The diffusion model learns to add and remove noise entirely within the latent space, which reduces memory requirements and hardware costs needed for generation.

Conditioning mechanisms

To get a diffusion model to generate anything other than purely random outputs, you need to condition it. AI systems use encoders such as CLIP (Contrastive Language-Image Pre-training) to translate written prompts, or text conditioning, into mathematical instructions that the model can follow.

Classifier-free guidance (CFG) is a technique used to make the model adhere to those text prompts. During the denoising process, CFG calculates the difference between a prediction guided by the prompt and an unguided prediction. It then steers the final output closer to the specific details requested by the user.

Models can also use image conditioning to direct the generation process. This method uses visual inputs, such as depth maps or reference illustrations, to guide the output rather than relying solely on text.

There are several distinct subtypes of diffusion models.

Denoising Diffusion Probabilistic Models (DDPMs)

DDPMs served as the basic framework for most modern diffusion models. They use the standard diffusion and denoising process described above. While this approach reliably produces detailed media, it requires the system to calculate hundreds or thousands of steps in sequence. The inference phase requires significant computing power and operates relatively slowly compared to other models.

Score-based generative models

Rather than predicting and removing the exact noise added at each discrete timestep, these models learn the "score" of the data distribution. Here, they seek to learn the gradient function toward the highest data density. By following these gradients, a score-based model can efficiently navigate from random noise back to structured data.

Latent diffusion models (LDMs)

Standard DDPMs and score-based models typically operate directly in “pixel space,” meaning they process the full resolution image at every step. That processing is computationally expensive. Latent diffusion models (LDMs) perform more efficiently by compressing the training data into a smaller, mathematical latent space. Once the latent representation is restored, a decoder expands it back into a full-resolution image. This model architecture is what makes it possible to run complex models like Stable Diffusion on consumer-grade hardware.

Read about the Stable Diffusion generative AI model »

Consistency models

Consistency models are a newer class of architecture specifically designed to solve the slow sampling speeds of traditional diffusion models. Instead of requiring a model to denoise input incrementally, consistency models are trained to mathematically map any noisy data point back to its origin. This approach allows for significantly faster inference. It can produce high-quality output in only a few steps.

Because they can generate very realistic media, diffusion models have the potential to introduce several technical and ethical challenges.

Compute and inference cost

Diffusion models denoise data iteratively. Unlike single-pass generative models that produce an output in one mathematical step, a diffusion model must run its neural network many times to generate a single piece of media. This can be resource-intensive and require specialized hardware to run at scale.

Sampling speed

Running many denoising steps can bottleneck applications. Denoising Diffusion Implicit Models (DDIMs) approximate denoising steps mathematically, which is faster. DPM-Solver is another framework used to speed sampling. Both can generate images in 10 to 20 steps, instead of hundreds or thousands. Consistency distillation is an even more refined technique to generate high-quality outputs in just one or two steps.

Controllability and conditioning quality

Generating a high-quality image is only useful if that image accurately reflects the user's prompt. Ensuring careful adherence to text prompts requires careful engineering. Developers must tune their conditioning mechanisms to prevent the model from ignoring specific instructions or hallucinating unintended elements.

Training data and bias

Diffusion models are trained on massive datasets. As such, they can unintentionally amplify biases in the underlying data. Organizations must carefully evaluate their models to ensure they produce fair outputs.

Responsible AI and content safety

Because diffusion models can generate highly convincing photorealistic media, they carry significant potential for misuse, such as the creation of deepfakes. Organizations should implement controls to filter inappropriate prompts and outputs. Data provenance standards and watermarking can also help viewers track the origin of AI-generated media.

Here is how diffusion models compare to other established generative AI architectures.

Diffusion models vs. GANs

Generative Adversarial Networks rival two neural networks against each other: a generator and a discriminator. GANs produce their outputs in a single mathematical pass. While faster than diffusion models, their competitive architecture can more easily lead to mode collapse, in which the model produces only limited variations. The step-by-step noise reduction process of diffusion models is relatively slower, but it provides highly stable training and more diverse outputs.

Read about Generative Adversarial Networks »

Diffusion models vs. VAEs

Variational Autoencoders compress data into a smaller latent space. While VAEs are computationally efficient and stable to train, their latent compression and reconstruction techniques might generate less detailed outputs. Diffusion models can produce higher-quality outputs than VAEs.

Diffusion models vs. autoregressive models

Autoregressive models, the architecture powering modern Large Language Models (LLMs), generate data sequentially, predicting the next word or pixel one at a time. For media generation, this approach is computationally inefficient and scales poorly. Diffusion models process the entire image during each denoising step. They are more effective at generating higher-quality visual, spatial, and audio data.

Read about autoregressive models »

Read about Large Language Models »

AWS offers a range of services for you to build, train, and deploy diffusion models at scale:

Amazon Bedrock is the AWS platform for building generative AI applications and agents at a production scale. You have access to hundreds of pre-trained models from leading AI companies, along with evaluation tools to pick the best model based on your unique performance and cost needs.

Amazon SageMaker is the center for all your data, analytics, and AI. Collaborate and build faster from a unified studio using familiar AWS tools for model development in SageMaker AI (including HyperPod, JumpStart, and MLOps), generative AI, data processing, and SQL analytics, accelerated by Amazon Q Developer, the most capable generative AI assistant for software development.

Amazon EC2 Trn2 instances, powered by 16 AWS Trainium2 chips, are purpose-built for generative AI and offer high-performance EC2 instances for training and deploying models with hundreds of billions to trillion+ parameters.

Get started with diffusion models on AWS by creating a free account today.