Diffusion Transformer (DiT) Explained: Architecture, adaLN and Scaling

For most of 2021 and 2022, “diffusion model” meant “U-Net with attention sprinkled in.” Then a paper from William Peebles and Saining Xie asked a plain question: does the convolutional U-Net carry any special inductive bias that diffusion needs? Their answer, published as Scalable Diffusion Models with Transformers, was no. A plain transformer over latent patches, conditioned through a clever normalization trick, matched and then beat every U-Net on class-conditional ImageNet. That design, the diffusion transformer, now sits underneath the most capable image and video generators, from Stable Diffusion 3 to Wan 2.1 and the architecture OpenAI describes for Sora.

This matters now because the architecture decides your costs. A diffusion transformer turns generation into a token-sequence problem, so everything you know about transformer scaling, attention cost, and sequence length applies, including the parts that hurt. A five-second video is tens of thousands of tokens, and quadratic attention dominates.

You will leave with a precise picture of the DiT block, why adaLN-Zero works, how flow matching and MM-DiT changed training, and worked numbers for what video does to compute.

What this covers: the pipeline from pixels to tokens, the DiT block and adaLN-Zero conditioning, flow matching and rectified flow, MM-DiT text conditioning, video token arithmetic, failure modes, and a practical checklist.

Context and Background

Latent diffusion, introduced with Stable Diffusion and the LDM paper, moved the denoising process out of pixel space. A variational autoencoder (VAE) compresses an image by a fixed spatial factor, the diffusion model works on that compact latent, and the decoder returns pixels. DiT inherits this design wholesale. In the paper, the off-the-shelf Stable Diffusion VAE downsamples 8x, so a 256x256x3 image becomes a 32x32x4 latent. Every diffusion transformer you meet in 2026 is a latent model; pixel-space transformers exist but are the exception.

Before DiT, the denoiser was a U-Net: a convolutional encoder-decoder with skip connections, ResNet blocks, and attention at a few low resolutions. U-Nets are efficient at small scale and carry a locality bias that helps early in training. Their weakness is that scaling recipes were never as clean as for transformers. Width, depth, resolution stages, and attention placement all interact, and each lab tuned its own.

The DiT paper (arXiv 2212.09748, first submitted in December 2022) showed that scaling the transformer by Gflops, whether through depth, width, or more input tokens, produced a smooth drop in FID. Its largest model, DiT-XL/2, reached an FID of 2.27 on ImageNet 256×256 and 3.04 on 512×512 with classifier-free guidance at scale 1.5, beating the prior best of 3.60 (LDM-4-G) and 3.85 (ADM-G, ADM-U) respectively. Those figures come from the paper itself. The compute story is just as important: at 118.6 Gflops per forward pass, DiT-XL/2 is far cheaper than ADM at 1,120 Gflops on the same 256×256 task, and comparable to LDM-4 at 103.6.

Two further ideas completed the modern stack. Flow matching (Lipman et al., arXiv 2210.02747) gave a simulation-free way to train continuous generative models along straight or Gaussian probability paths, and rectified flow applied it at scale. Stable Diffusion 3 (Esser et al., arXiv 2403.03206) combined rectified flow with a transformer that keeps separate weights for image and text tokens. Meanwhile OpenAI’s Sora report described video as spacetime patches fed to a diffusion transformer. If you want the neighbouring pieces of the 2026 generative stack, our posts on flow-matching text-to-speech architectures and on native sparse attention for long-context training cover the same flow-matching and attention-cost ideas from other angles. The second matters a lot for video, as we will see.

My thesis is simple. The diffusion transformer won not because attention is magic at denoising, but because it converted a bespoke architecture into a standard sequence-modelling problem, and in doing so imported two decades of transformer engineering: tensor parallelism, FlashAttention, FP8, KV-style caching tricks, and scaling laws. The costs of that bargain are the same costs, namely sequence length and attention.

The Reference Architecture: From Pixels to Tokens and Back

Figure 1: The diffusion transformer pipeline. Pixels are compressed by a VAE, cut into patches, processed by a stack of conditioned transformer blocks, and decoded back to pixels.

Figure 1 shows the data path. A frozen VAE encoder maps the image or video into a latent grid. The latent is cut into non-overlapping patches, each patch is linearly projected into a token of the model’s hidden width, and positional information is added. A stack of transformer blocks, each modulated by the timestep and condition, predicts a per-token output. A final linear layer maps tokens back to patch-shaped noise or velocity predictions, which are reassembled into a latent the same shape as the input. The VAE decoder turns the final clean latent into pixels.

Direct answer. A diffusion transformer is a denoising network built from standard transformer blocks that operate on a sequence of latent patch tokens, with the diffusion timestep and conditioning injected through adaptive layer normalization instead of a convolutional U-Net’s skip-connected stages. It predicts noise, clean data, or a velocity for every token, and scales predictably with compute.

Patchify: the only hyperparameter that changes compute without changing parameters

The patch size p sets how many latent pixels collapse into one token. For a latent of size I x I, the sequence has (I/p)² tokens. With the 32×32 latent of a 256×256 image, p=2 yields 256 tokens, p=4 yields 64, and p=8 yields 16. This is the “/2” in DiT-XL/2.

The DiT paper tested patch sizes 8, 4, and 2 and noted a counterintuitive property: parameter counts barely change, while Gflops climb sharply because halving p quadruples the token count. For DiT-XL, the paper’s tables list 29.1 Gflops at p=4 on a 32×32 latent and 118.64 at p=2, a ratio of about 4.1, slightly above 4 because attention is quadratic. That tells you something deep. In a diffusion transformer, quality is governed by total forward-pass compute, and the model can spend that compute on width and depth or on finer tokenization. Smaller patches buy resolution of detail at the price of tokens.

At 512×512 the same model sees a 64×64 latent, which at p=2 is 1,024 tokens. The reported cost is 524.6 Gflops versus 118.6 at 256, a ratio of 4.4 for 4x the tokens, again reflecting the attention term growing faster than linear.

The block, in one paragraph

Each DiT block looks like a standard pre-norm transformer layer: LayerNorm, multi-head self-attention, residual add, LayerNorm, MLP, residual add. What differs is that the LayerNorms have no learned affine parameters of their own. Instead, a small network regresses scale and shift (gamma, beta) for each norm, and a gate (alpha) for each residual branch, from the conditioning vector. We dig into this in the next section because it is the single most copied design decision in the paper.

The paper’s four configurations are worth memorizing, since later models reuse the shape-names. DiT-S has 12 layers, hidden size 384, and 6 heads. DiT-B has 12 layers, 768, and 12 heads. DiT-L has 24 layers, 1024, and 16 heads. DiT-XL has 28 layers, 1152, and 16 heads. Those are lifted directly from Table 1 of the paper.

Why latent space is part of the architecture

It is tempting to treat the VAE as plumbing. It is not. The latent channel count, spatial compression ratio, and the VAE’s reconstruction quality set a ceiling on what any denoiser can achieve. DiT used 4 latent channels at 8x downsampling. Later work moved to more latent channels for fidelity, and video models added temporal compression through 3D causal VAEs; the Wan 2.1 README describes its Wan-VAE as a 3D causal video autoencoder that can encode and decode long 1080p video while preserving temporal history. When a new model claims “better text rendering” or “sharper faces,” part of the gain frequently sits in the autoencoder, not the transformer.

A causal temporal design matters for a second reason: it lets the same encoder treat an image as a one-frame video, which is how a single model can be trained on both. The Sora report makes the same point, describing images as single-frame videos in a unified patch representation.

Positional information

Original DiT used fixed sine-cosine 2D positional embeddings added to tokens. This is simple, but it ties the model to a trained grid. Production systems since then have used rotary position embeddings or other schemes that extrapolate to new aspect ratios, because the training data is no longer fixed-size ImageNet crops. The SD3 paper lists positional encodings for varying aspect ratios and resolution-dependent timestep shifting as part of its high-resolution finetuning section. I could only read the section headings in this run, not the details, so treat specifics as something to check in the paper before you implement.

adaLN-Zero: Conditioning Without Cross-Attention

Figure 2: adaLN-Zero conditioning. The conditioning vector produces scale, shift, and gate parameters for both the attention and MLP sub-layers. The gates start at zero, so every block begins as an identity function.

The DiT authors compared three ways to inject the timestep and class label. In-context conditioning appends the embeddings as extra tokens and lets attention sort it out; the paper reports 119.4 Gflops for DiT-XL/2. Cross-attention adds a cross-attention layer over a short conditioning sequence, at about 15% overhead and 137.6 Gflops. Adaptive layer norm (adaLN) regresses scale and shift from the condition and is the cheapest at 118.6 Gflops. The fourth variant, adaLN-Zero, adds a zero-initialized gate on each residual branch and gave the best FID in their experiments.

The mechanics

Let c be the sum of the timestep embedding and the class embedding. A small MLP, SiLU followed by one linear layer, maps c to six vectors per block: gamma1, beta1, alpha1 for the attention sub-layer, and gamma2, beta2, alpha2 for the MLP. The block computes:

h = x + alpha1 * Attn( LN(x) * (1 + gamma1) + beta1 )
y = h + alpha2 * MLP ( LN(h) * (1 + gamma2) + beta2 )

The LayerNorm is applied without learnable affine terms; the condition supplies the modulation. This is conceptually close to FiLM and to the style modulation in StyleGAN, but applied to every transformer block.

Why zero initialization helps

In adaLN-Zero the final linear layer producing the alpha values is initialized so that alpha equals zero everywhere. At step zero, each block’s residual branch is multiplied by zero, so the whole network is the identity function: the output equals the input. Training starts from a trivially stable function and each block gradually learns how much to contribute. This follows the same logic as zero-initializing the last batch-norm gamma in each residual block of ResNets, and it matters more as depth grows. A 28-layer model with random residual branches can have large early activations; zero gates remove that failure mode. The paper reports this variant clearly beating plain adaLN and cross-attention in FID, with compute at the adaLN level.

A worked parameter count

Here is a derived calculation, not a number from the paper. For hidden size d=1152 (DiT-XL), a standard transformer block has about 12d² parameters: 4d² in the attention projections and 8d² in an MLP with 4x expansion. That is 12 x 1,152² = 15.9 million. The adaLN modulation network maps d to 6d, which is 6d² = 7.96 million parameters per block. Over 28 blocks, the modulation network alone holds about 223 million parameters, and the full stack comes to roughly 669 million (excluding embeddings and the output layer), which is consistent with the roughly 675 million I recall the paper reporting for DiT-XL/2 (I did not re-verify that specific figure this run).

The surprise: the conditioning pathway is a third of the model’s weights in each block, yet it adds almost no Gflops because it runs once on a single vector per sample, not once per token. That asymmetry is exactly why adaLN is compute-cheap and parameter-heavy. Several later models notice this and share modulation weights across blocks or reduce the modulation MLP to cut parameters; this is a design lever worth knowing about if you are memory-bound rather than FLOP-bound.

Limits of adaLN for text

adaLN works beautifully for a small global vector such as a timestep or a class label. It does not scale to rich text. A prompt is a sequence of tokens whose individual words must bind to particular regions of the image. Compressing that to one vector throws away the structure. This is why text-to-image models either add cross-attention (as Wan 2.1 does, where the README states a T5 encoder feeds text into each block through cross-attention) or switch to joint attention, which is the MM-DiT idea in a later section. The common pattern in 2026 is a hybrid: adaLN carries the timestep and any pooled global vector, while attention carries the sequence-level text.

Flow Matching, Rectified Flow and MM-DiT

Figure 3: MM-DiT. Text and image tokens keep separate projection and MLP weights but attend jointly over the concatenated sequence, so information flows in both directions.

From noise prediction to velocity prediction

The original DiT used the DDPM-style objective: the network predicts the noise added to a latent, with a 1,000-step linear variance schedule from 1e-4 to 2e-2 (the ADM settings the paper reuses). That formulation is inherited from the U-Net era. It works, but its sampling trajectories are curved, and few-step sampling requires distillation or special solvers.

Flow matching reframes the problem. Lipman and colleagues train a continuous normalizing flow by regressing a vector field along a fixed conditional probability path, which is a simulation-free procedure: no ODE has to be solved during training. They show that Gaussian paths include diffusion as a special case, and that optimal-transport-style paths are an alternative. Rectified flow, the variant used in Stable Diffusion 3, draws the path as a straight line between data x0 and noise epsilon: z_t = (1 – t) x0 + t epsilon. The regression target is the constant velocity (epsilon – x0).

At sampling time you integrate the learned velocity field from t=1 (pure noise) to t=0 (data) with an ODE solver. An Euler step is just:

z_{t - dt} = z_t - dt * v_theta(z_t, t, prompt)

Straight paths mean fewer discretization errors per step, which is the basic reason flow-matching transformers often produce good images in tens of steps rather than hundreds. The SD3 paper adds a training refinement: because the velocity target is hardest to predict at intermediate noise levels, it replaces the uniform timestep distribution with a non-uniform density that emphasizes the middle of the trajectory. The abstract describes this as biasing noise sampling toward perceptually relevant scales; the paper’s logit-normal sampler is the named instance, and I only confirmed its name in this run, not the parameter values.

MM-DiT: letting text and image be equals

Stable Diffusion 3 introduced what is commonly called MM-DiT, for multimodal diffusion transformer. The abstract states the design: separate weights for the two modalities, with bidirectional information flow between image and text tokens. Concretely, text tokens and image tokens each get their own query, key, value projections and their own MLPs. The attention operation then runs once over the concatenated sequence. Every image token can attend to every text token and vice versa, and the text tokens are updated at every layer instead of staying frozen as in cross-attention.

The authors link this design to better text comprehension, typography, and human-preference ratings, and they report that validation loss follows a smooth scaling trend which tracks results on GenEval, T2I-CompBench, and human evaluations. Their largest models reportedly outperform SDXL, SDXL-Turbo, PixArt-alpha, and DALL-E 3 on prompt following in their evaluations. These are the authors’ own comparisons; independent reproductions vary by prompt set.

Why the extra complexity? In cross-attention, text acts as a static memory the image queries. The image can read the text but the text representation never learns from the image. In joint attention, a word like “left” can be refined by seeing where the subject currently is. Typography benefits particularly, since glyph shapes require tight token-to-token alignment.

The cost is sequence length. Joint attention runs over image tokens plus text tokens. With 4,096 image tokens (a 1024×1024 image at 8x VAE and p=2 gives 128×128 latent and 64×64 = 4,096 tokens) and perhaps 333 text tokens, the text adds about 8% to the sequence, which is cheap. The doubled projection and MLP weights, however, nearly double the parameter count compared to a single-stream block of the same width.

The 2026 design space

Putting it together, these are the families you will meet:

Family	Text conditioning	Time conditioning	Objective	Example in this post
Original DiT	class embedding via adaLN-Zero	adaLN-Zero	noise prediction, 1000-step schedule	DiT-XL/2
MM-DiT	joint attention, separate weights	adaLN-style modulation	rectified flow	Stable Diffusion 3
Cross-attention DiT	T5 via cross-attention in each block	modulation	flow matching	Wan 2.1
Spacetime-patch DiT	text conditioning	not fully disclosed	diffusion, predicts clean patches	Sora (per OpenAI report)

For SD3 the timestep conditioning detail is something I did not fully confirm in the excerpt I could read, so I describe it as adaLN-style on the basis of the DiT lineage; check the paper’s architecture figure before relying on it.

Video Diffusion Transformers: Where the Tokens Go

Figure 4: Sampling a video diffusion transformer. Each step needs a conditional and an unconditional forward pass for classifier-free guidance, then an ODE update; the VAE decodes the final latent.

A video diffusion transformer is the same recipe with one more axis. The OpenAI report states the representation: videos are compressed into a lower-dimensional latent space and then split into spacetime patches that serve as transformer tokens. The model receives noisy patches plus conditioning and is trained to predict the original clean patches. The report also says Sora trains on data at native resolution and duration rather than cropping, which let it sample widescreen 1920×1080, vertical 1080×1920, and sizes between, and says sample quality improves markedly with training compute.

The open Wan 2.1 family gives concrete public numbers. Its repository lists 1.3B and 14B parameter variants, a 3D causal Wan-VAE, a flow-matching framework in a diffusion transformer, a multilingual T5 text encoder injected through cross-attention, Apache 2.0 licensing, release on February 25, 2025, and a stated 8.19 GB VRAM requirement for the T2V-1.3B model. The 1.3B model has 30 layers and 12 heads, and the 14B has 40 layers and 40 heads.

A worked token budget (illustrative)

The following arithmetic is derived, with assumptions stated. Suppose we generate a clip of 81 frames at 720×1280. Assume a VAE with 4x temporal and 8x spatial compression, so frames become (81 – 1)/4 + 1 = 21 latent frames and the spatial grid is 90 x 160. Assume a 1x2x2 spacetime patch, giving 21 x 45 x 80 = 75,600 tokens. The compression ratio and patch size here are assumptions for illustration; check the specific model’s config.

Self-attention scores form an N x N matrix with N = 75,600, so about 5.7 billion entries per head per layer. Take a hidden width of 5,120 and 40 layers (a size in the range of the 14B class; the width is my assumption, not a verified spec). The attention matmuls cost roughly 4 N² d FLOPs per layer, which is 1.17e14, or 117 TFLOPs, and 4.7 PFLOPs over 40 layers. The dense layers cost about 2 x parameters x tokens = 2 x 14e9 x 75,600, which is 2.1 PFLOPs. So attention accounts for roughly 69% of one forward pass, and one pass is about 6.8 PFLOPs.

Now sample: with 50 steps and classifier-free guidance, each step needs two passes, so 100 passes, about 680 PFLOPs, or 0.68 exaFLOPs for one clip. On an accelerator sustaining 400 TFLOPs effective, that is roughly 1,700 seconds, around 28 minutes, ignoring VAE decode and assuming perfect utilization. Real systems divide this across GPUs with sequence parallelism, and use fewer steps, guidance distillation, or caching. The point is the shape of the cost: double the clip length and the attention term quadruples.

Contrast with the ImageNet 256 case: 256 tokens and 118.6 Gflops. The video clip has about 295 times more tokens, and its forward pass is tens of thousands of times more expensive. This is why video research concentrates on three levers: stronger VAE compression, so N falls; sparse or windowed attention, so the N² term shrinks; and few-step distillation, so the number of passes falls. Our native sparse attention post works through the sparse-attention lever for language models; the same blocks apply to spacetime tokens.

Classifier-free guidance and its price

Figure 4 shows the loop. Classifier-free guidance combines a conditional and an unconditional velocity: v = v_uncond + s (v_cond – v_uncond), where s is the guidance scale. In DiT, going from no guidance to cfg=1.5 moved ImageNet 256 FID from 9.62 to 2.27, a dramatic gain, at the price of doubling the forward passes (or batching both in one larger batch). Larger scales tend to oversaturate and reduce diversity. Guidance distillation folds the effect into the weights for one pass per step, and is common in distilled production models.

Temporal consistency

A video DiT has no explicit physics. It learns correlations across spacetime tokens. OpenAI’s report is candid about the result: the model does not accurately model many basic interactions such as glass shattering, state changes like eating are not always rendered correctly, long samples can develop incoherencies, and objects can appear spontaneously. Long-range consistency holds often, though not always. Treat that as the realistic status of the entire class, not a Sora quirk.

Operational Notes: Training and Serving a DiT

Training recipe from the original paper

The DiT paper’s recipe was strikingly plain. AdamW with default betas, a constant learning rate of 1e-4, no weight decay, no warmup, batch size 256, horizontal flips as the only augmentation, and an exponential moving average of weights with decay 0.9999, with all reported results from the EMA model. DiT-XL/2 trained for 7 million steps at 256×256 and 3 million at 512×512, on TPU v3 pods at about 5.7 iterations per second on a v3-256. The simplicity is the message: transformers did not require the careful per-architecture tuning that U-Nets had accumulated.

Modern large runs differ in scale, not in spirit, and they add things the original paper did not need: QK-normalization to keep attention logits bounded at higher resolution (SD3 lists it in the high-resolution finetuning section), mixed precision, and sharded optimizers. If you replicate at scale, expect loss spikes without QK-norm or similar protections.

Serving checklist

At inference, the levers map to the cost model above. Fewer sampling steps help linearly. Guidance distillation halves passes. Quantizing the transformer to 8-bit or lower reduces memory and often gives near-linear speedups on supporting hardware, though the quantization-error profile of diffusion differs from language models because errors accumulate across denoising steps. Caching activations between adjacent timesteps exploits the fact that consecutive steps produce similar features; it trades small quality drops for large speedups. All of these are engineering trade-offs I label as common practice rather than benchmarked here.

For readers working in industrial settings, generative video is increasingly used to synthesize training data for perception systems and to build visual scenarios for simulation; the planning and control side of those stacks is a different discipline, as covered in our post on model predictive control with digital twins. A caution: generated frames are not measurements. They carry no calibration, so they should be labelled and kept out of any pipeline that treats images as ground truth.

Trade-offs, Gotchas, and What Goes Wrong

Quadratic attention is the tax. Every claim of “scalable” in DiT is about parameters and Gflops, and Gflops grow with token count. Halving the patch size gives 4x tokens and at least 4x compute; doubling clip length does worse. If your product needs long, high-resolution video, you are paying the attention bill or engineering around it.

Weak inductive bias means data hunger. A U-Net’s convolutional locality helps with small data. A transformer must learn locality from examples, so low-data regimes can favour convolutional hybrids. DiT’s ImageNet results used millions of steps, 7M for the top 256×256 model, which is a long run.

adaLN is parameter-heavy. As computed above, the modulation MLP holds roughly a third of each block’s weights while adding almost no FLOPs. In memory-bound deployments that is wasted capacity, and in fine-tuning it is a large set of parameters coupled to the timestep.

Text binding fails in predictable ways. Models with a single pooled text vector, or with weak joint attention, confuse attribute binding (a red cube and a blue sphere become a purple blob). MM-DiT improves this; it does not eliminate it. Counting, spatial relations, and rendered text remain the failure surface to test.

VAE ceilings. Over-compressed latents lose fine detail such as small text, hands, and thin lines. No transformer fixes information the autoencoder discarded.

Guidance artifacts. High guidance scales oversaturate and flatten diversity. In the DiT paper cfg=1.5 was best for FID but the scale that is best for FID is not necessarily best for human preference.

Evaluation traps. FID is computed on ImageNet class-conditional samples with a specific feature extractor and does not predict text-to-image quality. Benchmarks like GenEval and human preference have their own contamination and prompt-distribution issues. When a vendor quotes a single score, ask for the sample count and evaluation protocol.

Physics is not learned. As OpenAI’s own report says, the model does not accurately capture many basic physical interactions. Never use generated video as evidence of physical behaviour.

Untested claim boundary. The compute numbers in the video section are first-principles estimates with stated assumptions; real kernels use FlashAttention and fused ops, so measured time will differ. I have not run any benchmark for this post.

Practical Recommendations

Choose the conditioning mechanism by the kind of signal. A timestep, class label, or pooled vector belongs in adaLN-Zero. A sequence of text tokens belongs in attention, and if prompt fidelity or typography matters, favour joint attention over a cross-attention add-on.

Spend compute deliberately. If you must pick between a bigger transformer and a smaller patch, remember the DiT finding that Gflops predict FID across both, but a smaller patch buys finer spatial tokens at the cost of quadratic attention. For video, reduce tokens first: stronger VAE compression and sensible patching save more than any kernel tuning.

Use flow matching for new training runs. It is simulation-free, the target is simple, and straight paths cooperate with few-step samplers. Sample timesteps with an emphasis on the middle of the trajectory rather than uniformly.

Plan serving around passes, not parameters. Count steps x guidance passes x tokens, then attack each factor.

A short checklist before you ship a diffusion transformer system:

Confirm VAE channels, compression ratio, and decoder quality on your target content, especially text and thin structures.
Use zero-initialized residual gates (adaLN-Zero) for deep stacks, and QK-norm for high-resolution training.
Track EMA weights; evaluate with the EMA model.
Measure token count N for every resolution and duration you support, and budget attention at N squared.
Test attribute binding, counting, and rendered text with a fixed prompt suite, not only an aggregate score.
Decide guidance scale by human review, not just FID.
Label generated outputs and keep them out of measurement pipelines.
Verify licensing: Wan 2.1 is Apache 2.0 per its repository; check each model you adopt.

Frequently Asked Questions

What is a diffusion transformer?

A diffusion transformer is a denoising network that replaces the usual convolutional U-Net with a stack of transformer blocks operating on latent patch tokens. The timestep and conditioning enter through adaptive layer norm or attention, and the model predicts noise, clean data, or a velocity per token. DiT, introduced in 2022 by Peebles and Xie, showed that this design scales smoothly with compute and beat the best U-Net diffusion models on ImageNet.

What does adaLN-Zero do in DiT?

adaLN-Zero regresses a per-block scale, shift, and residual gate from the timestep and class embedding. The gate is initialized to zero, so every block starts as an identity function and training begins from a stable point. In the DiT paper it gave lower FID than in-context and cross-attention conditioning, while keeping compute at the adaLN level of 118.6 Gflops for DiT-XL/2.

How is MM-DiT different from the original DiT?

MM-DiT, used in Stable Diffusion 3, keeps separate weights for image tokens and text tokens but computes attention jointly over the combined sequence. Text and image representations therefore update each other at every layer, which the authors link to better text comprehension and typography. The original DiT only conditioned on a class label through adaLN-Zero, with no sequence-level text at all.

Why are diffusion transformers expensive for video?

Video multiplies the token count. A five-second clip can produce tens of thousands of spacetime tokens after VAE compression, and self-attention cost grows with the square of that number. In the worked example above, about 75,600 tokens put roughly 69 percent of each forward pass into attention. Sampling then repeats that pass for dozens of steps, usually twice per step with classifier-free guidance.

Is flow matching better than standard diffusion training?

Flow matching trains with a simple regression to a vector field and needs no simulation during training. Lipman and colleagues report better likelihood and sample quality than diffusion baselines on ImageNet, and the SD3 paper reports rectified flow beating established diffusion formulations in a large text-to-image study. It is not a free win: schedules, timestep sampling, and guidance still need tuning.

Do diffusion transformers understand physics?

No, not in a reliable sense. They learn spatiotemporal statistics from data. OpenAI’s report on Sora states that the model does not accurately model the physics of many basic interactions, such as glass shattering, and that long samples can become incoherent. Scaling improves consistency, but generated video should never be treated as physical evidence or measurement data.

Diffusion Transformers (DiT) Explained: Architecture, adaLN and Scaling for Image and Video (2026)