Written by Ethan Smith

Yes, but maybe not for the reason you think.

What are Diffusion Transformers (DiTs)?

Diffusion Transformers were first introduced in this paper

Conceptually they are not too complicated if you are familiar with transformers. (the remainder expects some familiarity so worth getting a basic introduction if not!)

The architecture is identical to that of LLMs except:

• Remove the embedding table and LM head as we’ll be operating in continuous space
  • This is achieved by patchifying the input as seen in ViT models.
• Ditch causal masking, all tokens get to interact fully.
• A 2d type of positional embeddings to deal with image data.
• Something like AdaNorm to condition the model on the diffusion timestep/condition as per usual.

The work presents the architecture as having good scaling properties when compared to counterparts, achieving good FID scores relative to FLOP or parameter count.