Taming Transformers

The approach has two stages. First, a convolutional encoder-decoder with a learned codebook (often called VQGAN) maps images to a grid of discrete indices, and reconstructs them with perceptual and adversarial losses so the latent space is compact but detailed. Second, an autoregressive Transformer learns the distribution over these latent tokens, optionally conditioned on text, class labels, or layouts. By modeling a short sequence of discrete latents instead of raw pixels, the Transformer becomes fast enough and data-efficient enough to handle high resolutions while keeping textures and structure intact. The result is a flexible recipe for controllable image generation and editing, and it influenced many later latent and token-based generative models used in creative and design workflows.