Back to paper library
arXiv 2024NeurIPS 2024Image generationPaper notes

MAR

Reforging the Order of Generation

Original paper

Autoregressive Image Generation without Vector Quantization

Paper authors

Tianhong Li · Yonglong Tian · He Li · Mingyang Deng · Kaiming He

Publication
NeurIPS 2024Conference
First submitted to arXiv
arXiv:2406.11838
Cite this paper · BibTeX

The citation identifies the linked source version. It cites the original paper and its authors.

@misc{imagegenerationmar2024,
  title = {{Autoregressive Image Generation without Vector Quantization}},
  author = {Tianhong Li and Yonglong Tian and He Li and Mingyang Deng and Kaiming He},
  year = {2024},
  eprint = {2406.11838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.11838}
}
On this page

Current autoregressive image generation models often use vector quantization (VQ) to discretize images into tokens, mimicking the success of autoregressive models in the NLP domain. However, the authors argue that such a discrete space is not necessary for autoregressive image generation. Therefore, they propose an autoregressive image generation model based on continuous space, which achieves higher accuracy and faster inference time.

Background Knowledge

Vector Quantization (VQ)

This is a long-established technique to accelerate vector search. Basic VQ maps each input vector to the nearest representative in a finite codebook and stores its discrete index. Splitting a vector into subvectors is a specific approach such as product quantization, not the definition of all VQ.

For details, please refer to Survey Of Vector Space Search or Vector quantization wiki.

Auto-regressive Image Generation

Early autoregressive image models (AR here, not the specific 2024 method named VAR) treated the image generation task as GPT-like autoregressive text generation, predicting discrete pixel or channel values. The model’s task was to perform multi-class prediction using categorical cross-entropy. Examples include Google's Image Transformer, 2018 and OpenAI’s ImageGPT, 2020.

To accelerate image generation speed, current autoregressive image generation models commonly introduce VQ for two-stage training: the first stage learns a codebook in the latent space for image reconstruction, and the second stage autoregressively generates images based on the learned codebook.

  • Take VQ-VAE, 2017 as an example:

    • In the encode stage, a CNN extracts image features, then vector quantization is applied on the feature map zez_e to obtain centroid vectors (purple vectors, e1∼eKe_1 \sim e_K). Next, distances between each feature point in zez_e and the centroid vectors are computed, and the nearest centroid vector index replaces each feature point, producing the discrete representation q(z∣x)q(z|x).
    • In the decode stage, the image is generated using the mapped back ZqZ_q from qq.

Autoregressive models speeding up image generation with VQ sounds great, but are there no drawbacks?

The paper highlights two costs: codebook training can be unstable, and quantization error can limit reconstruction quality.

Method

Abandoning VQ, Embracing Diffusion

Since diffusion models can represent the joint probability distribution of all pixels or tokens in an image, why not use them to represent the probability distribution of each token?

  • Diffusion models can generate images from noise conditioned on an input prompt/image.
  • The current approach conditions on the output of a transformer to generate images from noise (details of the transformer input will be explained later).

Their method autoregressively predicts the conditional latent variable zz for each token, then uses a diffusion model (a small MLP) to perform denoising and obtain the output xx's probability distribution p(x∣z)p(x|z).

They propose a Diffusion Loss to replace the original categorical cross-entropy.

  • ε\varepsilon is a noise sample drawn from a standard normal distribution N(0,I)\mathcal{N}(0,I); zero and one describe mean and per-component variance, not a bounded sampling interval.

  • εθ\varepsilon_{\theta} is a small MLP, where εθ(xt∣t,z)\varepsilon_{\theta}(x_t | t, z) means the model takes timestep tt and condition zz as inputs and predicts the noise vector εθ\varepsilon_{\theta} given xtx_t.

    • tt is the timestamp in the noise schedule, and zz is the condition vector generated by the Transformer based on context.

Finally, similar to DDPM, during inference the model uses the reverse diffusion process to generate images (from xtx_t to x0x_0).

Additionally, readers familiar with diffusion models for image generation will recognize the hyperparameter called temperature, which controls sampling randomness—higher temperature means more randomness, lower means less. In this work, temperature is controlled by τ∗σtδ\tau * \sigma_t \delta, following the method introduced in Diffusion Models Beat GANs on Image Synthesis.

Reflecting on this approach reveals a clever combination of VAE’s conditional generation and DDPM’s denoising: compressing image features with a VAE and generating images with DDPM.

Integrating Concepts from Autoregressive and Masked Generative Models

As the author of MAE (Masked Autoencoders Are Scalable Vision Learners), 2021, Kaiming He naturally considered incorporating MAE’s ideas into the model.

  • The main pipeline refers to follow-up works on MAE: MaskGIT, 2022 and MAGE, 2022, both using Transformer-based masked generation, related to MAR but with different sampling-order and temperature choices. MaskGIT’s contribution is using a bidirectional transformer decoder to predict multiple tokens simultaneously, while MAGE unifies image representation learning and image synthesis.

  • This work adopts MAE’s bidirectional attention mechanism, placing mask tokens [m] in the middle layers so that all tokens can see each other, rather than only previous tokens. Loss is computed only on masked, unknown tokens. Known tokens can attend to each other; unknown positions receive known content and positional information without seeing their ground-truth target values.

    • Note: The MAE-style encoder/decoder names do not specify whether attention is causal or bidirectional.

    • The advantage is improved image generation quality; the disadvantage is that training and inference cannot leverage kv cache acceleration. However, since multiple tokens are predicted simultaneously, the method is still reasonably fast.

  • The figure below compares standard sequential AR, random-order AR (random masking of one token), and their approach (random masking of multiple tokens with simultaneous prediction)—this reveals that the transformer input mentioned earlier is a masked image.

Implementation

  • Diffusion Loss: uses a cosine-shaped noise schedule; training uses 1000 DDPM steps while inference uses only 100 steps.
  • Denoising MLP (small MLP): consists of 3 blocks with 1024 channels each; each block contains LayerNorm, linear layer, SiLU activation, and residual connections. AdaLN is used to inject the transformer output zz into the LayerNorm layers.
  • Tokenizer: uses publicly available tokenizers from LDM, including VQ-16 and KL-16. VQ-16 is based on VQ-GAN with GAN loss and perceptual loss; KL-16 uses KL divergence regularization and does not rely on VQ.
  • Transformer: a ViT receives the token sequence from the tokenizer, adds positional encoding and a class token [CLS], then passes through 32 layers of transformer blocks each with 1024 channels.
  • Masked autoregressive models: training uses a masking ratio between 0.7 and 1.0 (e.g., 70% tokens randomly masked). To avoid sequences becoming too short, 64 [CLS] tokens are always padded. During inference, the masking ratio is gradually reduced from 1.0 to 0 using a cosine schedule over 64 steps by default.
  • Baseline autoregressive model: a GPT model with causal attention, input appended with a [CLS] token, supporting kv cache and temperature parameters.

Experiments

The main ablations below use AR/MAR-L (about 400M parameters), 400 epochs and ImageNet 256×256. The final system comparison includes other scales and training settings.

Diffusion Loss vs. Cross-Entropy Loss

Among all AR/MAR variants, models trained with Diffusion Loss consistently outperformed those trained with cross-entropy loss. The AR model saw the smallest improvement, with larger gains in bidirectional MAR variants, demonstrating the critical importance of Diffusion Loss for MAR models.

  • Diffusion Loss also incorporates Classifier-Free Guidance (CFG), commonly used in diffusion models, to enhance generation quality.
  • Fréchet Inception Distance (FID) is better when lower, while Inception Score (IS) is better when higher; both are proxies for generated-image quality and do not fully capture human perception.

Tokenizers

Experiments tested different tokenizers paired with Diffusion Loss. Moving away from discrete space to continuous space allows Diffusion Loss to work on both continuous and discrete tokenizers.

  • VQ-16 refers to taking the continuous latent before vector quantization in VQ-VAE as tokens. Both have public LDM versions; only the KL-16 row marked † is retrained by the authors on ImageNet instead of OpenImages.
  • Consistency Decoder is a non-VQ tokenizer originating from DALL·E 3.
  • Reconstruction FID (rFID) is better when lower, used to evaluate tokenizer quality.

Results show continuous tokenizers like KL-16 outperform discrete ones like VQ-16. The model also works compatibly with different tokenizers such as Consistency Decoder.

Denoising MLP

Width 1024 is the default trade-off. With CFG, width 1536 has a slightly lower FID, so 1024 is not best on every metric.

Sampling Steps of Diffusion Loss

Different diffusion steps impact generation quality. Using 100 steps already achieves good performance.

Temperature of Diffusion Loss

Temperature is also an important factor for Diffusion Loss.

Speed/Accuracy Trade-off

Since kv cache cannot be used, this evaluation is critical. Testing was done on an A100 GPU with batch size=256.

  • MAR: each point represents different autoregressive steps (8 to 128).
  • DiT: each point represents different diffusion steps (50, 75, 150, 250), using DiT-XL here.
  • AR: uses kv cache.

Despite no kv cache, this model maintains decent inference speed, and the default 64-step cost should be considered alongside quality. Throughput measured with batch size 256 is not single-image interactive latency.

System-Level Comparison

Compared with other models, only the largest model MAR-H achieves the best performance, but MAR-L already performs well.

Conclusion

In summary, this work breaks away from traditional autoregressive image generation methods by combining Diffusion and MAE concepts to open a new direction. It demonstrates promising results using the simplest DDPM; whether other diffusion methods improve the result still needs experiments. Looking forward to more derivative works.

Below are some example images generated by the model.

Reference

https://zhouyifan.net/2024/07/27/20240717-ar-wo-vq/