RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolution

Project Page: https://syjmelody.github.io/RankE/
GitHub: https://github.com/syjmelody/RankE

⚡ TL;DR

RankE is the first end-to-end post-training framework for discrete text-to-image generation that jointly optimizes the Generator and the Decoder. Instead of improving reward scores at the cost of image quality, RankE improves both alignment and fidelity at the same time.

🤔 Background: What is the problem?

Most discrete text-to-image models still follow a two-stage pipeline:

1. train a VQ-VAE / tokenizer to map images into discrete visual tokens;
2. train an autoregressive Generator to model those tokens.

This pipeline works well for pretraining, but post-training is usually incomplete: existing methods optimize the Generator only and keep the Decoder frozen.

That creates a mismatch. As the Generator is optimized to chase higher rewards, its output token distribution gradually drifts away from the real token distribution that the Decoder saw during tokenizer training. The result is a frustrating trade-off:

• reward scores go up,
• but decoded image quality can get worse.

The paper identifies this issue as Latent Covariate Shift.

🔍 Why existing solutions are not enough

Recent work such as REPA-E has shown that in continuous diffusion models, the autoencoder is not just a supporting module — it can be a real bottleneck for alignment and visual quality.

But discrete T2I is harder.
Because token sampling and vector quantization are discrete operations, gradients cannot flow cleanly through the entire generation process. That is why most existing RL or preference-optimization methods for discrete generation still update only the Generator while leaving the Decoder unchanged.

So the field already knows the decoder matters — but a practical end-to-end solution for discrete generation has been missing.

🚀 What RankE does

RankE addresses this directly by making the Generator and Decoder co-evolve.

Its core insight is simple:
if Generator optimization already behaves like a ranking process over latent token sequences, why not extend the same ranking principle to pixel-space Decoder optimization?

RankE therefore uses alternating optimization:

• Generator step: optimize the policy so that higher-reward latent token sequences receive stronger updates;
• Decoder step: optimize the Decoder so it can better adapt to the Generator’s evolving token distribution, while also favoring higher-reward decoded images.

In other words, RankE does not just make the model “better at scoring well.” It aligns optimization across both latent space and pixel space.

This is the key difference from standard frozen-decoder RL.

🧠 Why this matters

In standard RL post-training for discrete T2I, the Generator keeps changing, but the Decoder stays fixed. Over time, the Decoder is forced to decode token patterns it was never really trained to handle.

RankE removes this bottleneck by continuously adapting the Decoder during post-training. This turns reward optimization into actual visual improvement, rather than reward hacking in latent space.

📈 Results

The gains are clear.

On LlamaGen-XL (775M) under CLIP-based optimization:

• Standard RL: improves CLIP, but hurts FID
• RankE: improves both

Specifically:

• CLIP: 32.45 → 33.76
• FID: 17.76 → 15.21

That is the main message of the paper:
RankE breaks the common fidelity–alignment trade-off in discrete text-to-image post-training.

The improvements are also consistent across:

• different backbones,
• different reward functions,
• and multiple evaluation settings.

✨ One-line takeaway

RankE is a more natural way to post-train discrete text-to-image models: instead of optimizing only the Generator, it lets the Generator and Decoder improve together.