YuCollection/Lens-Diffusers

Note: This repository is an archived mirror and is not the original upstream source.
The original model, weights, and documentation are developed and maintained by Microsoft.

All hosted model weights are unmodified.

This project is released under the MIT License, which permits use, modification, and redistribution under its terms.

This repository is not affiliated with, endorsed by, or sponsored by Microsoft.

Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models

Contributors (Alphabetical Order):
Baining Guo, Chong Luo, Dong Chen†, Dongdong Chen, Fangyun Wei†, Ji Li, Jianmin Bao, Jiawei Zhang*, Jinjing Zhao*, Lei Shi, Qinhong Yang, Sirui Zhang*, Xiuyu Wu, Xuelu Feng, Yan Lu, Yanchen Dong, Yang Yue*, Yitong Wang, Yunuo Chen, Zhiyang Liang*, Ziyu Wan†
Microsoft  |  *Core Contributors  |  †Project Lead

     

---

Lens is a 3.8B-parameter foundational text-to-image model designed for efficient training and fast high-resolution generation. It combines dense-caption pre-training, mixed-resolution learning, GPT-OSS multi-layer text features, and the FLUX.2 semantic VAE to reach competitive quality with substantially less training compute than larger T2I models.

This repository provides the minimal inference code for generating images from Lens DiT checkpoints.

Highlights

• Efficient Foundation — Trained on Lens-800M, an 800M image-text corpus with long GPT-4.1 captions, maximizing information density per training batch.
• Compact & Expressive — A 48-block MMDiT denoiser leverages FLUX.2 latents and concatenated multi-layer GPT-OSS features for stronger prompt following and multilingual generalization.
• Flexible Resolution — Mixed-resolution training enables inference across aspect ratios from 1:2 to 2:1 and resolutions up to 1440×1440.
• Post-trained Variants — RL tuning improves visual quality and artifact suppression; the distilled Lens-Turbo supports fast 4-step generation.

Installation

Tested environment: Python 3.12 · CUDA 12.6 · PyTorch 2.11.0+cu126 · TorchVision 0.26.0+cu126

conda create -n lens python=3.12 -y
conda activate lens
uv pip install torch==2.11.0+cu126 torchvision==0.26.0+cu126 \
    --index-url https://download.pytorch.org/whl/cu126
uv pip install -r requirements.txt

The default GPT-OSS encoder and FLUX.2 VAE are loaded from Hugging Face. Make sure your environment has access to any gated model repositories you use.

Checkpoints

Repo	Description	Steps	CFG
microsoft/Lens	Default. RL-tuned for visual quality	20	5.0
microsoft/Lens-Turbo	Distilled from the RL model for fast 4-step sampling	4	1.0
microsoft/Lens-Base	Supervised base model (no RL, no distillation)	50	5.0

Pick a variant by passing its repo id to --repo_id (CLI) or LensPipeline.from_pretrained(...) (Python).

Inference

Important: run from the cloned repo root so from lens import LensPipeline resolves to this package — importing lens is what registers LensGptOssEncoder / LensTransformer2DModel with the transformers and diffusers namespaces that model_index.json references.

Python API:

import torch
from lens import LensPipeline
pipe = LensPipeline.from_pretrained(
    "microsoft/Lens", torch_dtype=torch.bfloat16
).to("cuda")
image = pipe(
    prompt="A cat holding a sign that says \"hello world\"",
    base_resolution=1440, aspect_ratio="1:1",
    num_inference_steps=20, guidance_scale=5.0,
    generator=torch.Generator("cuda").manual_seed(0),
).images[0]
image.save("lens.png")

To trade speed for VRAM, replace .to("cuda") with pipe.enable_model_cpu_offload().

CLI — basic usage:

python inference.py \
    --repo_id "microsoft/Lens" \
    --prompt "A cinematic mountain lake at sunrise, soft mist, detailed reflections" \
    --base_resolution 1440 --aspect_ratio 1:1 \
    --steps 20 --cfg 5.0 --n 1 --seed 42 \
    --out ./outputs

Batch generation — join multiple prompts with |:

python inference.py \
    --repo_id "microsoft/Lens" \
    --steps 20 --cfg 5.0 \
    --prompt "a red fox in snow|a glass greenhouse at night"

A100 / V100 (no MXFP4 kernels) — dequantize the GPT-OSS encoder to bf16:

python inference.py \
    --repo_id "microsoft/Lens" \
    --steps 20 --cfg 5.0 \
    --prompt "a cat" \
    --disable_mxfp4 --offload

Options

Flag	Description	Default
--repo_id	HF repo id (or local path) of the assembled Lens pipeline	microsoft/Lens
--base_resolution	1024 or 1440	1440
--aspect_ratio	1:2, 9:16, 2:3, 3:4, 1:1, 4:3, 3:2, 16:9, 2:1	1:1
--steps	Number of denoising steps	20
--cfg	Classifier-free guidance scale	5.0
--n	Number of images per prompt	1
--seed	Random seed (omit for non-deterministic)	—
--out	Output directory	./outputs
--dtype	Compute dtype: bfloat16, float16, float32	bfloat16
--disable_mxfp4	Dequantize the GPT-OSS text encoder to --dtype (required on A100 / V100; Hopper+ keeps MXFP4 by default for less VRAM)	—
--offload	Enable diffusers CPU offload (text_encoder->transformer->vae) to reduce peak VRAM	—
--reasoner	Refine prompts with the loaded GPT-OSS encoder before generation	—
--api_url / --api_key / --api_model	Use an OpenAI-compatible API for prompt refinement (takes precedence over --reasoner)	—

Citation

@article{zhao2026lens,
  title   = {Lens: Rethinking Training Efficiency for Foundational Text-to-Image Models},
  author  = {Guo, Baining and Luo, Chong and Chen, Dong and Chen, Dongdong and Wei, Fangyun and Li, Ji and Bao, Jianmin and Zhang, Jiawei and Zhao, Jinjing and Shi, Lei and Yang, Qinhong and Zhang, Sirui and Wu, Xiuyu and Feng, Xuelu and Lu, Yan and Dong, Yanchen and Yue, Yang and Wang, Yitong and Chen, Yunuo and Liang, Zhiyang and Wan, Ziyu},
  journal = {arXiv preprint arXiv:2605.21573},
  year    = {2026}
}

Responsible AI

The model is released for research purposes only and is not intended for product or service deployment. Responsible AI considerations were incorporated throughout the development process, including data selection, model training, and evaluation. The training data includes a combination of public, licensed, and internal datasets that were processed to remove clearly identifiable personal information and reduce harmful content where possible. However, as the data is largely sourced from web-scale collections, it may contain biases or uneven representation. As a result, the model may generate outputs that are inaccurate, biased, or inappropriate under certain prompts, including content that could be misleading or raise copyright or IP-related concerns. Given these limitations, the model should be used in controlled research settings, with appropriate human oversight. Downstream users are responsible for applying additional safeguards, such as content moderation, validation, and compliance checks, before using the model in broader applications.

Privacy

This project does not collect any usage data. For more information, see the Microsoft Privacy Statement.

License

This project is released under the MIT License.