PhatcatDK/t5gemma-2b-2b-ul2-encoder-only

Encoder-only extraction for google/t5gemma-2b-2b-ul2

This repo contains the extracted T5 encoder-only weights for google/t5gemma-2b-2b-ul2. Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms.

Reference table

This extraction is part of a complete T5Gemma suite (270m, 1b, 2b, 4b, 9b) created to provide a unified environment for LLM-SDXL adapter research.

Model	T5Gemma Series	Architecture	Model Size (GB)	Saved VRAM	Note
270m-270m	V2 (Gemma-3)	Multimodal	1.37	~13%	Vision tower takes up most of the full model
1b-1b	V2 (Gemma-3)	Multimodal	2.84	~33%
2b-2b	V1 (Gemma-2)	Text-only	5.23	~53%	Most VRAM saved of V1
4b-4b	V2 (Gemma-3)	Multimodal	8.60	~43%	Most VRAM saved of V2
9b-2b	V1 (Gemma-2)	Text-only	18.46	~25%	Encoder takes up most of the full model

Why?

Standard T5-Gemma checkpoints are designed for seq2seq tasks, carrying heavy decoder overhead. This extraction isolates the V1 text-only encoder (Gemma-2 based), stripping the decoder layers to reduce VRAM consumption by ~53%.

How to load the model

from transformers import T5GemmaEncoderModel

model_path = "PhatcatDK/t5gemma-2b-2b-ul2-encoder-only"

# Load the weights
model = T5GemmaEncoderModel.from_pretrained(
    model_path,
    torch_dtype="auto", # Recommended: bfloat16
    is_encoder_decoder=False, # CRITICAL: Tells Transformers there is no decoder
    trust_remote_code=True
)

Acknowledgements & Inspiration

This work was directly inspired by the pioneering research of Minthy.

Minthy was the first to demonstrate the power of the t5gemma-2b-2b-ul2 encoder within this ecosystem. This specific repository was created to round out a standardized suite (270m through 9b) with a "clean-config" setup, ensuring consistent architectural parity for cross-attention testing.

I am incredibly grateful for the groundwork Minthy laid; this suite exists to support and verify the technical infrastructure he built.