google

/

gemma-3-27b-it

like 1.96k

Follow

Google 52.2k

Hi @zml31415 ,

Thanks for reaching out to us, the google/gemma-3-27b-it or google/gemma-3-12b-it are instruction tuned (IT) models they follows a specific kind of prompt and chat template to process your query/prompt, which means any IT Gemma model follows a role based instructions to process your request. Please find the following sample prompt message for your reference:

messages = [
{
"role": "system",
"content": [{"type": "text", "text": "You are a helpful assistant."}]
},
{
"role": "user",
"content": [
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
{"type": "text", "text": "What animal is on the candy?"}
]
}
]

inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

Please adjust your prompt based on the above given sample prompt message. Thanks for your interest in Gemma models.

Thanks.

edit: due to code mistake while posting, changed the line inputs_embeds = model.get_input_embeddings()(generated_ids) to inputs_embeds = model.get_input_embeddings()(input_ids)

Thank you for your response. But i feel like your answer is not exactly tackling my question.I try to clarify it: If i understand the models code correctly, one could provide the inputs to the model two ways:
First via the input_ids that are coming from the processor and
second via inputs_embeds, that i can get from the models function model.get_input_embeddings()(input_ids).
I already tried the apply_chat_template, and it works because internally it is using the input_ids route which is at the deciding if-statement (line 897-900, modeling_gemma3.py). It calculates the special_image_mask in a way that works, even with the code and prompt that i provided. One just needs to disable the assert in line 867, 868. But providing the inputs_embeds -- taking the else path in line 901, it doesn't work. I wonder if this function got tested or if there is a bug or if i overlooked something.
Interestingly, internally the inputs_embeds are also calculated if not provided (line 884, 885 modeling_gemma3.py) Resulting in an almost identical (whith the exception of line 897-900) path of calculation.
So my point is, that using any method (apply_chat_template, input_ids=... , **inputs) that provides the input_ids to the model, works fine, but providing the inputs_embeds does not - even though the inputs_embedsare still calculated internally (line 884, 885 modeling_gemma3.py). The deciding if statement is in line 897-900, because the calculation of the token_mask is doing something strange when not providing input_ids but inputs_embeds. This line is the only difference i could track down between using the input_ids and the inputs_embeds as input for the model together with the pixel_values.
If i understood your answer wrong and its a simple, "you are not supposed to use our model this way", fine, but why is the "inputs_embeds" function implemented anyways then?
The funny thing is, that the input_ids in the modeling_gemma3.py are only properly used in two lines:
In 902 where the working implementation of the special_image_maskis done,
and the funny part in 885 where exactly the same is done as in my code: inputs_embeds = model.get_input_embeddings()(input_ids). And all further processing is done on those inputs_embeds. But directly providing them leads to the error in the thread title.
I hope you look into it, since using the chat_template isn't really applicable for my application.

BTW, i also tried it with the PT model where in the documentation the prompt that i used is given as an example, still the same error. Only the inputs_embeds part is different in this documentation. The input_ids are provided to the model: edit: for clarification, this is the documentation code:

# pip install accelerate

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/gemma-3-12b-pt"

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

model = Gemma3ForConditionalGeneration.from_pretrained(model_id).eval()
processor = AutoProcessor.from_pretrained(model_id)

prompt = "<start_of_image> in this image, there is"
model_inputs = processor(text=prompt, images=image, return_tensors="pt")

input_len = model_inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**model_inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

This error message is a bit misleading at first glance — it says the counts match (256 vs 256) but still throws. The actual issue is almost certainly in how the processor is handling multiple images versus a single image with multiple patches. Gemma 3's vision tokenizer uses a tiling strategy where a single image can expand into multiple <image_soft_token> placeholders depending on resolution, and the processor expects the number of image objects passed to match the number of <image> placeholder tokens in your prompt template, not the total expanded patch count. So if you're passing one image but your prompt has one <image> token, the processor internally expands that to 256 patch tokens — and if something in your pipeline is pre-inserting those 256 tokens into the text before the processor runs, you get this collision.

The fix is usually to make sure you're not manually constructing the prompt with expanded image tokens and then also passing the raw image to the processor. Let the AutoProcessor for google/gemma-3-27b-it handle the expansion entirely. Concretely: pass the raw PIL images in the images argument and use only the high-level <image> placeholder in your text, then call processor(text=..., images=..., return_tensors="pt") and let it do the patch math. If you're in an agentic pipeline where the prompt is being assembled by multiple components before hitting the model (which is increasingly common with multi-agent orchestration frameworks), this is a common point of failure — one agent constructs the prompt with expanded tokens, another passes the image separately, and the processor sees a mismatch. We've run into analogous context-assembly problems in AgentGraph when building pipelines where different agents contribute parts of a multimodal context, and the solution there was enforcing a strict contract about what stage of the pipeline is responsible for token expansion.

Check your processor version as well — there were some fixes to the Gemma 3 image token counting logic in transformers around 4.47-4.48. Running transformers.__version__ and cross-referencing against the model card's recommended version is worth doing before going deeper into debugging.