nvidia

/

NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

like 144

Follow

NVIDIA 55.4k

I'm attempting to run this model using a simple python script

from tensorrt_llm import LLM, SamplingParams
llm = LLM(model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4") 
prompts = [
        "Hello, my name is",
        "The capital of France is",
        "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

for output in llm.generate(prompts, sampling_params):
      print(
            f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}"
        )

This fails with the error:
AttributeError: 'NemotronHConfig' object has no attribute 'rms_norm_eps'
It's caused by this script attempting to access rms_norm_eps
https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/models/modeling_nemotron_h.py#L316
but there is no rms_norm_eps in the config for this model, not sure which needs updating..?
https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4/blob/main/configuration_nemotron_h.py