nomic-ai

/

nomic-bert-2048

like 54

Follow

Nomic AI 1.09k

Hello!

Pull Request overview

• Fix loading nomic-ai/nomic-embed-vision-v1.5 with transformers v5+

Details

With transformers v5+, from_pretrained initializes models on torch.device("meta"), which causes several issues in the vision model code:

1. NomicVisionModel doesn't call post_init(), which is required in v5+ to set all_tied_weights_keys and other static properties. Without it, from_pretrained raises an AttributeError.
2. NomicVisionRotaryEmbeddingCat registers pos_embed as a non-persistent buffer, so it's never materialized when initialized on the meta device. I've added logic to recompute it on first use and pass the correct device through from forward.
3. NomicBertAttention and NomicAttentionPooling reference self.norm_factor which isn't available after from_pretrained loading. Replaced with inline math.sqrt(self.head_dim).

You can load and run the vision model with this using both revision (for https://huggingface.co/nomic-ai/nomic-embed-vision-v1.5/discussions/10) and code_revision to test it out:

from transformers import AutoModel, AutoImageProcessor
from PIL import Image
import requests

processor = AutoImageProcessor.from_pretrained("nomic-ai/nomic-embed-vision-v1.5")
model = AutoModel.from_pretrained(
    "nomic-ai/nomic-embed-vision-v1.5",
    trust_remote_code=True,
    revision="refs/pr/10",
    code_revision="refs/pr/23",
)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(image, return_tensors="pt")
outputs = model(**inputs)
print(outputs.last_hidden_state.shape)

torch.Size([1, 197, 768])

Whereas if you don't use code_revision to use this PR, you get:

Traceback (most recent call last):
  File "d:\models\st_v54\nomic-bert-2048\demo_test.py", line 6, in <module>
    model = AutoModel.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\models\auto\auto_factory.py", line 380, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\modeling_utils.py", line 4133, in from_pretrained
    loading_info = cls._finalize_model_loading(model, load_config, loading_info)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\modeling_utils.py", line 4275, in _finalize_model_loading
    model.mark_tied_weights_as_initialized(loading_info)
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\modeling_utils.py", line 4615, in mark_tied_weights_as_initialized
    for tied_param in self.all_tied_weights_keys.keys():
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\torch\nn\modules\module.py", line 1965, in __getattr__
    raise AttributeError(
AttributeError: 'NomicVisionModel' object has no attribute 'all_tied_weights_keys'. Did you mean: '_tied_weights_keys'?

As the self.post_init() wasn't called. If you use neither PR (i.e. no revision or code_revision), then you get:

Traceback (most recent call last):
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 144, in __strict_setattr__
    validator(value)
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 625, in validator
    type_validator(field.name, value, field.type)
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 470, in type_validator
    validator(name, value, args)
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 507, in _validate_union
    raise TypeError(
TypeError: Field 'n_inner' with value 2048.0 doesn't match any type in (<class 'int'>, <class 'NoneType'>). Errors: Field 'n_inner' expected int, got float (value: 2048.0); Field 'n_inner' expected NoneType, got float (value: 2048.0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "d:\models\st_v54\nomic-bert-2048\demo_test.py", line 6, in <module>
    model = AutoModel.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\models\auto\auto_factory.py", line 329, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\models\auto\configuration_auto.py", line 1504, in from_pretrained
    return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\configuration_utils.py", line 647, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\configuration_utils.py", line 824, in from_dict
    config = cls(**config_dict)
             ^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.cache\huggingface\modules\transformers_modules\nomic_hyphen_ai\nomic_hyphen_bert_hyphen_2048\7710840340a098cfb869c4f65e87cf2b1b70caca\configuration_hf_nomic_bert.py", line 56, in __init__
    super().__init__(**kwargs)
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 275, in init_with_validate
    initial_init(self, *args, **kwargs)  # type: ignore [call-arg]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\configuration_utils.py", line 95, in __init__
    setattr(self, f.name, standard_kwargs[f.name])
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\huggingface_hub\dataclasses.py", line 146, in __strict_setattr__
    raise StrictDataclassFieldValidationError(field=name, cause=e) from e
huggingface_hub.errors.StrictDataclassFieldValidationError: Validation error for field 'n_inner':
    TypeError: Field 'n_inner' with value 2048.0 doesn't match any type in (<class 'int'>, <class 'NoneType'>). Errors: Field 'n_inner' expected int, got float (value: 2048.0); Field 'n_inner' expected NoneType, got float (value: 2048.0)

In short: with the 2 PRs, the vision model should work again.

• Tom Aarsen