jmroth/nlp-biencoder-finetuned

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jmroth/my-awesome-model")
# Run inference
sentences = [
    '...there [is] anecdotal and other evidence suggesting similar melts from 1938-43 and on other occasions.',
    'They were formed by the melting of sulfur deposits at temperatures as low as 113\xa0°C (235\xa0°F).',
    'Consequently, summers are 2.3\xa0°C (4\xa0°F) warmer in the Northern Hemisphere than in the Southern Hemisphere under similar conditions.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4966, 0.1535],
#         [0.4966, 1.0000, 0.3254],
#         [0.1535, 0.3254, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: claims-dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.2403
cosine_accuracy@3	0.4416
cosine_accuracy@5	0.5455
cosine_accuracy@10	0.6818
cosine_precision@1	0.2403
cosine_precision@3	0.1905
cosine_precision@5	0.1545
cosine_precision@10	0.1071
cosine_recall@1	0.0958
cosine_recall@3	0.2148
cosine_recall@5	0.2753
cosine_recall@10	0.3661
cosine_ndcg@10	0.2932
cosine_mrr@10	0.3743
cosine_map@100	0.23

Training Details

Training Dataset

Unnamed Dataset

• Size: 4,122 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 8 tokens mean: 26.75 tokens max: 65 tokens	min: 7 tokens mean: 38.71 tokens max: 256 tokens

• Samples:

  anchor	positive
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• weight_decay: 0.01
• warmup_steps: 0.1
• fp16: True
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: jmroth/nlp-biencoder-finetuned
• hub_strategy: end
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 128
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: jmroth/nlp-biencoder-finetuned
• hub_strategy: end
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	claims-dev_cosine_ndcg@10
0.0775	10	1.4212	-
0.1550	20	1.4229	-
0.2326	30	1.1129	-
0.3101	40	0.9966	-
0.3876	50	0.9207	0.2829
0.4651	60	0.8326	-
0.5426	70	0.8989	-
0.6202	80	0.9630	-
0.6977	90	0.8394	-
0.7752	100	0.8764	0.2893
0.8527	110	0.8208	-
0.9302	120	0.7684	-
1.0078	130	0.7049	-
1.0853	140	0.7378	-
1.1628	150	0.6265	0.2941
1.2403	160	0.6832	-
1.3178	170	0.6365	-
1.3953	180	0.5991	-
1.4729	190	0.5456	-
1.5504	200	0.6355	0.2943
1.6279	210	0.5927	-
1.7054	220	0.7117	-
1.7829	230	0.5096	-
1.8605	240	0.6036	-
1.9380	250	0.6768	0.2896
2.0155	260	0.6589	-
2.0930	270	0.5436	-
2.1705	280	0.5173	-
2.2481	290	0.5544	-
2.3256	300	0.5583	0.2911
2.4031	310	0.5903	-
2.4806	320	0.5265	-
2.5581	330	0.5107	-
2.6357	340	0.6144	-
2.7132	350	0.5175	0.2932
2.7907	360	0.5805	-
2.8682	370	0.5299	-
2.9457	380	0.5621	-

• The bold row denotes the saved checkpoint.

Training Time

• Training: 32.6 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}