jmroth/nlp-reranker-finetuned_v1

CrossEncoder based on cross-encoder/ms-marco-MiniLM-L6-v2

This is a Cross Encoder model finetuned from cross-encoder/ms-marco-MiniLM-L6-v2 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: cross-encoder/ms-marco-MiniLM-L6-v2
• Maximum Sequence Length: 512 tokens
• Number of Output Labels: 1 label
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Full Model Architecture

CrossEncoder(
  (0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("jmroth/nlp-reranker-finetuned")
# Get scores for pairs of inputs
pairs = [
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Use of fertilizers are beneficial in providing nutrients to plants although they have some negative environmental effects.'],
    ['Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.', 'Studies have shown that higher CO2 levels lead to reduced plant uptake of nitrogen (and a smaller number showing the same for trace elements such as zinc) resulting in crops with lower nutritional value.'],
]
scores = model.predict(pairs)
print(scores)
# [0.7564 0.6164 0.567  0.083  0.8961]

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.',
    [
        'At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.',
        'Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.',
        'Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.',
        'Use of fertilizers are beneficial in providing nutrients to plants although they have some negative environmental effects.',
        'Studies have shown that higher CO2 levels lead to reduced plant uptake of nitrogen (and a smaller number showing the same for trace elements such as zinc) resulting in crops with lower nutritional value.',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

• Size: 18,858 training samples
• Columns: sentence1, sentence2, and label
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	label
  type	string	string	float
  details	min: 9 tokens mean: 26.48 tokens max: 54 tokens	min: 4 tokens mean: 33.89 tokens max: 475 tokens	min: 0.0 mean: 0.24 max: 1.0

• Samples:

  sentence1	sentence2	label
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	At very high concentrations (100 times atmospheric concentration, or greater), carbon dioxide can be toxic to animal life, so raising the concentration to 10,000 ppm (1%) or higher for several hours will eliminate pests such as whiteflies and spider mites in a greenhouse.	1.0
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Higher carbon dioxide concentrations will favourably affect plant growth and demand for water.	1.0
  Not only is there no scientific evidence that CO2 is a pollutant, higher CO2 concentrations actually help ecosystems support more plant and animal life.	Plants can grow as much as 50 percent faster in concentrations of 1,000 ppm CO 2 when compared with ambient conditions, though this assumes no change in climate and no limitation on other nutrients.	1.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• learning_rate: 1e-06
• weight_decay: 0.01
• num_train_epochs: 1
• warmup_steps: 0.1
• fp16: True

All Hyperparameters

Click to expand

• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 8
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-06
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: None
• warmup_steps: 0.1
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• enable_jit_checkpoint: False
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• use_cpu: False
• seed: 42
• data_seed: None
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: -1
• ddp_backend: None
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_for_metrics: []
• eval_do_concat_batches: True
• auto_find_batch_size: False
• full_determinism: False
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• use_cache: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss
0.0085	10	1.0417
0.0170	20	1.1143
0.0254	30	1.4366
0.0339	40	1.1697
0.0424	50	1.1378
0.0509	60	1.3185
0.0594	70	1.2562
0.0679	80	0.9840
0.0763	90	0.9804
0.0848	100	0.9873
0.0933	110	1.3078
0.1018	120	1.3021
0.1103	130	0.7902
0.1187	140	1.2104
0.1272	150	1.3424
0.1357	160	0.8652
0.1442	170	0.9543
0.1527	180	1.0987
0.1612	190	0.9230
0.1696	200	0.8114
0.1781	210	1.1211
0.1866	220	0.9010
0.1951	230	1.0351
0.2036	240	1.2003
0.2120	250	0.9269
0.2205	260	1.0230
0.2290	270	1.1586
0.2375	280	1.1140
0.2460	290	1.1156
0.2545	300	1.1983
0.2629	310	1.3360
0.2714	320	0.8958
0.2799	330	0.7516
0.2884	340	0.9000
0.2969	350	0.9625
0.3053	360	0.9588
0.3138	370	0.8129
0.3223	380	0.9421
0.3308	390	0.9417
0.3393	400	0.8714
0.3478	410	0.8628
0.3562	420	0.7971
0.3647	430	0.8672
0.3732	440	0.8361
0.3817	450	0.7054
0.3902	460	0.8236
0.3986	470	0.8694
0.4071	480	0.7517
0.4156	490	0.9431
0.4241	500	0.8135
0.4326	510	0.8570
0.4411	520	0.9081
0.4495	530	0.7691
0.4580	540	0.7911
0.4665	550	0.9490
0.4750	560	0.9187
0.4835	570	0.8568
0.4919	580	0.9042
0.5004	590	0.8659
0.5089	600	0.6557
0.5174	610	0.6866
0.5259	620	0.5841
0.5344	630	0.7899
0.5428	640	0.9242
0.5513	650	0.6446
0.5598	660	0.7487
0.5683	670	0.6401
0.5768	680	0.7829
0.5852	690	0.7574
0.5937	700	0.8913
0.6022	710	0.6415
0.6107	720	0.6815
0.6192	730	0.8599
0.6277	740	0.6971
0.6361	750	0.8237
0.6446	760	0.8097
0.6531	770	0.7816
0.6616	780	0.7718
0.6701	790	0.7392
0.6785	800	0.9359
0.6870	810	0.6208
0.6955	820	0.7195
0.7040	830	0.5577
0.7125	840	0.7267
0.7209	850	0.7275
0.7294	860	0.7157
0.7379	870	0.9963
0.7464	880	0.7719
0.7549	890	0.9122
0.7634	900	0.8108
0.7718	910	0.7258
0.7803	920	0.7794
0.7888	930	0.7896
0.7973	940	0.6627
0.8058	950	0.8455
0.8142	960	0.8049
0.8227	970	0.8466
0.8312	980	0.7153
0.8397	990	0.8069
0.8482	1000	0.6359
0.8567	1010	0.8079
0.8651	1020	0.8066
0.8736	1030	0.7665
0.8821	1040	0.7733
0.8906	1050	0.8064
0.8991	1060	0.7036
0.9075	1070	0.6739
0.9160	1080	0.8151
0.9245	1090	0.6668
0.9330	1100	0.6761
0.9415	1110	0.8783
0.9500	1120	0.7548
0.9584	1130	0.7671
0.9669	1140	0.8369
0.9754	1150	0.8658
0.9839	1160	0.7275
0.9924	1170	0.7911

Training Time

• Training: 1.1 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.0.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}