SentenceTransformer based on hkunlp/instructor-xl

This is a sentence-transformers model finetuned from hkunlp/instructor-xl. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: hkunlp/instructor-xl
Maximum Sequence Length: 128 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'T5EncoderModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': False})
  (2): Dense({'in_features': 1024, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ahmedHamdi/ir-pt-en-instructor-xl")
# Run inference
sentences = [
    'Represent the plot: Mortimer and his daughter Meggie share a passion for books and travel. He has a gift for bringing the characters and stories from books to life, so much so that his wife disappeared nine years earlier after reading Inkheart, when the villains from the book jumped into the real world. Mortimer needs to bring his wife back, but at the same time he has to protect his daughter and return the villains to the pages of the book.',
    "Represent the plot: Mo and Resa Folchart read the fairy tale Little Red Riding Hood to their baby daughter Meggie, as a red velvet hood appears out of thin air. Twelve years later, Meggie visits an old book shop in Europe with her father, unaware that he is secretly looking for a copy of the book Inkheart. Shortly after Mo finds the book, Meggie encounters a horned marten outside the shop and it tries to bite her fingers. A man appears from the shadows, claiming to be an old friend. Mo comes out of the shop, recognizing the man as Dustfinger, who quickly asks him to read him back into the book. Mo flees with Meggie. Mo takes Meggie to visit her great-aunt Elinor in Italy. There he tells Meggie that nine years ago, while reading to her from Inkheart, he inadvertently brought Dustfinger out of the book and into the real world, through a gift he possessed from birth. Unfortunately, the book's villain, Capricorn, and one of his henchmen, Basta, were also brought out of the book. Fleeing with his infant daughter, Mo realizes Resa has been taken into the book. Capricorn arrives with Basta, who captures the group, destroys Elinor's library and takes Inkheart. Mo, Meggie, and Elinor are taken to Capricorn's new castle in the real world. During their imprisonment, Mo explains his gift to Meggie and Elinor, stating that when he reads a person or an object out of a book, someone from the real world is sent into it, hence his wife's disappearance. Capricorn forces Mo to use his gift, acquiring treasure from one of the stories in The Arabian Nights, and imprisoning Farid, one of the 40 Thieves. Dustfinger, who was promised a return into the book, is betrayed by Capricorn. The group escapes using the tornado from The Wonderful Wizard of Oz. Elinor leaves to recover what is left of her library as the others head for Alassio, the town of Inkheart author Fenoglio, who may have another copy. Dustfinger is initially afraid of learning his fate in the book, but eventually joins them. When they meet Fenoglio, the author's ecstatic reaction to seeing his creation alive leads Dustfinger to learn he dies at the end of the book while trying to save his marten, Gwin. He berates Fenoglio before eventually telling Mo that Resa is trapped in the real castle and has lost her voice. Mo and Dustfinger take Fenoglio's car and Farid stows away in the trunk. At Fenoglio's house, Meggie begins reading out loud, bringing Toto from The Wizard of Oz. Basta bursts in with his men and takes the group to Capricorn. Elinor, having decided to stay, realizes something is amiss and pursues them. At the castle, Capricorn orders Meggie to bring a monster called the Shadow out of Inkheart and threatens to harm Resa if she does not comply. Mo sneaks into the castle. Dustfinger is captured, but escapes after Meggie informs him of Capricorn's intentions. Mo attempts to free Meggie and the others, as Capricorn forces her to read the Shadow out of the book. Dustfinger returns to the castle with Farid and sets it on fire. The distraction allows Fenoglio to give Meggie a rewritten part of Inkheart that should help Meggie stop Capricorn's plan. As Elinor arrives with the creatures Capricorn imprisoned, Meggie writes out the story on her arm. As she reads it out, Capricorn turns into ashes as the Shadow devours him, his henchmen vanish, the Shadow explodes, and all the read-out creatures are returned where they belong. In addition, she grants Fenoglio's wish to live in the world he created, while reuniting herself and Mo with Resa, whose voice is restored. Dustfinger is also safely read back into Inkheart, where he is reunited with his wife Roxanne. In the real world, Farid reveals he kept Gwin with him, thus allowing Dustfinger to avoid his own death. As Mo and his family leave with Farid, Meggie agrees to teach him how to read and Farid agrees to teach her how to use the dragon breath, the fire breathing ability of Dustfinger's which Farid learned.",
    "Represent the plot:  Eight years after the death of Gotham City District Attorney Harvey Dent, Bane, a former member of the League of Shadows, leads an attack on a CIA plane over Uzbekistan to abduct nuclear physicist, Dr. Leonid Pavel, and fake Pavel's death in the crash. Meanwhile, organized crime has been eradicated in Gotham by legislation, the Dent Act, giving expanded powers to the police. James Gordon, police commissioner, has kept Dent's killing spree a secret and allowed the blame for his crimes to fall on Batman. Bruce Wayne, still mourning the death of Rachel Dawes, has become a recluse, and Wayne Enterprises has stagnated. Bane enlists businessman John Daggett to buy Bruce's fingerprints. Cat burglar Selina Kyle steals Bruce's prints from Wayne Manor for Daggett, but he double-crosses her and she alerts the police, who pursue Bane and Daggett's henchmen into the sewers while Kyle flees. The henchmen capture Gordon and take him to Bane, but he escapes and is found by Officer John Blake, an orphan who has deduced Bruce's secret identity and persuades him to resume his vigilantism. Bane attacks the Gotham Stock Exchange and uses Bruce's fingerprints to verify a series of fraudulent transactions, leaving Bruce bankrupt; Bane escapes while the police chase Batman. Fearing Bruce will get himself killed fighting Bane, his butler, Alfred Pennyworth, resigns in the hope of saving him after admitting to burning a letter that Rachel left for him saying she was going to marry Dent. Bane expands his operations and kills Daggett while Bruce and Wayne Enterprises's new CEO, Miranda Tate, become lovers. Kyle agrees to take Batman to Bane but instead leads him into a trap under Wayne Tower. Bane gloats that he intends to fulfill Ra's al Ghul's mission to destroy Gotham City before crippling Batman in combat. He then takes Bruce to an ancient underground prison in the Middle East he calls home, where Bruce learns that Ra's al Ghul's child was born and raised in the prison but had a protector that aided the child in escaping, thought to be impossible. Back in Gotham, Bane traps the police forces in the sewers, destroys all but one bridge surrounding the city, kills Mayor Anthony Garcia, forces Pavel to convert a fusion reactor core into a decaying neutron bomb before killing him, exposes Dent's crimes to the city (thus undermining the legal system), releases the prisoners of Blackgate Penitentiary, takes over the city, and has Gotham's elite exiled and killed in proletarian kangaroo courts presided over by Jonathan Crane. Bruce recovers from his injuries five months later, escapes captivity, and returns to Gotham. As Batman, he frees the police, and together, they clash with Bane's army in the streets. During the battle, Batman overpowers Bane, but Tate stabs Batman, revealing herself as Talia al Ghul, Ra's al Ghul's child. Unable to activate the bomb's detonator due to Gordon blocking the signal, Talia leaves to find the bomb. Kyle returns, killing Bane and helping Batman pursue Talia, hoping to bring the bomb back to the reactor chamber where it can be stabilized. Talia's truck crashes, but she remotely floods and destroys the reactor chamber before dying. With no way to stop the detonation, Batman, after revealing his identity to Gordon, uses his aerial craft, the Bat, to haul the bomb far over the bay, where it safely explodes. In the aftermath, Batman is presumed dead and honored as a hero. Wayne Manor becomes an orphanage, and Bruce's estate is left to Alfred. Gordon finds the Bat Signal repaired, while Lucius Fox discovers that Bruce had fixed the Bat's malfunctioning auto-pilot. Blake, whose legal first name is revealed as Robin, resigns from the GCPD and receives a package leading him to the Batcave. In Florence, Alfred discovers that Bruce is alive and in a relationship with Kyle, and they silently acknowledge each other before parting ways.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.6825, -0.1307],
#         [ 0.6825,  1.0000, -0.1595],
#         [-0.1307, -0.1595,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 5,800 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1
type string string
details
min: 19 tokens
mean: 98.78 tokens
max: 128 tokens

min: 20 tokens
mean: 125.63 tokens
max: 128 tokens

	sentence_0	sentence_1
type	string	string
details	min: 19 tokens mean: 98.78 tokens max: 128 tokens	min: 20 tokens mean: 125.63 tokens max: 128 tokens

Samples:

sentence_0	sentence_1
`Represent the plot: An account of the events that took place in 1943, when Nazi commander Heinrich Himmler was tasked with kidnapping the first British Prime Minister, Winston Churchill, who was resting in a small village.`	Represent the plot: Admiral Canaris, head of the Abwehr, is ordered by Adolf Hitler to make a feasibility study into capturing the British Prime Minister, Winston Churchill. Canaris considers it a meaningless exercise that will soon be forgotten by the Führer, but he knows this will not be the case with Heinrich Himmler. Canaris therefore orders staff officer Oberst Radl to begin the study, to avoid being discredited. Radl receives intelligence from an Abwehr sleeper agent in England, saying Churchill will stay in a Norfolk village near the coast. He begins to see potential in the operation, which he code-names Eagle. Firstly, Radl recruits an agent, an IRA man named Liam Devlin, who lectures at a Berlin university. Secondly, he selects Kurt Steiner, a highly decorated and experienced Fallschirmjäger officer, to lead the mission. However, while the Luftwaffe parachute troops are returning from the Eastern Front, Steiner unsuccessfully attempts to save the life of a Jewish girl who is ...
`Represent the plot: Caterina is a 13-year-old girl, the daughter of a teacher and a housewife, whose family moves from Montalto di Castro to Rome.`	Represent the plot: Caterina (Alice Teghil) is the 13-year-old only child of Giancarlo Iacovoni (Sergio Castellitto), an aspiring novelist and teacher of accounting at a country school in an area north of Rome that one character describes as hillbilly country. In spite of his often lucid assessments of modern society, Iacovoni is a typically burned-out teacher whom his job (perhaps among other, undisclosed personal experiences) has imbued with bitterness and social resentment, with a sheer lack of perspective concerning human relations, and with an overbearing, holier-than-thou demeanor that is a major plot point throughout the movie. He relocates his daughter Caterina and his timid, long-suffering wife Agata (Margherita Buy) to his birthplace, Rome, after having finally secured a long-coveted teaching position. The family settles in Giancarlo's former district, where he reconnects with his neighbor and childhood friend, Fabietto (Silvio Vannucci). Once settled in the Italian capital,...
`Represent the plot: In the Catfish Row slum, the disabled Porgy loves the beautiful, dangerous, and drug-addicted Bess. Bess has other suitors, including Crown, a muscular laborer, and the sophisticated drug dealer Sportin' Life. After killing a rival, Crown flees, and Bess, not wanting to be alone, settles down with Porgy. When Crown reappears, Porgy kills him and tries to hide from the police, while the fickle Bess follows Sportin' Life, seeking the bright lights of New York. Main awards`	Represent the plot: Set in the early 1900s in the fictional Catfish Row section of Charleston, South Carolina, which serves as home to a black fishing community, the story focuses on the title characters: crippled beggar Porgy, who travels about in a goat-drawn cart, and the drug-addicted Bess, who lives with stevedore Crown, the local bully. While high on cocaine supplied by Sportin' Life, Crown kills Robbins after the latter vanquishes him in a craps game; Bess urges Crown to flee. Sportin' Life suggests she accompany him to New York City, an offer Bess declines. She seeks refuge with her neighbors, all of whom refuse to help her. Porgy finally agrees to let her stay with him. Bess and Porgy settle into domestic life together and soon fall in love. Just before a church picnic on Kittiwah Island, Sportin' Life once again approaches Bess, but Porgy warns him to leave her alone. Bess wishes to stay with Porgy, since he cannot attend the picnic because of his disability, but he urges her...

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false
}

Training Hyperparameters

Non-Default Hyperparameters

multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: no
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 3
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.6897	500	0.062
1.3793	1000	0.0218
2.0690	1500	0.0102
2.7586	2000	0.0037

Framework Versions

Python: 3.9.21
Sentence Transformers: 5.1.2
Transformers: 4.57.6
PyTorch: 2.8.0+cu128
Accelerate: 1.10.1
Datasets: 4.5.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}