ahmedHamdi/narrative-similarity-all-en-masked-instructor-xl

SentenceTransformer based on hkunlp/instructor-xl

This is a sentence-transformers model finetuned from hkunlp/instructor-xl. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: hkunlp/instructor-xl
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False, 'architecture': 'T5EncoderModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': False})
  (2): Dense({'in_features': 1024, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("ahmedHamdi/narrative-similarity-all-en-masked-instructor-xl")
# Run inference
sentences = [
    'Represent the plot: PERSON (PERSON) is a lazy, middle-aged petty criminal living in GPE, GPE. He works there as a ticket scalper, approaching tourists with offers of cheap souvenirs and his services as a tour guide. It is in this way that he meets PERSON (PERSON) and his companion, PERSON (Melina Mercouri). They are actually high-level thieves planning to steal a priceless piece from ORG at Topkapi Palace in GPE, GPE: the priceless gold dagger, adorned with emeralds, diamonds, and pearls, that once belonged to Sultan Mahmud I and is displayed in a secure glass case. Harper hires PERSON as his driver to take his luxury convertible to GPE and drop him off at his hotel. Simpson arrives at the border, and the car is searched by Turkish customs officials, who find a rifle and smoke grenades hidden inside—items that would be used to cover the thieves\' escape. PERSON is arrested and handed over to the Turkish secret police, suspected of being a terrorist involved in an assassination attempt. The police eventually agree to let him go, on the condition that he become an informant, spying on PERSON and everyone ORG contacts in GPE. ORG has no choice but to accept. Once in GPE, he meets with ORG, and the rest of the gang of criminals who will participate in the robbery: PERSON (PERSON), an alarm expert; PERSON (PERSON), the strongman; and PERSON "the human fly" (Gilles Ségal), an acrobat. After completing his mission as a spy, with meager results, ORG eventually becomes, by chance, a member of the gang. Even after he finally confesses his role as an informant, PERSON accepts him out of necessity, as he has to replace one of the robbers. Unaware of the plan, on the day of the heist, ORG is forced to climb onto the museum roof, despite suffering from vertigo, making him long for his life in GPE more than ever. The heist is carried out, but with an unexpected outcome.',
    "Represent the plot:  PERSON (Melina Mercouri) visits GPE, where she sees a traveling fair featuring replicas of treasures from the Topkapı Palace.  Next she cases the ORG, fascinated by the emerald-encrusted dagger of Sultan PERSON, she recruits her ex-lover, Swiss master-criminal PERSON (PERSON), to plan a  theft of the dagger. They engage PERSON (PERSON), master of all things mechanical; PERSON, ORG), a mute acrobat; and the burly PERSON (PERSON), who will provide the muscle needed for the job. Harper and Lipp then hire small-time hustler PERSON (PERSON) to drive a car into GPE to transport hidden explosives and firearms for use in the burglary. ORG, knowing nothing of ORG's and PERSON's plans, is arrested at the border when ORG find the firearms.  Because ORG has no information for Turkish police, they conclude that the weapons are to be used in an assassination. Turkish Major PERSON decides to use ORG to spy on ORG for the police. Page, picking up the car in GPE, is told by a police ruse that only the importer PERSON is permitted to drive it in GPE. While traveling with the gang, PERSON leaves cryptic notes for his police handlers, but most of his intelligence is worthless since ORG is still ignorant of the plan. PERSON' hands are injured in a scuffle with the drunken cook, Gerven (PERSON), and ORG is engaged as a substitute, prompting him to confess that the police are watching them. Knowing they face arrest if they try to escape GPE, or use their equipment, ORG improvises a new plan in which they will give the still-oblivious police the slip, and steal the dagger without using their weapons.  Then they'll surrender to the police, and claim to have found explosives in their car.  Just before they leave, ORG discards his last note, then leaves with the others. Harper arranges to give the police the slip. That evening, PERSON, PERSON, and PERSON, after attending a competition of Turkish wrestling, steal the dagger and leave a replica in its place. Unnoticed by the thieves, during the robbery a bird flies through the window they entered by and is trapped inside the room when the window is closed. The gang deliver the dagger to PERSON (PERSON), proprietor of the traveling fair display, who will smuggle it out of the country. The gang members then go to police headquarters to reveal their discovery of weapons in the car. The inspector asks ORG to vouch for ORG whereabouts that day. PERSON, seeming to waver, throws in his lot with the others, and backs up their alibi.  Before the police release PERSON and the others, the trapped bird in the ORG triggers the alarm, alerting police officers across GPE.  When word of the Topkapı alarm reaches the police, Major PERSON confronts the thieves, displaying ORG's last note, which has just enough information to link all of them to the theft.  PERSON tells them all that he now knows why they were in GPE.  A little bird told me, he says. Ultimately, the gang is seen in a Turkish prison, where PERSON begins to tell them of her fascination with ORG ORG in the ORG. The end title sequence shows them apparently having escaped from jail some time later and walking in snow by a Russian city.",
    "Represent the plot: On Christmas Eve in GPE, GPE, PERSON runs into rising pop star PERSON on a train. He mistakes her for a fan when she tries to return his phone to him, to which she takes offense, despite his apologies. After the train is stopped by snowed-over tracks, PERSON gets off to walk home, PERSON joins her and offers lunch at a local diner called ORG. She reluctantly agrees and later has to save him from a group of fangirl cheerleaders. PERSON reveals to PERSON she got accepted to ORG and her mom is also deathly sick, but if she delays leaving, she will lose the scholarship. They go sledding and meet PERSON's mother, ORG. Going to Julie's, ORG is introduced to her grandpa and they bond over PERSON. Everyone dances to one of his songs. While dancing, ORG has a coughing fit, causing PERSON concern. Stuart offers to get a nurse for her mom, offending her. He dismisses her concerns, stating he offered it because he cares about PERSON. They almost kiss before ORG's publicist shows up to take him back to his hotel. He asks PERSON to come with him, but she declines, feeling a need to stay with her mother. Across town, ORG employee Dorrie tries to juggle her best friend, PERSON, who is worried that her boyfriend is going to break up with her, and PERSON, a cheerleader Dorrie hooked up with. PERSON visits the diner with her friends and pretends not to know Dorrie; ORG makes a scene with her boyfriend in the diner. Dorrie tries to calm PERSON down, but she storms off. Later, Dorrie and PERSON meet in the bathroom, kissing passionately. Dorrie tells PERSON that PERSON will probably be at the party. Meanwhile, GPE plans to tell his best friend, PERSON, 'The Duke', he likes her. They are invited to a party by ORG's friend, ORG, whom GPE sees as competition. The three steal the keg from the party, ending up in a ditch. Waiting for the tow truck in a nearby church, ORG has GPE play Whole of the PERSON, but when ORG and Duke start dancing together, GPE leaves. Duke attempts to talk to him about it, but he brushes her off, angering her. GPE's friend, PERSON, attempts to arrange a party to impress a big time DJ at his house, but his parents shut down the party attempt and he gets called into work at ORG. His friend and fellow co-worker, PERSON, offers ORG as a party spot, provided he get the necessary supplies. Back at home, PERSON's mom convinces her to go to GPE, stating when life offers you something special, you take it. Everyone ends up at ORG's party after GPE arrives with the stolen keg. PERSON comes and GPE confesses his love for her, to which she admits the same. Addie returns to ORG and apologizes to Dorrie for her rude behavior. PERSON also apologizes to her for being hot and cold, kisses her in the open, and they begin a relationship. Dorrie and PERSON are telling each other about their day when PERSON returns, wanting to see PERSON again before he leaves town. They kiss and make plans to meet in GPE when she's at ORG. PERSON's party ends up a success, despite the DJ not coming, and they dance all night long.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000,  0.5635,  0.0321],
#         [ 0.5635,  1.0000, -0.0145],
#         [ 0.0321, -0.0145,  1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 28,376 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1
  type	string	string
  details	min: 9 tokens mean: 99.35 tokens max: 128 tokens	min: 7 tokens mean: 124.1 tokens max: 128 tokens

• Samples:

  sentence_0	sentence_1
  Represent the plot: PERSON is a cold-blooded corporate liquidator determined to take down the company run by patriarch PERSON. Knowing his company is targeted, PERSON hires his daughter, a brilliant lawyer, to keep an eye on PERSON and prevent him from gaining control.	Represent the plot: PERSON the Liquidator PERSON (PERSON) is a successful corporate raider who has become rich buying up companies and selling off their assets. With the help of a computerized stock analyzing program called GPE, PERSON has identified ORG as his next target. The struggling company is run by the benevolent and folksy PERSON (Gregory Peck) and is the primary employer in its small GPE town in LOC. PERSON decides to take over the company. After PERSON learns that he has filed a Schedule 13D report, and after stubbornly insisting that no outsider can seize control of a business his father began, PERSON is finally persuaded to hire his stepdaughter PERSON (Penelope PERSON), a big-city lawyer, to defend against a hostile takeover. PERSON is instantly smitten with the beautiful PERSON, although he is on to her tactics and does not waver from his goal of becoming the majority stockholder of ORG. PERSON tactlessly and unsuccessfully tries to seduce her. Despite their antagonism, ...
  Represent the plot: GPE in 1972, after receiving a distress call from soldiers missing for six months, a group of South Korean soldiers (allied with the GPE) is sent to point R to investigate. But this area has a very bad reputation...	Represent the plot: In January 1972, at a South Korean base in GPE, GPE, a missing platoon sends a radio transmission from R-Point, a strategic island near GPE. Lieutenant ORG receives orders to lead a squad, including PERSON, on a mission to extract the missing soldiers within a week. In return, they will receive an early honorable discharge and a clean record for ORG, who had previously shown insubordination. Upon arrival at ORG, the squad is ambushed by a Vietnamese woman and discovers a week-old corpse. The next day, a mysterious, deserted French Plantation appears near their base. Corporal PERSON gets separated from the group and encounters other soldiers, mistaking them for his unit. When he reunites with his comrades, he shares his sighting of the missing soldiers, but they dismiss his claims. Later, while repairing the radio, Corporal PERSON intercepts a transmission from a nearby French unit. A French army corporal named PERSON claims to have a twin brother named PERSON, raisi...
  Represent the plot: PERSON is a young scientist clinging to life on LOC, which is devastated and on the brink of death. When she is told she must travel as soon as possible to GPE, a temporary space colony housing most of the world's surviving population, Micah unexpectedly appears—a man who has inexplicably survived on LOC and who will try to help PERSON reach the colony.	Represent the plot: The film is set in a post-apocalyptic present, where LOC's atmosphere has become toxic. Most humans have fled the planet, to live on a space station near LOC, a moon of LOC. PERSON is one of the few humans remaining on LOC. She lives alone at high altitude, where the air is still breathable, trying to raise bees that can survive in the atmosphere. Her intent is to use the bees to eventually clean the air via the pollination of oxygen-producing plants. Her long-distance boyfriend, who lives on the LOC station, urges her to leave LOC on the last shuttle from the planet. A storm creates a toxic cloud that passes through PERSON's shelter, killing all of her bees. Micah, a man travelling by helium balloon, arrives shortly thereafter. He intends to reach the launch site and leave LOC, but first wants to speak with Dr. PERSON, PERSON's father, who had urged mankind to stay because he still saw hope. Initially, PERSON states that her father is conducting field work elsewher...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.1410	500	0.2915
0.2819	1000	0.1936
0.4229	1500	0.1997
0.5639	2000	0.1787
0.7048	2500	0.1926
0.8458	3000	0.1638
0.9867	3500	0.1613
1.1277	4000	0.1011
1.2687	4500	0.1134
1.4096	5000	0.0955
1.5506	5500	0.1045
1.6916	6000	0.1025
1.8325	6500	0.1004
1.9735	7000	0.1054
2.1145	7500	0.0524
2.2554	8000	0.0534
2.3964	8500	0.0493
2.5374	9000	0.062
2.6783	9500	0.0565
2.8193	10000	0.0508
2.9602	10500	0.0503

Framework Versions

• Python: 3.9.21
• Sentence Transformers: 5.1.2
• Transformers: 4.57.6
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}