SentenceTransformer based on nlpaueb/legal-bert-base-uncased

This is a sentence-transformers model finetuned from nlpaueb/legal-bert-base-uncased. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: nlpaueb/legal-bert-base-uncased
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '7.  The applicants were born in 1945 and 1941 respectively and live in Gdańsk. 8.  The applicants share a plot of land and a house with another family (hereinafter: the neighbours). 9.  On 17 June 1987 they filed with the Gdańsk District Court (Sąd Rejonowy) a petition in which they requested that the co-ownership of the plot of land and the house be dissolved. 10.  On 19 April 1989 the court forbade the neighbours to carry out any modifications in the cellar until the completion of the proceedings. On 27 September 1989 a similar order concerning the whole house was issued in respect of the applicants. 11.  On 3 June 1992 the Gdańsk District Court gave judgment. The neighbours appealed and on 11 March 1993 the Gdańsk Regional Court quashed the judgment and remitted the case for re-examination. 12.  Subsequently, the District Court held a number of hearings and ordered several expert opinions. 13.  On 28 October 1996 it forbade the neighbours to carry out any works in the house. On 23 May 1997 the Regional Court dismissed their appeal against that decision. 14.  In its letter of 30 January 1997 the Ministry of Justice confirmed that the proceedings were lengthy and found that the applicants’ neighbours had contributed to the delay by their petitions contesting the expert opinions. It further noted that since 1994 the President of the Gdańsk Regional Court had supervised the course of the proceedings and made monthly reports on their progress. However, the Ministry found these measures ineffective and decided to take the proceedings under its administrative supervision. 15.  The proceedings are still pending.',
    'Right to a fair trial 1. In the determination of his civil rights and obligations or of any criminal charge against him, everyone is entitled to a fair and public hearing within a reasonable time by an independent and impartial tribunal established by law. Judgment shall be pronounced publicly but the press and public may be excluded from all or part of the trial in the interests of morals, public order or national security in a democratic society, where the interests of juveniles or the protection of the private life of the parties so require, or to the extent strictly necessary in the opinion of the court in special circumstances where publicity would prejudice the interests of justice. 2. Everyone charged with a criminal offence shall be presumed innocent until proved guilty according to law. 9 3. Everyone charged with a criminal offence has the following minimum rights: (a) to be informed promptly, in a language which he understands and in detail, of the nature and cause of the accusation against him; (b) to have adequate time and facilities for the preparation of his defence; (c) to defend himself in person or through legal assistance of his own choosing or, if he has not sufficient means to pay for legal assistance, to be given it free when the interests of justice so require; (d) to examine or have examined witnesses against him and to obtain the attendance and examination of witnesses on his behalf under the same conditions as witnesses against him; (e) to have the free assistance of an interpreter if he cannot understand or speak the language used in court.',
    'Right to marry Men and women of marriageable age have the right to marry and to found a family, according to the national laws governing the exercise of this right.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7361, 0.2542],
#         [0.7361, 1.0000, 0.3289],
#         [0.2542, 0.3289, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

  • Size: 12,180 training samples
  • Columns: sentence1, sentence2, and sentence3
  • Approximate statistics based on the first 1000 samples:
    sentence1 sentence2 sentence3
    type string string string
    details
    • min: 149 tokens
    • mean: 448.29 tokens
    • max: 512 tokens
    • min: 15 tokens
    • mean: 216.86 tokens
    • max: 422 tokens
    • min: 15 tokens
    • mean: 111.68 tokens
    • max: 422 tokens
  • Samples:
    sentence1 sentence2 sentence3
    11. At the beginning of the events relevant to the application, K. had a daughter, P., and a son, M., born in 1986 and 1988 respectively. P.’s father is X and M.’s father is V. From March to May 1989 K. was voluntarily hospitalised for about three months, having been diagnosed as suffering from schizophrenia. From August to November 1989 and from December 1989 to March 1990, she was again hospitalised for periods of about three months on account of this illness. In 1991 she was hospitalised for less than a week, diagnosed as suffering from an atypical and undefinable psychosis. It appears that social welfare and health authorities have been in contact with the family since 1989. 12. The applicants initially cohabited from the summer of 1991 to July 1993. In 1991 both P. and M. were living with them. From 1991 to 1993 K. and X were involved in a custody and access dispute concerning P. In May 1992 a residence order was made transferring custody of P. to X. 13. K. was again hospitalis... Right to respect for private and family life 1. Everyone has the right to respect for his private and family life, his home and his correspondence. 2. There shall be no interference by a public authority with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others. Prohibition of collective expulsion of aliens Collective expulsion of aliens is prohibited.
    9. In June 1949 plots of agricultural land owned by the applicant’s father were expropriated by the former Doksy District National Council (okresní národní výbor) under the Czechoslovak New Land Reform Act No. 46/1948 (“the 1948 Act”). The applicant’s father had never obtained any compensation. In 1957 some of these plots were transferred to the ownership of natural persons in an assignment procedure under the 1948 Act. In 1977 the applicant’s father died and the applicant’s rights over his estate were confirmed. 10. After the fall of the communist regime in Czechoslovakia, the Act No. 229/1991 on Adjustment of Ownership Rights in respect of Land and Other Agricultural Property (“zákon o půdě”, the “Land Ownership Act”) entered into force on 24 June 1991. The Act provided that the 1948 Act was no longer applicable and that under certain conditions property confiscated pursuant to that Act without compensation could be returned to its former owners or their heirs if it was still in th... Right to a fair trial 1. In the determination of his civil rights and obligations or of any criminal charge against him, everyone is entitled to a fair and public hearing within a reasonable time by an independent and impartial tribunal established by law. Judgment shall be pronounced publicly but the press and public may be excluded from all or part of the trial in the interests of morals, public order or national security in a democratic society, where the interests of juveniles or the protection of the private life of the parties so require, or to the extent strictly necessary in the opinion of the court in special circumstances where publicity would prejudice the interests of justice. 2. Everyone charged with a criminal offence shall be presumed innocent until proved guilty according to law. 9 3. Everyone charged with a criminal offence has the following minimum rights: (a) to be informed promptly, in a language which he understands and in detail, of the nature and cause of the acc... Right to liberty and security 1. Everyone has the right to liberty and security of person. No one shall be deprived of his liberty save in the following cases and in accordance with a procedure prescribed by law: (a) the lawful detention of a person after conviction by a competent court; (b) the lawful arrest or detention of a person for noncompliance with the lawful order of a court or in order to secure the fulfilment of any obligation prescribed by law; (c) the lawful arrest or detention of a person effected for the purpose of bringing him before the competent legal authority on reasonable suspicion of having committed an offence or when it is reasonably considered necessary to prevent his committing an offence or fleeing after having done so; (d) the detention of a minor by lawful order for the purpose of educational supervision or his lawful detention for the purpose of bringing him before the competent legal authority; (e) the lawful detention of persons for the prevention of the...
    8. In 1991 Mr Dušan Slobodník, a research worker in the field of literature, published an autobiography entitled Paragraph: Polar Circle. He described in it, inter alia, his conviction by a Soviet military tribunal in 1945 on the ground that he had been ordered to spy on the Soviet army after having been enrolled, in 1944 when he was 17 years old, in a military training course organised by Germans. In the book, Mr Slobodník also wrote about his detention in Soviet gulags and his rehabilitation by the Supreme Court of the Union of the Soviet Socialist Republics in 1960. In June 1992 Mr Slobodník became Minister for Culture and Education of the Slovak Republic. 9. On 20 July 1992 the newspaper Telegraf published a poem by the applicant. It was dated 17 July 1992 (the day when the sovereignty of the Slovak Republic was solemnly proclaimed) and entitled “Good night, my beloved” (“Dobrú noc, má milá”). One of its verses read as follows:
    “In Prague prisoner Havel is giving up his president... Freedom of expression 1. Everyone has the right to freedom of expression. This right shall include freedom to hold opinions and to receive and impart information and ideas without interference by public authority and regardless of frontiers. This Article shall not prevent States from requiring the licensing of broadcasting, television or cinema enterprises. 2. The exercise of these freedoms, since it carries with it duties and responsibilities, may be subject to such formalities, conditions, restrictions or penalties as are prescribed by law and are necessary in a democratic society, in the interests of national security, territorial integrity or public safety, for the prevention of disorder or crime, for the protection of health or morals, for the protection of the reputation or rights of others, for preventing the disclosure of information received in confidence, or for maintaining the authority and impartiality of the judiciary. Prohibition of torture No one shall be subjected to torture or to inhuman or degrading treatment or punishment. Have agreed as follows:
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 16
  • learning_rate: 2e-05
  • warmup_steps: 0.1
  • fp16: True

All Hyperparameters

Click to expand
  • do_predict: False
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 8
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 3
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: None
  • warmup_ratio: None
  • warmup_steps: 0.1
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • enable_jit_checkpoint: False
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • use_cpu: False
  • seed: 42
  • data_seed: None
  • bf16: False
  • fp16: True
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: -1
  • ddp_backend: None
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • group_by_length: False
  • length_column_name: length
  • project: huggingface
  • trackio_space_id: trackio
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • auto_find_batch_size: False
  • full_determinism: False
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_num_input_tokens_seen: no
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: True
  • use_cache: False
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss
0.6562 500 2.5875
1.3123 1000 2.3216
1.9685 1500 2.2555
2.6247 2000 2.1670

Training Time

  • Training: 44.4 minutes

Framework Versions

  • Python: 3.12.13
  • Sentence Transformers: 5.4.1
  • Transformers: 5.0.0
  • PyTorch: 2.10.0+cu128
  • Accelerate: 1.13.0
  • Datasets: 4.0.0
  • Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
104
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for spraxx/legal-bert-ecthr-biencoder

Finetuned
(105)
this model

Papers for spraxx/legal-bert-ecthr-biencoder