Update README.md

b662714 verified 1 day ago

10 kB

license: apache-2.0
base_model: microsoft/deberta-v3-base
library_name: transformers
pipeline_tag: text-classification
tags:
  - deberta-v3
  - human value detection
  - schwartz values
  - moral values
  - political text
  - retrieval augmented classification
  - rag
  - multi-label classification
model-index:
  - name: value-context-rag-deberta-v3-base-doc-rag
    results:
      - task:
          type: text-classification
          name: Multi-label text classification
        dataset:
          name: ValuesML / Touché24-ValueEval
          type: restricted
        metrics:
          - type: f1
            name: Macro-F1 (seed 1701)
            value: 0.3224
          - type: f1
            name: Micro-F1 (seed 1701)
            value: 0.3617
language:
  - en
metrics:
  - f1

Value Context RAG - DeBERTa-v3-base Document RAG

This model is the representative checkpoint for the best-performing supervised encoder condition from:

More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts
Víctor Yeste, Paolo Rosso (2026), arXiv:2605.22641, submitted to ARR May 2026 / EMNLP 2026.

It is a multi-label classifier over the 19 refined Schwartz values. The model was trained with document context and early-fusion retrieval-augmented classification: the input contains a target sentence marked inside its document context, followed by two retrieved moral-knowledge snippets.

This checkpoint corresponds to:

Base model: microsoft/deberta-v3-base
Context condition: document context
Retrieval: enabled, early fusion
Retrieval top-k: 2 KB chunks
Seed: 1701
Recommended global threshold: 0.18

The paper reports this condition as mean±standard deviation over three seeds (7, 42, 1701). This uploaded checkpoint is the seed-1701 run of the best aggregate condition.

Intended Use

This model is intended for research on:

Schwartz value detection
Moral and value-aware NLP
Political text analysis
Retrieval-augmented text classification
Multi-label classification under label imbalance

The model should be used as a noisy research classifier, not as an authoritative source of a person's values or beliefs.

Do not use this model for:

Individual-level profiling
Automated moderation
Surveillance
Ranking political actors or viewpoints
High-stakes decision-making

Labels

The model predicts the 19 refined Schwartz values:

Self-direction: thought
Self-direction: action
Stimulation
Hedonism
Achievement
Power: dominance
Power: resources
Face
Security: personal
Security: societal
Tradition
Conformity: rules
Conformity: interpersonal
Humility
Benevolence: caring
Benevolence: dependability
Universalism: concern
Universalism: nature
Universalism: tolerance

The original benchmark distinguishes attained and constrained values. For this model, both variants are collapsed into a single binary value-presence label.

Input Format

The model was trained with early-fusion RAG inputs of the form:

TEXT:
... previous document sentence. <TGT>target sentence</TGT> following document sentence ...

KNOWLEDGE:
retrieved moral knowledge snippet 1

retrieved moral knowledge snippet 2

For exact reproduction of the paper condition, use the accompanying GitHub repository to build document contexts, retrieve the top-2 KB chunks, and apply the same token budgets. The released HF model contains the classifier and tokenizer, but not the restricted benchmark texts.

The model can still be used with manually provided document context and knowledge snippets, but performance may differ from the paper if the input format or retrieval setup changes.

How to Use

This is a standard Transformers sequence-classification model. It does not require trust_remote_code=True.

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "VictorYeste/value-context-rag-deberta-v3-base-doc-rag"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

id2label = {int(k): v for k, v in model.config.id2label.items()}

def build_doc_rag_input(document_sentences, target_index, kb_snippets):
    """Build a simplified version of the paper's document+RAG input format."""
    marked_sentences = []
    for i, sentence in enumerate(document_sentences):
        if i == target_index:
            marked_sentences.append(f"<TGT>{sentence}</TGT>")
        else:
            marked_sentences.append(sentence)

    document = " ".join(marked_sentences)
    knowledge = "\n\n".join(kb_snippets[:2])
    return f"TEXT:\n{document}\n\nKNOWLEDGE:\n{knowledge}"

def predict_values(text, threshold=0.18):
    enc = tokenizer(
        text,
        return_tensors="pt",
        truncation=True,
        max_length=1024,
    )

    with torch.no_grad():
        logits = model(**enc).logits.squeeze(0)
        probs = torch.sigmoid(logits).cpu()

    scores = {id2label[i]: float(probs[i]) for i in range(len(id2label))}
    labels = [label for label, score in scores.items() if score >= threshold]

    return {
        "labels": labels,
        "scores": scores,
    }

document = [
    "The city council debated a new climate adaptation plan.",
    "The proposal aims to protect rivers, parks, and future generations.",
    "Opponents argued that the cost would be too high for local businesses.",
]

kb_snippets = [
    "Universalism: nature concerns preservation of the natural environment and protection of nature.",
    "Universalism: concern concerns equality, justice, and protection for all people.",
]

model_input = build_doc_rag_input(
    document_sentences=document,
    target_index=1,
    kb_snippets=kb_snippets,
)

print(predict_values(model_input, threshold=0.18))

If no labels pass the threshold, treat the prediction as NONE.

Training Data

The model was trained on the English ValuesML / Touché24-ValueEval human value detection benchmark.

Domain: political and socially oriented texts
Prediction unit: target sentence
Context: document-level context reconstructed with text_id and sent_id
Labels: 19 refined Schwartz values
Label formulation: attained/constrained annotations collapsed into value presence

Important: the original dataset is distributed under a restricted Data Usage Agreement. This model repository does not redistribute raw dataset text. To reproduce training or evaluation, users must obtain the official dataset separately and comply with its terms.

Training Setup

Base model: microsoft/deberta-v3-base
Architecture: DebertaV2ForSequenceClassification
Task: 19-label multi-label classification
Objective: binary cross-entropy with logits
Input condition: document context + early-fusion RAG
RAG top-k: 2 moral-knowledge chunks
KB budget: 200 tokens
Maximum sequence length: 1024
Batch size: 8
Gradient accumulation steps: 2
Learning rate: 1e-5
Weight decay: 0.15
Maximum epochs: 20
Early stopping: validation macro-F1
Training precision: fp32
Gradient checkpointing: enabled
Seed: 1701
Decision threshold: 0.18

The moral KB contains short, paraphrased chunks based on Schwartz value definitions, annotation guidance, and theory-level contrasts. The repository associated with the paper documents the KB construction and retrieval setup.

Performance

On the held-out test split used in the paper, this specific seed-1701 checkpoint achieves:

Model condition	Macro-F1	Micro-F1
DeBERTa-v3-base doc + early RAG, seed 1701	0.3224	0.3617

The paper reports the same condition as an aggregate over three seeds:

Model condition	Macro-F1	Micro-F1
DeBERTa-v3-base doc + early RAG, seeds 7/42/1701	0.314 ± 0.008	0.369 ± 0.010

This was the strongest aggregate condition among the tested supervised encoders and zero-shot LLMs in the paper.

Limitations

The model is trained and evaluated on one benchmark and one broad genre: political and socially oriented text.
It may not generalize to social media, dialogue, literary text, legal text, other languages, or non-political domains.
Value labels are sparse and imbalanced; rare labels such as Humility and Conformity: interpersonal remain difficult.
The model depends on the input format. Using sentence-only inputs, missing KB snippets, or different retrieval chunks can change predictions.
The retrieved KB can shape predictions through its wording.
No systematic fairness audit has been conducted.
Predictions should be treated as uncertain analytical signals, not ground truth about people or groups.

License

The model artifacts in this repository are released under the Apache License 2.0.

This license does not grant rights over:

The underlying ValuesML / Touché24-ValueEval dataset
Third-party copyrighted text in the benchmark
The upstream microsoft/deberta-v3-base model beyond its own license terms

Users must obtain and use the original dataset under its own Data Usage Agreement.

Citation

If you use this model, please cite the associated paper:

@misc{yeste2026contextlargermodelsmoral,
  title={More Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts}, 
  author={Víctor Yeste and Paolo Rosso},
  year={2026},
  eprint={2605.22641},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.22641}, 
}

Please also cite the dataset/shared-task resources where appropriate:

@misc{ValueEval24Zenodo,
  author    = {{The ValuesML Team}},
  title     = {Touch{\'e}24{-}ValueEval},
  year      = {2024},
  month     = {8},
  version   = {2024-08-09},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.13283288},
  url       = {https://doi.org/10.5281/zenodo.13283288}
}