illuminator22/openbrief-nano-qwen35-entity-extractor

OpenBrief Nano — Legal Entity Extraction Model

A LoRA fine-tuned adapter on top of Qwen3.5-9B for extracting structured entities from legal contract text. Built as part of OpenBrief, an open-source legal document intelligence platform by BrainX Corp.

What It Does

Given a chunk of legal contract text, OpenBrief Nano returns structured JSON with 8 entity types:

• party — named legal entities, organizations, persons
• date — agreement, effective, expiration dates (with subtypes)
• deadline — notice periods, renewal terms, payment windows (with subtypes)
• jurisdiction — venue, forum, arbitration seat
• obligation — duties imposed on a party
• monetary_amount — dollar values, liability caps, liquidated damages (with subtypes)
• governing_law — applicable law
• termination_right — rights to terminate (with subtypes)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch, json

# Load base model + adapter
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-9B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, "illuminator22/openbrief-nano-qwen35-entity-extractor")
model.eval()

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-9B", trust_remote_code=True)

# Build prompt
chunk = "Your legal text here..."
prompt = (
    "<|im_start|>system\n"
    "You are OpenBrief Nano, a legal entity extraction model. "
    "Return only valid JSON. Do not include explanations, commentary, or hidden reasoning.<|im_end|>\n"
    "<|im_start|>user\n"
    "Extract entities from this legal text:\n\n"
    f"{chunk}<|im_end|>\n"
    "<|im_start|>assistant\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False,
                              eos_token_id=tokenizer.eos_token_id,
                              pad_token_id=tokenizer.pad_token_id)

gen_tokens = outputs[0][inputs["input_ids"].shape[1]:]
result = json.loads(tokenizer.decode(gen_tokens, skip_special_tokens=True))
print(json.dumps(result, indent=2))

Evaluation: OpenBrief Nano vs GPT-5.4 Nano

Test Setup

• 50 chunks from 51 held-out CUAD contracts (contract-level split, no data leakage)
• 25 chunks manually reviewed with gold labels for per-type accuracy
• Both models evaluated on identical chunks

Overall Metrics

Metric	OpenBrief Nano	GPT-5.4 Nano (prompted)
Valid JSON rate	100%	100%
Total entities extracted (45 comparable chunks)	407	386

Per-Type Extraction Volume (45 comparable chunks)

Entity Type	OpenBrief Nano	GPT-5.4 Nano	Winner
party	196	146	Nano
date	67	57	Nano
deadline	30	13	Nano
jurisdiction	1	13	GPT-Nano
obligation	94	132	GPT-Nano
monetary_amount	6	12	GPT-Nano
governing_law	3	3	Tie
termination_right	10	10	Tie

Per-Type Accuracy (25 gold-reviewed chunks)

Based on manual expert review of 25 held-out chunks, comparing each model's output against human-verified gold labels:

Entity Type	Nano Wins	GPT-Nano Wins	Tie	Assessment
party	12	4	9	Nano stronger — finds more real parties, occasionally over-extracts
date	5	3	17	Nano slightly better — sometimes over-extracts temporal references
deadline	7	1	17	Nano clearly stronger — catches notice periods and payment windows GPT misses
jurisdiction	0	5	20	GPT-Nano stronger — Nano's biggest weakness
obligation	2	6	17	GPT-Nano stronger — finds more, but sometimes misclassifies rights as duties
monetary_amount	3	2	20	Roughly even — both sparse
governing_law	0	0	25	Tied — both correct when present
termination_right	5	2	18	Nano stronger — catches termination rights GPT misses

Key Findings

OpenBrief Nano wins on: party extraction (12-4), deadline detection (7-1), termination rights (5-2). These are high-frequency, CUAD-anchored entity types where training data was strongest.

GPT-5.4 Nano wins on: jurisdiction (0-5), obligation volume (2-6). Jurisdiction was fully GPT-assisted during training with limited examples. Obligation extraction finds fewer entities but with better party attribution.

Both models struggle with: jurisdiction in general (sparse in contracts), monetary amounts (often redacted in CUAD source data).

Qualitative observations:

• Nano produces better party attribution in obligations (uses defined party names)
• GPT-Nano occasionally extracts signature blocks as obligations and documents as parties
• Nano occasionally over-extracts section titles and generic temporal references as entities
• Both models correctly return empty arrays when no entities are present

Cost Comparison

Model	Cost per chunk	Monthly cost (1000 docs)	Self-hosted
OpenBrief Nano	$0.00	$0.00	Yes
GPT-5.4 Nano	~$0.002	~$40-60	No
GPT-5.4	~$0.02	~$400-600	No

Training Details

• Base model: Qwen/Qwen3.5-9B
• Method: LoRA (bf16, not QLoRA — Qwen3.5 architecture has higher quantization loss at 4-bit)
• Training data: 708 contract chunks from 407 CUAD contracts
• Labeling: GPT-5.4 with CUAD expert annotations as anchors (hybrid silver labeling)
• LoRA config: rank=32, alpha=64, dropout=0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Epochs: 3
• Learning rate: 2e-4 (cosine schedule)
• Final training loss: 0.62
• Hardware: NVIDIA A100-SXM4-80GB (Google Colab)
• Trainable parameters: 58M / 9B (0.65%)

Training Data Pipeline

1. 510 CUAD commercial contracts split at contract level (407 train / 51 val / 51 test)
2. Contracts chunked into 512-token windows with 50-token overlap
3. CUAD expert annotations matched to chunks as anchors for 6 entity types (party, date, governing_law, termination_right, deadline, monetary_amount)
4. GPT-5.4 used for schema conversion, gap-filling, and labeling 2 additional types (obligation, jurisdiction)
5. Post-processing cleanup to remove generic party references and noisy labels
6. Balanced sampling: 558 anchor-rich chunks + 150 no-anchor chunks = 708 training examples

Label Reliability Tiers

Tier	Entity Types	Label Source	Quality
Tier 1 (high confidence)	party, date, governing_law	CUAD expert annotations + GPT formatting	High
Tier 2 (anchor-partial)	deadline, termination_right, monetary_amount	CUAD clause anchors + GPT extraction	Medium-High
Tier 3 (GPT-assisted)	obligation, jurisdiction	GPT-5.4 only	Medium

Limitations

• Jurisdiction extraction is weak (0 wins vs GPT-Nano in gold review) — limited training signal
• Obligation extraction lags GPT — fewer examples found, though party attribution is often better
• English only — trained exclusively on English commercial contracts from CUAD
• Contract-focused — not trained on court opinions, legislation, or other legal document types
• Silver labels — training data labeled by GPT-5.4 with CUAD anchors. Model inherits systematic biases from the labeling model
• Occasional over-extraction — may extract section titles, generic temporal references, or product names as entities
• Recommended max_new_tokens: 1024 — use 1024 or higher for entity-dense chunks to avoid output truncation

Known Issues and Future Improvements

• Add more jurisdiction training examples from contracts with explicit forum selection clauses
• Add court case chunks to expand beyond commercial contracts
• Post-processing regex to strip think blocks as safety net (model was trained without thinking mode)

Project

OpenBrief Nano is part of OpenBrief, an open-source legal document intelligence platform featuring agentic RAG, persistent memory, multi-provider LLM support, and transparent evaluation metrics.

Author: Ivan Arshakyan / BrainX Corp

Citation

@misc{openbrief-nano-2026,
  title={OpenBrief Nano: LoRA Fine-Tuned Legal Entity Extraction on Qwen3.5-9B},
  author={Ivan Arshakyan},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/illuminator22/openbrief-nano-qwen35-entity-extractor}
}