chhatramani/nyayalm1.7B_civil9law

⚖️ NyayaLM — 1.7B | Civil Code & 9-Law Legal Assistant

NyayaLM is a fine-tuned large language model specialized in Nepal law, built on top of Qwen3-1.7B. It is designed to answer legal questions accurately and professionally with reference to the Nepal Civil Code and nine major Nepali statutes.

"Nyaya" (न्याय) means justice in Nepali and Sanskrit.

---

🧠 Model Details

Property	Details
Base Model	unsloth/Qwen3-1.7B
Model Type	Causal Language Model (fine-tuned)
Fine-tuning Method	LoRA (QLoRA, 4-bit) via Unsloth
LoRA Rank	r = 32
LoRA Alpha	64
LoRA Dropout	0.05
LoRA Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
RSLoRA	✅ Enabled (Rank Stabilized LoRA)
Chat Template	qwen3-instruct (ChatML format)
Max Sequence Length	2048 tokens
Language	English (primary), Nepali (supported)
Domain	Nepal Legal / Judiciary
HuggingFace Repo	chhatramani/nyayalm1.7B_civil9law

---

📚 Training Data

The model was trained on a combined dataset of two custom legal corpora:

1. 9-Law RAG Dataset (9 law_rag_train.jsonl)

A Retrieval-Augmented Generation (RAG)-style dataset covering nine major Nepali laws, including question-answer pairs grounded in statutory text. This dataset teaches the model to cite and reason from specific legal provisions.

2. Nepal Civil Code QA Dataset (civil_train_dataset_noinput.jsonl)

A curated instruction-following dataset based on the Nepal Civil Code (Muluki Dewani Sanhita, 2074 B.S.), containing structured legal Q&A without additional input context — training the model for direct legal reasoning.

Dataset Split

Split	Proportion	Purpose
Train	80%	Model training
Validation	10%	Evaluation during training
Test	10%	Final held-out evaluation

All records were formatted using the Qwen3-Instruct ChatML template with the following system prompt:

You are NyayaLM, a legal assistant specializing in Nepal law.
Answer legal questions accurately based on Nepal Civil Code and other Nepal laws.

---

⚙️ Training Configuration

Hyperparameter	Value
Optimizer	AdamW 8-bit
Learning Rate	2e-5
LR Scheduler	Cosine
Warmup Ratio	0.08 (~51 warmup steps)
Epochs	1
Batch Size (per device)	4
Gradient Accumulation Steps	4 (effective batch = 16)
Weight Decay	0.001
Max Grad Norm	1.0
Precision	FP16
Eval Strategy	Every 40 steps
Save Strategy	Every 80 steps
Best Model Selection	Lowest eval_loss
Seed	3407
Framework	Unsloth + TRL SFTTrainer

Training used train_on_responses_only — only the assistant turns were used to compute the loss, ignoring user instruction tokens. This improves instruction-following quality.

---

📈 Training Curves

Training converged smoothly over ~500 steps. Both training and validation loss decreased steadily with no signs of overfitting.

• Training Loss: Started at ~1.6, converged to ~0.53
• Validation Loss: Started at ~0.86, converged to ~0.60

---

📊 Evaluation Results

Evaluated on the held-out test set (10%) using 10 sampled examples with the following inference settings:

• temperature = 0.4, top_p = 0.9, repetition_penalty = 1.1, max_new_tokens = 800

Metric	Score
ROUGE-1	0.3393
ROUGE-L	0.2209
BERTScore (F1)	0.9167
METEOR	0.4076
Perplexity	2.35

Note on BERTScore (0.9167): The high BERTScore reflects strong semantic similarity between generated and reference answers — the model captures the legal meaning well even when surface phrasing varies. ROUGE scores are lower by nature since legal answers can be paraphrased in multiple correct ways.

---

🚀 Usage

Basic Inference

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "chhatramani/nyayalm1.7B_civil9law",
    max_seq_length = 2048,
    load_in_4bit = True,
)
model = FastLanguageModel.for_inference(model)

SYSTEM_PROMPT = (
    "You are NyayaLM, a legal assistant specializing in Nepal law. "
    "Answer legal questions accurately based on Nepal Civil Code and other Nepal laws."
)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "What are the grounds for divorce under Nepal Civil Code?"}
]

input_text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

inputs = tokenizer(input_text, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=800,
        temperature=0.4,
        top_p=0.9,
        repetition_penalty=1.1,
    )

response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Using with Transformers (standard)

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("chhatramani/nyayalm1.7B_civil9law")
model = AutoModelForCausalLM.from_pretrained(
    "chhatramani/nyayalm1.7B_civil9law",
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are NyayaLM, a legal assistant specializing in Nepal law."},
    {"role": "user", "content": "Explain the inheritance rights of a daughter under Nepal Civil Code."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

🏛️ Covered Legal Domains

NyayaLM is trained to assist with questions related to (but not limited to):

• Nepal Civil Code (Muluki Dewani Sanhita, 2074 B.S.) — marriage, divorce, inheritance, property, contracts, torts
• 9 Major Nepal Laws covered in the RAG dataset (civil, criminal, procedural statutes)
• Legal reasoning, statutory interpretation, and rights-based queries under Nepali law

---

⚠️ Limitations & Disclaimer

• This model is for informational and research purposes only. It is not a substitute for professional legal advice.
• Evaluations were conducted on a small test sample (n=10); performance may vary on diverse real-world queries.
• The model may occasionally hallucinate legal provisions. Always verify answers against official legal texts.
• Laws change — this model reflects training data up to its creation date and may not capture recent amendments.
• The model performs best on English-language legal queries related to Nepal law.

---

📋 Citation

If you use NyayaLM in your research or projects, please cite:

@misc{nyayalm2025,
  author       = {chhatramani},
  title        = {NyayaLM: A Legal Language Model for Nepal Law},
  year         = {2025},
  publisher    = {HuggingFace},
  howpublished = {\url{https://huggingface.co/chhatramani/nyayalm1.7B_civil9law}},
}

---

🙏 Acknowledgements

• Unsloth for the efficient fine-tuning framework
• Qwen Team (Alibaba) for the Qwen3-1.7B base model
• HuggingFace TRL for SFTTrainer
• The open-source legal NLP community for inspiring domain-specific LLM work

---

Built with ❤️ for accessible legal information in Nepal.