DomainLLM/gemma-3-12b-it-law-fine-tuned

Gemma-3-12B-IT Law Fine-Tuned

A domain-adapted version of Google's Gemma 3 12B Instruct model, fine-tuned on German legal data for improved performance on law-related tasks.

Model Details

Property	Value
Base Model	google/gemma-3-12b-it (via unsloth/gemma-3-12b-it)
Fine-tuning Method	LoRA (Low-Rank Adaptation)
Domain	German Law
Parameters	12B
Precision	bfloat16
License	Gemma License

Training Configuration

Hyperparameter	Value
LoRA Rank (r)	16
LoRA Alpha	32
LoRA Dropout	0.05
Learning Rate	1e-05
Batch Size	16
Max Sequence Length	2048
Epochs	7
Training Steps	21
Warmup Steps	3 (linear)
Optimizer	AdamW
Final Training Loss	1.775
Final Eval Loss	1.781

LoRA Target Modules

The adapter was applied to all attention and MLP projection layers: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Loss Curve

Epoch	Step	Training Loss	Eval Loss
0.28	1	2.047	—
1.55	5	1.990	—
3.28	10	3.828*	1.892
4.55	14	1.845	—
5.55	17	1.809	—
6.55	20	1.785	1.781
6.83	21	1.775	—

*Higher loss at some steps is due to epoch boundary effects with small datasets.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "DomainLLM/gemma-3-12b-it-law-fine-tuned"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Erkläre den Unterschied zwischen Verbrechen und Vergehen im deutschen Strafrecht."}
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Intended Use

This model is designed for:

• 🏛️ Legal Question Answering — Answering questions about German law
• 📜 Legal Text Understanding — Comprehending and summarizing legal documents
• ⚖️ Legal Reasoning — Assisting with legal analysis and argumentation
• 🔍 Legal Research — Supporting legal research tasks in the German legal domain

Limitations

• This model is fine-tuned for German legal domain tasks and may not generalize well to other domains or languages.
• The model should not be used as a substitute for professional legal advice.
• Outputs should always be verified by qualified legal professionals.
• The model may still produce hallucinations or inaccurate legal information.

Citation

If you use this model in your research, please cite:

@misc{domainllm-gemma3-law,
  title={Gemma-3-12B-IT Law Fine-Tuned},
  author={DomainLLM},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/DomainLLM/gemma-3-12b-it-law-fine-tuned}
}

Acknowledgements

• Built on Google's Gemma 3 model family
• Fine-tuned using PEFT/LoRA with the Hugging Face Transformers library