DomainLLM/gemma-3-12b-german-legal-lora

Gemma-3-12B German Legal LoRA

Model Description

This is a LoRA (Low-Rank Adaptation) fine-tuned version of Google's Gemma-3-12B-Instruct model, specifically optimized for German legal question-answering tasks. The model has been trained on the GerLayQA dataset, which contains high-quality German legal questions and answers covering various areas of German law including BGB (Civil Code), StGB (Criminal Code), and other legal domains.

Training Details

Base Model

• Model: google/gemma-3-12b-it
• Parameters: 12B
• Architecture: Transformer-based language model

Training Configuration

• Training Data: DomainLLM/gerlayqa_5k_filtered_raw
• Training Samples: 4,500 (train split)
• Validation Samples: 500 (validation split)
• Max Sequence Length: 4096
• Batch Size: 32
• Epochs: 7
• Learning Rate: 2e-5
• Optimizer: adamw_8bit

LoRA Configuration

• LoRA Rank: 16
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Target Modules: Attention and MLP layers
• Trainable Parameters: ~16M (0.13% of base model)

Dataset Information

The model was trained on the GerLayQA 5K Filtered Raw dataset, which contains:

• Total Samples: 5,000 German legal Q&A pairs
• Train Split: 4,500 samples (90%)
• Validation Split: 500 samples (10%)
• Average Question Length: ~1,100 characters (162 words)
• Average Answer Length: ~2,000 characters (278 words)
• Legal Domains: BGB, StGB, and other German legal codes
• Quality: GPT-4o-mini cleaned and filtered

Dataset Statistics

• Unique Legal Paragraphs: 1,146
• Top Legal References: § 1004, § 823, § 280, § 242, § 812
• Language: German
• Domain: Legal question-answering

Usage

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model = "google/gemma-3-12b-it"
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load LoRA weights
model = PeftModel.from_pretrained(model, "DomainLLM/gemma-3-12b-german-legal-lora")

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model)

Inference Example

def generate_legal_answer(question, context=None):
    # Format the prompt
    if context:
        prompt = f"Du bist ein erfahrener Rechtsanwalt mit Spezialisierung auf deutsches Recht. Beantworte die folgende Rechtsfrage basierend auf dem gegebenen Kontext.\n\nKontext: {context}\n\nFrage: {question}\n\nAntwort:"
    else:
        prompt = f"Du bist ein erfahrener Rechtsanwalt mit Spezialisierung auf deutsches Recht. Beantworte die folgende Rechtsfrage.\n\nFrage: {question}\n\nAntwort:"
    
    # Tokenize and generate
    inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096)
    
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.1,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id
        )
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response.split("Antwort:")[-1].strip()

# Example usage
question = "Welche Rechte hat ein Mieter bei einer Eigenbedarfskündigung?"
answer = generate_legal_answer(question)
print(answer)

Performance

The model has been fine-tuned specifically for German legal question-answering tasks and demonstrates:

• Domain Expertise: Specialized knowledge in German law (BGB, StGB, etc.)
• Language Proficiency: Native-level German language understanding
• Legal Accuracy: Proper citation of legal paragraphs and statutes
• Structured Responses: Clear, professional legal explanations

Limitations

• Language: Optimized for German legal text only
• Domain: Specialized for legal question-answering, may not perform well on general tasks
• Context Length: Limited to 4096 tokens maximum
• Legal Advice: This model is for educational/research purposes only, not for actual legal advice

Training Hardware

• GPU: Training performed on high-performance computing infrastructure
• Memory: Requires significant VRAM for inference (recommended: 24GB+)
• Precision: Mixed precision training with bfloat16

Citation

If you use this model in your research, please cite:

@misc{gemma-3-12b-german-legal-lora,
  title={Gemma-3-12B German Legal LoRA},
  author={DomainLLM Team},
  year={2025},
  url={https://huggingface.co/DomainLLM/gemma-3-12b-german-legal-lora},
  note={LoRA fine-tuned Gemma-3-12B for German legal question-answering}
}

License

This model is released under the Apache 2.0 License. The base model (Gemma-3-12B) is also licensed under Apache 2.0.

Acknowledgments

• Google: For the Gemma-3-12B base model
• GerLayQA Dataset: For providing high-quality German legal Q&A data
• Unsloth: For efficient LoRA training framework
• DomainLLM: For dataset preparation and cleaning

Contact

For questions or issues related to this model, please open an issue on the Hugging Face model page.

---

This model is part of the DomainLLM project, focusing on domain-specific language model adaptations for specialized tasks.