GovOn-EXAONE-LoRA-v2

Introduction

GovOn-EXAONE-LoRA-v2 is a specialized QLoRA (4-bit) adapter for the EXAONE-Deep-7.8B model, specifically fine-tuned for the Korean civil complaint domain.

This version (v2) significantly improves response stability and data balance compared to v1. It is designed to assist local government officials in categorizing civil inquiries and generating professional draft responses across 8 major administrative categories.

Quickstart

We recommend using transformers (v4.44-4.49) and peft for inference.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

model_id = "LGAI-EXAONE/EXAONE-Deep-7.8B"
adapter_id = "umyunsang/GovOn-EXAONE-LoRA-v2"

# Load base model with 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

base_model = AutoModelForCausalLM.from_pretrained(
    model_id, quantization_config=bnb_config, device_map="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)

# (Inference code same as Merged-v2)

Specifications

Model Details

• Base Model: LGAI-EXAONE/EXAONE-Deep-7.8B
• Method: QLoRA (4-bit NormalFloat)
• Trainable Parameters: ~154M (Rank 16, Alpha 32)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Training Hyperparameters

Parameter	Value
Learning Rate	2e-4
LR Scheduler	Cosine
Warmup Ratio	0.03
Batch Size (Eff.)	16
Optimizer	paged_adamw_8bit
Max Seq Length	2048

Training and Evaluation

Training Metrics

Metric	Value
Initial Train Loss	3.3224
Final Train Loss	1.5320
Final Eval Loss	1.7872
Token Accuracy (Eval)	60.46%
Total Steps	1,902

Dataset Distribution (12.6K Samples)

Category	Train	Val	Test
Total Samples	10,148	1,265	1,265
Categories	Administration, Transport, Environment, Welfare, Culture, Economy, Safety, etc.

Improvements over v1

Metric	v1	v2	Change
Eval Loss	1.7909	1.7872	-0.21%
EOS Generation Rate	0%	20%	+20%p
Category Bias (Admin)	89.6%	~30%	Normalized

Limitation and Usage

1. Infrastructure: Requires ~7.5GB VRAM for 4-bit inference.
2. CoT Handling: EXAONE-Deep CoT (<thought>) tokens are generated. Removal is recommended for UI display.
3. Domain Constraint: Optimized for 8 civil complaint categories; performance outside these domains is not guaranteed.

License

This model is licensed under the Apache License 2.0. However, users must also comply with the EXAONE AI Model License Agreement of the base model.

Citation

@misc{govon-exaone-lora-v2,
  title={GovOn-EXAONE-LoRA-v2: QLoRA Fine-tuned EXAONE-Deep-7.8B for Korean Civil Complaint Assistance},
  author={GovOn Team},
  year={2026},
  url={https://huggingface.co/umyunsang/GovOn-EXAONE-LoRA-v2}
}