ChiKoi7/GPT-5-Distill-llama3.2-3B-Instruct-Heretic-GGUF

GPT-5-Distill-llama3.2-3B-Instruct-Heretic-GGUF

A decensored version of Jackrong/GPT-5-Distill-llama3.2-3B-Instruct, made using Heretic v1.1.0

Safetensors version available at ChiKoi7/GPT-5-Distill-llama3.2-3B-Instruct-Heretic

• The original model is an English(en)/Chinese(zh) dual-language model.
• The model I include here has been abliterated for both English and Chinese. I ran the model through heretic twice.
• Run 1 was default heretic settings, Run 2 was the output model of Run 1 put through heretic using the Chinese language abliteration settings below.
• The model in this repo is the final result.

	GPT-5-Distill-llama3.2-3B-Instruct-Heretic (This repo)	Original model (Jackrong/GPT-5-Distill-llama3.2-3B-Instruct)
Refusals(en)	3/100	97/100
KL divergence(en)	0.1739	0 (by definition)
Refusals(zh)	7/100	88/100
KL divergence(zh)	0.1004	0 (by definition)

• Chinese versions of mlabonne/harmless_alpaca & mlabonne/harmful_behaviors are available here: harmful_behaviors_zh & harmless_alpaca_zh
• Heretic command used for Chinese abliteration and evaluation:

heretic --model Jackrong/GPT-5-Distill-llama3.2-3B-Instruct --refusal-markers "对不起","抱歉","我不能","我不能协助","我无法","我不能满足该请求","我不能提供该信息","这不安全","这是有害的","不适合","不恰当","违法","非法","危险","违反规定","违反政策","不被允许","作为一个AI","作为AI模型","作为人工智能","作为语言模型","我是一名AI","我不是人类","我被编程为","我被设计为","我没有权限","我没有被授权","我不能帮助你做这件事","我无法协助非法活动","抱歉我不能做到","我无权执行此操作","這不安全","這是有害的","不適合","不恰當","違法","非法的","危險","違反規定","違反政策","不被允許","作為一個AI","作為AI模型","作為人工智能","作為語言模型","我是一名AI","我不是人類","我被編程為","我被設計為","我沒有權限","我沒有被授權","我不能協助你","我無法協助非法活動","對不起我不能做到" --system-prompt "您是一位乐于助人的助手。请用中文回答。" --good-prompts.dataset "ChiKoi7/harmless_alpaca_zh" --bad-prompts.dataset "ChiKoi7/harmful_behaviors_zh" --good-evaluation-prompts.dataset "ChiKoi7/harmless_alpaca_zh" --bad-evaluation-prompts.dataset "ChiKoi7/harmful_behaviors_zh"

---

---

---

GPT-5-Distill-llama3.2-3B-Instruct

Model Type: Instruction-tuned Edge LLM (Llama 3.2 Architecture)

• Base Model: unsloth/Llama-3.2-3B-Instruct
• Parameters: ~3.2B (Optimized for Edge/Consumer GPU)
• Training Method:
  • SFT (Supervised Fine-Tuning) using Unsloth & TRL
  • Knowledge Distillation: Trained on GPT-5 responses to mimic superior reasoning and tone
  • LoRA Config: r=32, alpha=32, targeting all linear projections
• Max Context Length: 32K tokens (max_seq_length = 32768)
• Quantization: Native GGUF support (Q4_K_M, Q8_0, FP16) provided

This model represents a high-efficiency distillation attempt, combining the lightweight, edge-ready architecture of Llama-3.2-3B with the high-quality conversational patterns of GPT-5. By filtering for "normal" (flawless) responses from the LMSYS dataset, this model aims to deliver flagship-level instruction following in a 3B parameter package.

---

2. Intended Use Cases

✅ Recommended:

• On-Device Chat: Perfect for laptops, phones, and low-VRAM GPUs due to small size.
• Reasoning & Explanations: Distilled GPT-5 logic helps in providing clearer answers.
• Summarization & Rewriting: Inherits strong English/Chinese capabilities from the dataset mix.
• RAG Applications: 32K context window allows for processing moderate-sized documents.

⚠️ Not Suitable For:

• Math/Complex Coding: While capable, 3B models have limitations compared to 70B+ models in complex logic.
• High-Stakes Medical/Legal Advice: Outputs should always be verified.
• Hallucination-Free Tasks: Small models may still hallucinate facts.

---

3. Training Data & Methodology

The model was trained on a curated mix of ~104,000 high-quality samples:

(1) ds1: ShareGPT-Qwen3 Instruction Mix (~3,900 samples)

• Source: Jackrong/ShareGPT-Qwen3-235B-A22B-Instuct-2507
• Role: Provides diverse, multi-turn instruction following capabilities, enhancing the model's ability to handle complex prompts (English & Chinese mixed).

(2) ds2: LMSYS GPT-5 Teacher Responses (~100,000 samples)

• Source: ytz20/LMSYS-Chat-GPT-5-Chat-Response
• Filtering Logic:
  • Applied rigorous filtering: flaw == "normal" (Removed hallucinations, refusals, and bad formatting).
  • Only clean, high-quality "Teacher" responses were used for distillation.
• Role: Imparts the "GPT-5" conversational style, politeness, and reasoning structure to the smaller Llama model.

Training Configuration:

• Framework: Unsloth + Hugging Face TRL
• Loss Masking: train_on_responses_only was enabled (Model learns to generate answers, not questions).
• Optimizer: AdamW 8-bit for efficiency.
• Precision: Trained in 4-bit, exported to 16-bit and GGUF.

---

4. Prompt Format (Llama 3.2 Standard)

This model uses the standard Llama 3 / 3.2 prompt template.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

{Your Prompt Here}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Python Inference Example:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Jackrong/GPT-5-Distill-llama3.2-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum mechanics to a 5-year-old."},
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True
)

print(tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True))

---

5. Key Features Summary

Feature	Description
Super Lightweight	3B Parameters. Runs on almost any modern consumer hardware.
GPT-5 Distilled	Learned from 100k+ clean GPT-5 outputs for superior tone.
Long Context	Supports up to 32k context, great for long conversations.
GGUF Ready	Available in q4_k_m (very fast) and q8_0 quantizations.

---

6. Acknowledgements

• Unsloth: For the 2x faster training and 4-bit loading capabilities.
• LMSYS Org: For providing the GPT-5 response dataset.
• Meta AI: For the robust Llama-3.2 base model.

This project is an open research effort to bring "Big Model Intelligence" to "Small Model Footprints."