akenginorhun/qwen3-8b-orchestrator-lora-with-failures

Qwen3-8B Orchestrator LoRA — With Failures

LoRA fine-tuned Qwen/Qwen3-8B for agentic orchestration tasks (tool use, multi-turn reasoning, web search).

Training Details

Parameter	Value
Base Model	Qwen/Qwen3-8B
LoRA Rank	128
LoRA Alpha	256
Learning Rate	2.38e-05
Epochs	1
Validation Loss	0.2352
Training Samples	52,204
Dataset	GLM-4.7-flash SFT traces (includes failed traces)
Context Length	16,384 tokens
Quantization	4-bit (QLoRA)

Evaluation Results

Benchmark	Score
SimpleQA (200)	30.0% accuracy
GAIA (165)	10.9% accuracy
HLE (180)	6.1% accuracy
DeepResearch (100)	0.2340 score

Usage

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained(
    "akenginorhun/qwen3-8b-orchestrator-lora-with-failures",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "akenginorhun/qwen3-8b-orchestrator-lora-with-failures"
)

W&B Sweep

• Sweep ID: 5unzmw85
• Project: hazy-research/limit
• Method: Bayesian optimization (6 runs)