SeaWolf-AI commited on
Commit
f119914
ยท
verified ยท
1 Parent(s): e56adcf

Add model card with parent lineage (Qwen3.6 + hesamation Opus)

Browse files
Files changed (1) hide show
  1. README.md +145 -0
README.md ADDED
@@ -0,0 +1,145 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - en
5
+ - ko
6
+ - multilingual
7
+ base_model:
8
+ - Qwen/Qwen3.6-35B-A3B
9
+ - hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled
10
+ tags:
11
+ - darwin-v7
12
+ - evolutionary-merge
13
+ - mri-guided
14
+ - slerp
15
+ - qwen3.6
16
+ - moe
17
+ - a3b
18
+ - reasoning
19
+ - thinking
20
+ - opus-series
21
+ - hybrid-vigor
22
+ library_name: transformers
23
+ pipeline_tag: text-generation
24
+ ---
25
+
26
+ # Darwin-36B-Opus
27
+
28
+ **Darwin Opus ์‹œ๋ฆฌ์ฆˆ โ€” Qwen3.6 ์„ธ๋Œ€ (A3B MoE)**
29
+
30
+ Qwen3.6-35B-A3B ๊ธฐ๋ฐ˜ ์ง„ํ™”์  ๋ณ‘ํ•ฉ ๋ชจ๋ธ. Father(์ˆœ์ • base) ร— Mother(Claude Opus 4.6 Reasoning Distilled)๋ฅผ Darwin V7 ์—”์ง„์˜ **MRI ์ฒ˜๋ฐฉ + CMA-ES ์ง„ํ™” + SLERP** ๊ธฐ๋ฒ•์œผ๋กœ ์ž๋™ ๋ณ‘ํ•ฉ.
31
+
32
+ ## ๐Ÿงฌ ๊ณ„๋ณด (Parents)
33
+
34
+ ### ๐Ÿ”ต Father โ€” Base Stability
35
+ - **[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)**
36
+ - 35B MoE (3B active), 40 layers
37
+ - Hybrid attention: **Gated DeltaNet 75% + Gated Attention 25%**
38
+ - GPQA 86.0% / MMLU-Pro 85.2% / AIME26 92.7% (official)
39
+
40
+ ### ๐Ÿ”ด Mother โ€” Reasoning Distillation
41
+ - **[hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled](https://huggingface.co/hesamation/Qwen3.6-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled)**
42
+ - Father์— Claude Opus 4.6 CoT ์ฆ๋ฅ˜ SFT
43
+ - LoRA rank=32, 2 epochs, 762 steps, 14,233 CoT ์ƒ˜ํ”Œ
44
+ - MMLU-Pro (70 limit-5): **75.71%** (+32.85%p vs Father base)
45
+ - qwen3-thinking ํ…œํ”Œ๋ฆฟ, response-only masking
46
+
47
+ ## ๐Ÿ”ฌ Darwin V7 ๊ต๋ฐฐ ๋ฐฉ์‹
48
+
49
+ ```
50
+ Phase 0: Auto-Profile (์•„ํ‚คํ…์ฒ˜ ํ˜ธํ™˜ ๊ฒ€์‚ฌ) โ†’ COMPATIBLE โœ“
51
+ Phase 1: MRI Scan (ํ…์„œ๋ณ„ norm/entropy/std + probe)
52
+ Phase 2a: CMA-ES Evolution (500 steps, 8-block genome)
53
+ โ†’ proxy score 0.8403
54
+ Phase 2b: Real SLERP Merge (top-5 candidate evaluation)
55
+ โ†’ method=SLERP ratio=0.416 mri_trust=0.783
56
+ Phase 3: Health check (perplexity + smoke gen) โ†’ healthy โœ“
57
+ Phase 4: Upload
58
+ ```
59
+
60
+ ### ๋ณ‘ํ•ฉ ๊ณต์‹
61
+
62
+ ```
63
+ ๊ฐ ํ…์„œ๋ณ„ ์ตœ์ข… ๋น„์œจ:
64
+ final_ratio = mri_ratio ร— 0.783 + genome_ratio ร— 0.217
65
+
66
+ - 0.416 = global blend ratio (Mother 41.6% + Father 58.4%)
67
+ - 0.783 = MRI ์ฒ˜๋ฐฉ ์‹ ๋ขฐ๋„ (norm/entropy ๊ธฐ๋ฐ˜ ์ฒ˜๋ฐฉ ๋น„์ค‘)
68
+ - 8 ๋ธ”๋ก ร— 40 ๋ ˆ์ด์–ด genome ์ง„ํ™” ์ตœ์ ํ™”
69
+ ```
70
+
71
+ ### ์™œ SLERP?
72
+ ๋‘ ๋ชจ๋ธ ๊ฐ€์ค‘์น˜๋Š” ๊ณ ์ฐจ์› ๊ณก๋ฉด ์œ„์˜ ๋ฒกํ„ฐ. ์„ ํ˜• ๋ณด๊ฐ„(linear avg)์€ ๋งค๋‹ˆํด๋“œ๋ฅผ ๋ฒ—์–ด๋‚˜ ๋ฌด์˜๋ฏธํ•œ ์œ„์น˜๋กœ ์ด๋™ํ•˜์ง€๋งŒ, **๊ตฌ๋ฉด์„ ํ˜•๋ณด๊ฐ„(SLERP)**์€ ๊ณก๋ฉด์„ ๋”ฐ๋ผ ๋ถ€๋“œ๋Ÿฝ๊ฒŒ ์ด๋™ํ•˜์—ฌ ์–‘์ชฝ ํŠน์„ฑ์„ ๋ณด์กด.
73
+
74
+ ## ๐Ÿท๏ธ ์‹œ๋ฆฌ์ฆˆ ํฌ์ง€์…”๋‹
75
+
76
+ | Darwin Opus ๋ชจ๋ธ | Father | Mother | GPQA |
77
+ |-----------------|--------|--------|:----:|
78
+ | Darwin-27B-Opus | Qwen3.5-27B | Jackrong Claude-4.6-Opus distilled | 86.9 |
79
+ | Darwin-31B-Opus | Gemma2-27B ร— ๋‹ค์–‘ | Opus variants | 85.9 |
80
+ | Darwin-35B-A3B-Opus | Qwen3.5-35B-A3B | Jackrong Opus distilled | (์ธก์ •์ค‘) |
81
+ | **Darwin-36B-Opus** | **Qwen3.6-35B-A3B** | **hesamation Qwen3.6 Opus distilled** | **(์ธก์ •์ค‘)** |
82
+
83
+ `36B`๋Š” **Qwen3.6 ์„ธ๋Œ€** ํ‘œ์‹œ๋กœ naming์— ๋ฐ˜์˜ (ํŒŒ๋ผ๋ฏธํ„ฐ๋Š” ์‹ค์ œ 36.0B).
84
+
85
+ ## ๐Ÿง  ์•„ํ‚คํ…์ฒ˜
86
+
87
+ - **Architecture**: Qwen3.5MoE (Qwen3.6๋Š” 3.5 ์ฝ”๋“œ๋ฒ ์ด์Šค ์žฌํ™œ์šฉ)
88
+ - **Total params**: 36.0B
89
+ - **Active params**: ~3B (MoE sparse)
90
+ - **Layers**: 40
91
+ - **Hidden size**: 2048
92
+ - **Experts**: 256 routed, top-8 activation
93
+ - **Hybrid attention**: 75% Gated DeltaNet + 25% Gated Attention
94
+ - **Chat template**: `<|im_start|>assistant\n<think>\n` (thinking mode default)
95
+
96
+ ## ๐Ÿ’ก ์‚ฌ์šฉ๋ฒ•
97
+
98
+ ```python
99
+ from transformers import AutoTokenizer, AutoModelForCausalLM
100
+ import torch
101
+
102
+ tok = AutoTokenizer.from_pretrained("FINAL-Bench/Darwin-36B-Opus", trust_remote_code=True)
103
+ model = AutoModelForCausalLM.from_pretrained(
104
+ "FINAL-Bench/Darwin-36B-Opus",
105
+ torch_dtype=torch.bfloat16,
106
+ device_map="auto",
107
+ trust_remote_code=True,
108
+ )
109
+
110
+ messages = [{"role": "user", "content": "What is the derivative of sin(xยฒ)?"}]
111
+ text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
112
+ inputs = tok(text, return_tensors="pt").to(model.device)
113
+ outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, do_sample=True)
114
+ print(tok.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
115
+ ```
116
+
117
+ ## โš ๏ธ ํ‰๊ฐ€ ์‹œ ์ฃผ์˜
118
+
119
+ Reasoning ๋ชจ๋ธ(`<think></think>` ์‚ฌ์šฉ)์ด๋ฏ€๋กœ ๋‹ต ์ถ”์ถœ ์‹œ:
120
+
121
+ ```python
122
+ # ๋ชจ๋ธ ์‘๋‹ต์—์„œ </think> ์ดํ›„ ๋ถ€๋ถ„๋งŒ ์ •๋‹ต์œผ๋กœ ์ถ”์ถœ
123
+ idx = response.rfind("</think>")
124
+ answer_part = response[idx + len("</think>"):].strip() if idx >= 0 else response
125
+ ```
126
+
127
+ ## ๐Ÿ—๏ธ ์ œ์ž‘
128
+
129
+ - **Engine**: Darwin V7 (FINAL-Bench proprietary)
130
+ - **Hardware**: 2ร— NVIDIA B200 (merge GPUs)
131
+ - **Evolution**: 500 steps in ~15 minutes
132
+ - **Cache ID**: `merged_6edaacaf`
133
+ - **Proxy fitness (arc_challenge)**: 0.8403
134
+ - **Commit**: `e56adcfb` (2026-04-22)
135
+
136
+ ## ๐Ÿ“œ ๋ผ์ด์„ ์Šค
137
+
138
+ Apache 2.0 (Qwen3.6 ๋ผ์ด์„ ์Šค ์Šน๊ณ„)
139
+
140
+ ## ๐Ÿ™ Credits
141
+
142
+ - Qwen Team (Father base)
143
+ - @hesamation (Mother: Opus distillation)
144
+ - Anthropic Claude Opus 4.6 (Teacher)
145
+ - FINAL-Bench / VIDRAFT_LAB (Darwin V7 engine + breeding)