docs: add base vs tuned bench comparison
Browse files
README.md
CHANGED
|
@@ -109,3 +109,22 @@ LoRA weights: **apache-2.0** — see License chain table above for derivation ra
|
|
| 109 |
## Related
|
| 110 |
|
| 111 |
See the full [Ailiance-fr LoRA collection](https://huggingface.co/Ailiance-fr).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 109 |
## Related
|
| 110 |
|
| 111 |
See the full [Ailiance-fr LoRA collection](https://huggingface.co/Ailiance-fr).
|
| 112 |
+
|
| 113 |
+
|
| 114 |
+
## Bench comparison (2026-05-11)
|
| 115 |
+
|
| 116 |
+
### Base model (Devstral-Small-2-24B-MLX-4bit) capability
|
| 117 |
+
|
| 118 |
+
| Task | Score | Notes |
|
| 119 |
+
|---|---:|---|
|
| 120 |
+
| GSM8K-CoT flex EM | **0.96** | W3 lm-eval-harness (--limit 100) |
|
| 121 |
+
| ARC-Easy acc / acc_norm | **0.80 / 0.75** | |
|
| 122 |
+
| MMLU-Pro Computer Science | **0.64** | |
|
| 123 |
+
|
| 124 |
+
Source: <https://github.com/ailiance/ailiance/tree/main/output/lm-eval-base-2026-05-11>
|
| 125 |
+
|
| 126 |
+
### This LoRA (tuned) — bench PENDING
|
| 127 |
+
|
| 128 |
+
Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
|
| 129 |
+
methodology:
|
| 130 |
+
<https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
|