docs: add upstream base model official evaluations
Browse files
README.md
CHANGED
|
@@ -143,3 +143,27 @@ Source: <https://github.com/ailiance/ailiance/tree/main/output/lm-eval-base-2026
|
|
| 143 |
Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
|
| 144 |
methodology:
|
| 145 |
<https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 143 |
Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
|
| 144 |
methodology:
|
| 145 |
<https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
|
| 146 |
+
|
| 147 |
+
## Upstream base model — official evaluations
|
| 148 |
+
|
| 149 |
+
This LoRA fine-tunes [`mistralai/Devstral-Small-2-24B-Instruct-2512`](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512),
|
| 150 |
+
Mistral's coding-specialist LLM. Headline software-engineering benchmarks
|
| 151 |
+
from the upstream model card:
|
| 152 |
+
|
| 153 |
+
| Benchmark | Devstral Small 2 (24B) | Devstral 2 (123B) | DeepSeek v3.2 (671B) | Claude Sonnet 4.5 |
|
| 154 |
+
|--------------------------|-----------------------:|------------------:|---------------------:|------------------:|
|
| 155 |
+
| **SWE Bench Verified** | **68.0 %** | 72.2 % | 73.1 % | 77.2 % |
|
| 156 |
+
| **SWE Bench Multilingual** | **55.7 %** | 61.3 % | 70.2 % | 68.0 % |
|
| 157 |
+
| **Terminal Bench 2** | **22.5 %** | 32.6 % | 46.4 % | 42.8 % |
|
| 158 |
+
|
| 159 |
+
(For reference, GPT-5.1 Codex High: 73.7 % SWE Verified · 52.8 % Terminal Bench 2.)
|
| 160 |
+
|
| 161 |
+
Devstral Small 2 (24B) is competitive with much larger open models on
|
| 162 |
+
SWE Bench Verified (e.g. matches GLM-4.6 at 355B). Architecture uses
|
| 163 |
+
rope-scaling per Llama 4 + Scalable-Softmax ([arXiv:2501.19399](https://arxiv.org/abs/2501.19399)).
|
| 164 |
+
|
| 165 |
+
**Source:** [official Devstral-Small-2-24B-Instruct-2512 model card](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512).
|
| 166 |
+
|
| 167 |
+
> **Reading these alongside this LoRA:** Devstral Small 2 is a strong
|
| 168 |
+
> coding base. This LoRA inherits its SWE-Bench performance and adds
|
| 169 |
+
> language- or domain-specific specialization.
|