clemsail commited on
Commit
c371f3c
·
verified ·
1 Parent(s): a07dc92

docs: add upstream base model official evaluations

Browse files
Files changed (1) hide show
  1. README.md +24 -0
README.md CHANGED
@@ -143,3 +143,27 @@ Source: <https://github.com/ailiance/ailiance/tree/main/output/lm-eval-base-2026
143
  Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
144
  methodology:
145
  <https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
143
  Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
144
  methodology:
145
  <https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
146
+
147
+ ## Upstream base model — official evaluations
148
+
149
+ This LoRA fine-tunes [`mistralai/Devstral-Small-2-24B-Instruct-2512`](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512),
150
+ Mistral's coding-specialist LLM. Headline software-engineering benchmarks
151
+ from the upstream model card:
152
+
153
+ | Benchmark | Devstral Small 2 (24B) | Devstral 2 (123B) | DeepSeek v3.2 (671B) | Claude Sonnet 4.5 |
154
+ |--------------------------|-----------------------:|------------------:|---------------------:|------------------:|
155
+ | **SWE Bench Verified** | **68.0 %** | 72.2 % | 73.1 % | 77.2 % |
156
+ | **SWE Bench Multilingual** | **55.7 %** | 61.3 % | 70.2 % | 68.0 % |
157
+ | **Terminal Bench 2** | **22.5 %** | 32.6 % | 46.4 % | 42.8 % |
158
+
159
+ (For reference, GPT-5.1 Codex High: 73.7 % SWE Verified · 52.8 % Terminal Bench 2.)
160
+
161
+ Devstral Small 2 (24B) is competitive with much larger open models on
162
+ SWE Bench Verified (e.g. matches GLM-4.6 at 355B). Architecture uses
163
+ rope-scaling per Llama 4 + Scalable-Softmax ([arXiv:2501.19399](https://arxiv.org/abs/2501.19399)).
164
+
165
+ **Source:** [official Devstral-Small-2-24B-Instruct-2512 model card](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512).
166
+
167
+ > **Reading these alongside this LoRA:** Devstral Small 2 is a strong
168
+ > coding base. This LoRA inherits its SWE-Bench performance and adds
169
+ > language- or domain-specific specialization.