clemsail commited on
Commit
0b62ae2
·
verified ·
1 Parent(s): bf78fd2

docs: add upstream base model official evaluations

Browse files
Files changed (1) hide show
  1. README.md +24 -0
README.md CHANGED
@@ -128,3 +128,27 @@ Source: <https://github.com/ailiance/ailiance/tree/main/output/lm-eval-base-2026
128
  Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
129
  methodology:
130
  <https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
128
  Will include kicad-sch / iact-bench validators + W3 lm-eval delta. See spec for
129
  methodology:
130
  <https://github.com/ailiance/ailiance-bench/blob/main/docs/superpowers/specs/2026-05-11-kicad-sch-gap-design.md>
131
+
132
+ ## Upstream base model — official evaluations
133
+
134
+ This LoRA fine-tunes [`mistralai/Devstral-Small-2-24B-Instruct-2512`](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512),
135
+ Mistral's coding-specialist LLM. Headline software-engineering benchmarks
136
+ from the upstream model card:
137
+
138
+ | Benchmark | Devstral Small 2 (24B) | Devstral 2 (123B) | DeepSeek v3.2 (671B) | Claude Sonnet 4.5 |
139
+ |--------------------------|-----------------------:|------------------:|---------------------:|------------------:|
140
+ | **SWE Bench Verified** | **68.0 %** | 72.2 % | 73.1 % | 77.2 % |
141
+ | **SWE Bench Multilingual** | **55.7 %** | 61.3 % | 70.2 % | 68.0 % |
142
+ | **Terminal Bench 2** | **22.5 %** | 32.6 % | 46.4 % | 42.8 % |
143
+
144
+ (For reference, GPT-5.1 Codex High: 73.7 % SWE Verified · 52.8 % Terminal Bench 2.)
145
+
146
+ Devstral Small 2 (24B) is competitive with much larger open models on
147
+ SWE Bench Verified (e.g. matches GLM-4.6 at 355B). Architecture uses
148
+ rope-scaling per Llama 4 + Scalable-Softmax ([arXiv:2501.19399](https://arxiv.org/abs/2501.19399)).
149
+
150
+ **Source:** [official Devstral-Small-2-24B-Instruct-2512 model card](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512).
151
+
152
+ > **Reading these alongside this LoRA:** Devstral Small 2 is a strong
153
+ > coding base. This LoRA inherits its SWE-Bench performance and adds
154
+ > language- or domain-specific specialization.