--- base_model: unsloth/Devstral-Small-2507-unsloth-bnb-4bit library_name: peft model_name: devstral-v3-sft tags: - base_model:adapter:unsloth/Devstral-Small-2507-unsloth-bnb-4bit - lora - sft - transformers - trl - unsloth license: apache-2.0 pipeline_tag: text-generation --- # Model Card for devstral-v3-sft This model is a fine-tuned version of [unsloth/Devstral-Small-2507-unsloth-bnb-4bit](https://huggingface.co/unsloth/Devstral-Small-2507-unsloth-bnb-4bit). It has been trained using [TRL](https://github.com/huggingface/trl). ## Quick start ```python from transformers import pipeline question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?" generator = pipeline("text-generation", model="None", device="cuda") output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0] print(output["generated_text"]) ``` ## Training procedure This model was trained with SFT. ### Framework versions - PEFT 0.18.1 - TRL: 0.24.0 - Transformers: 5.5.0 - Pytorch: 2.10.0 - Datasets: 4.3.0 - Tokenizers: 0.22.2 ## Citations Cite TRL as: ```bibtex @misc{vonwerra2022trl, title = {{TRL: Transformer Reinforcement Learning}}, author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec}, year = 2020, journal = {GitHub repository}, publisher = {GitHub}, howpublished = {\url{https://github.com/huggingface/trl}} } ``` # devstral-v3-sft ## ๐Ÿ‡ช๐Ÿ‡บ EU AI Act transparency This model is published under the AI Act framework (Regulation EU 2024/1689). | Field | Value | |---|---| | Provider | Ailiance (clemsail) | | Role under AI Act | GPAI provider | | Adapter type | LoRA / PEFT โ€” supervised fine-tune adapter | | Base model | `mistralai/Devstral-Small-2-24B-Instruct-2512` | | License | Apache-2.0 (this artefact); upstream Mistral licence applies separately | | Intended use | Code generation across Python / Rust / TypeScript / C++ / SQL / shell, with stronger reasoning on engineering questions | | Out of scope | Healthcare diagnosis, legal advice, autonomous safety-critical decisions, generation of malicious code or exploits | | Risk classification | Limited risk โ€” Article 50 transparency obligations apply | | Copyright respect | Training data does not include scraped copyrighted material. Public engineering documentation under permissive licences plus internal synthetic distillation. | | Full provenance | https://ailiance.fr/transparency | | Contact | postmaster@saillant.cc | โš ๏ธ **You are using an AI model.** Outputs may be inaccurate, biased or fabricated. Do not act on them without independent verification, especially in regulated domains. ## Benchmarks Run via `lm-eval-harness` v0.4.x against the FUSED checkpoint (base + this adapter merged for inference). Strict-match where applicable. | Task | Metric | Score | |---|---|---| | gsm8k | `exact_match,strict-match` | **0.844** | | ifeval | `prompt_level_strict_acc,none` | **0.691** | | bbh_cot_fewshot | `exact_match,get-answer` | **0.795** | | bbh_cot_fewshot_boolean_expressions | `exact_match,get-answer` | **0.900** | | bbh_cot_fewshot_causal_judgement | `exact_match,get-answer` | **0.600** | | bbh_cot_fewshot_date_understanding | `exact_match,get-answer` | **0.933** | | bbh_cot_fewshot_disambiguation_qa | `exact_match,get-answer` | **0.767** | | bbh_cot_fewshot_dyck_languages | `exact_match,get-answer` | **0.100** | | bbh_cot_fewshot_formal_fallacies | `exact_match,get-answer` | **0.600** | | bbh_cot_fewshot_geometric_shapes | `exact_match,get-answer` | **0.367** | | bbh_cot_fewshot_hyperbaton | `exact_match,get-answer` | **1.000** | | bbh_cot_fewshot_logical_deduction_five_objects | `exact_match,get-answer` | **0.767** | | bbh_cot_fewshot_logical_deduction_seven_objects | `exact_match,get-answer` | **0.533** | | bbh_cot_fewshot_logical_deduction_three_objects | `exact_match,get-answer` | **0.900** | | bbh_cot_fewshot_movie_recommendation | `exact_match,get-answer` | **0.833** | | bbh_cot_fewshot_multistep_arithmetic_two | `exact_match,get-answer` | **0.867** | | bbh_cot_fewshot_navigate | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_object_counting | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_penguins_in_a_table | `exact_match,get-answer` | **0.933** | | bbh_cot_fewshot_reasoning_about_colored_objects | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_ruin_names | `exact_match,get-answer` | **0.667** | | bbh_cot_fewshot_salient_translation_error_detection | `exact_match,get-answer` | **0.700** | | bbh_cot_fewshot_snarks | `exact_match,get-answer` | **0.700** | | bbh_cot_fewshot_sports_understanding | `exact_match,get-answer` | **0.900** | | bbh_cot_fewshot_temporal_sequences | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_tracking_shuffled_objects_five_objects | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_tracking_shuffled_objects_seven_objects | `exact_match,get-answer` | **0.933** | | bbh_cot_fewshot_tracking_shuffled_objects_three_objects | `exact_match,get-answer` | **0.967** | | bbh_cot_fewshot_web_of_lies | `exact_match,get-answer` | **1.000** | | bbh_cot_fewshot_word_sorting | `exact_match,get-answer` | **0.667** | | mmlu_pro | `exact_match,custom-extract` | **0.619** | | mmlu_pro_biology | `exact_match,custom-extract` | **0.768** | | mmlu_pro_business | `exact_match,custom-extract` | **0.660** | | mmlu_pro_chemistry | `exact_match,custom-extract` | **0.580** | | mmlu_pro_computer_science | `exact_match,custom-extract` | **0.676** | | mmlu_pro_economics | `exact_match,custom-extract` | **0.678** | | mmlu_pro_engineering | `exact_match,custom-extract` | **0.448** | | mmlu_pro_health | `exact_match,custom-extract` | **0.678** | | mmlu_pro_history | `exact_match,custom-extract` | **0.575** | | mmlu_pro_law | `exact_match,custom-extract` | **0.432** | | mmlu_pro_math | `exact_match,custom-extract` | **0.678** | | mmlu_pro_other | `exact_match,custom-extract` | **0.612** | | mmlu_pro_philosophy | `exact_match,custom-extract` | **0.549** | | mmlu_pro_physics | `exact_match,custom-extract` | **0.630** | | mmlu_pro_psychology | `exact_match,custom-extract` | **0.704** | | leaderboard_math_hard | `exact_match,none` | **0.341** | | leaderboard_math_algebra_hard | `exact_match,none` | **0.570** | | leaderboard_math_counting_and_prob_hard | `exact_match,none` | **0.252** | | leaderboard_math_geometry_hard | `exact_match,none` | **0.182** | | leaderboard_math_intermediate_algebra_hard | `exact_match,none` | **0.139** | | leaderboard_math_num_theory_hard | `exact_match,none` | **0.416** | | leaderboard_math_prealgebra_hard | `exact_match,none` | **0.523** | | leaderboard_math_precalculus_hard | `exact_match,none` | **0.126** | Raw `results_*.json` files are committed under `evals/`. ## Validated in `ailiance/ailiance-bench` v0.2 This model is referenced in the [Ailiance benchmark suite](https://github.com/ailiance/ailiance-bench) (Phase 6 scoreboard, 7-task hardware-design evaluation). See the full scoreboard: [ailiance-bench README#scoreboard-lora-phase-6](https://github.com/ailiance/ailiance-bench#scoreboard-lora-phase-6--2026-05-11). ## Upstream base model โ€” official evaluations This LoRA fine-tunes [`mistralai/Devstral-Small-2-24B-Instruct-2512`](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512), Mistral's coding-specialist LLM. Headline software-engineering benchmarks from the upstream model card: | Benchmark | Devstral Small 2 (24B) | Devstral 2 (123B) | DeepSeek v3.2 (671B) | Claude Sonnet 4.5 | |--------------------------|-----------------------:|------------------:|---------------------:|------------------:| | **SWE Bench Verified** | **68.0 %** | 72.2 % | 73.1 % | 77.2 % | | **SWE Bench Multilingual** | **55.7 %** | 61.3 % | 70.2 % | 68.0 % | | **Terminal Bench 2** | **22.5 %** | 32.6 % | 46.4 % | 42.8 % | (For reference, GPT-5.1 Codex High: 73.7 % SWE Verified ยท 52.8 % Terminal Bench 2.) Devstral Small 2 (24B) is competitive with much larger open models on SWE Bench Verified (e.g. matches GLM-4.6 at 355B). Architecture uses rope-scaling per Llama 4 + Scalable-Softmax ([arXiv:2501.19399](https://arxiv.org/abs/2501.19399)). **Source:** [official Devstral-Small-2-24B-Instruct-2512 model card](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512). > **Reading these alongside this LoRA:** Devstral Small 2 is a strong > coding base. This LoRA inherits its SWE-Bench performance and adds > language- or domain-specific specialization.