README.md

---
base_model: Qwen/Qwen2.5-7B-Instruct
language:
- es
license: apache-2.0
library_name: peft
pipeline_tag: text-generation
tags:
- cybersecurity
- spanish
- qlora
- peft
- qwen2.5
- arxiv:2605.13989
---

# VectraYX-Pro 7B

VectraYX-Pro 7B is a **QLoRA-32 adapter** for [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) fine-tuned on the VectraYX Spanish cybersecurity SFT corpus (~93,500 examples). It is part of the VectraYX model family presented in the paper [arXiv:2605.13989](https://arxiv.org/abs/2605.13989).

[![arXiv](https://img.shields.io/badge/arXiv-2605.13989-b31b1b.svg)](https://arxiv.org/abs/2605.13989)
[![Zenodo](https://zenodo.org/badge/DOI/10.5281/zenodo.20122226.svg)](https://doi.org/10.5281/zenodo.20122226)

> **This repo contains only the QLoRA adapter weights (~308 MB).** You need to load them on top of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct).

- **Paper:** [VectraYX-Nano arXiv:2605.13989](https://arxiv.org/abs/2605.13989)
- **Nano 42M (from-scratch headline model):** [jsantillana/vectrayx-nano](https://huggingface.co/jsantillana/vectrayx-nano)
- **Pro 3B:** [jsantillana/vectrayx-pro-3b](https://huggingface.co/jsantillana/vectrayx-pro-3b)
- **Author website:** https://jsantillana.com

---

## Results (VectraYX-Bench, single seed)

| Model | Params | B1 KW | B2 F1 | B3 TM | B4 Tool | B5 Chat |
|---|---|---|---|---|---|---|
| VectraYX-Nano v7 *(headline)* | 42M | 0.332±0.005 | — | — | 0.230±0.052 | 0.725±0.130 |
| VectraYX-Pro 3B | 3.2B | 0.341 | 0.695 | 0.686 | 0.600 | 0.800 |
| **VectraYX-Pro 7B** | 7B | **0.335** | **0.815** | **0.686** | **0.880** | **0.800** |
| GPT-4o *(frontier ref.)* | — | 0.333 | 0.110 | 0.520 | 0.615 | 0.631 |

**B4=0.880** — best tool-selection score in the VectraYX family. B2=0.815 (best threat classification).

---

## What is this?

This adapter applies the VectraYX cybersecurity specialization to Qwen2.5-7B-Instruct:
- **SFT corpus:** ~93,500 examples — 13K OASST1-ES + 4K CVE Q&A + 2.8K MCP tool-use traces + general cybersecurity Q&A
- **Training:** QLoRA rank=32 (4-bit quantized base), 3 epochs, lr=2e-4 on AWS SageMaker (`ml.g5.xlarge`)
- **Language:** Spanish (LATAM-focused)
- **Tool use:** Native MCP `<|tool_call|>` emission (B4=0.880, highest in family)

---

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

# Load 4-bit quantized base model (~5 GB VRAM)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_quant_type="nf4"
)
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    quantization_config=bnb_config,
    device_map="auto"
)

# Load VectraYX QLoRA adapter on top
model = PeftModel.from_pretrained(base_model, "jsantillana/vectrayx-pro-7b")
tokenizer = AutoTokenizer.from_pretrained("jsantillana/vectrayx-pro-7b")

# Inference
messages = [{"role": "user", "content": "¿Cuáles son los CVEs más críticos relacionados con Log4j?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Model family

| Model | Backbone | Params | B4 Tool | B2 F1 |
|---|---|---|---|---|
| [VectraYX-Nano v7](https://huggingface.co/jsantillana/vectrayx-nano) | from-scratch | 42M | 0.230±0.052 | — |
| [VectraYX-Base](https://huggingface.co/jsantillana/vectrayx-base) | from-scratch | 260M | 0.000* | 0.220 |
| [VectraYX-Pro 3B](https://huggingface.co/jsantillana/vectrayx-pro-3b) | Qwen2.5-3B + LoRA-64 | 3.2B | 0.600 | 0.695 |
| **VectraYX-Pro 7B** | Qwen2.5-7B + QLoRA-32 | 7B | **0.880** | **0.815** |

---

## Citation

```bibtex
@misc{santillana2026vectrayx,
  title     = {VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model
               with Curriculum Learning and Native Tool Use},
  author    = {Santillana, Juan S.},
  year      = {2026},
  eprint    = {2605.13989},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url       = {https://arxiv.org/abs/2605.13989}
}
```