Hugging Face's logo

xpertsystems
/
cyb009-baseline-classifier

Tabular Classification
PyTorch
cybersecurity
vulnerability-management
cve
cvss
epss
cisa-kev
synthetic-data
xgboost
baseline
leakage-diagnostic
data-quality-audit
Eval Results (legacy)
Model card Files
xet
 Community
cyb009-baseline-classifier / README.md
pradeep-xpert's picture
pradeep-xpert
Initial release: vulnerability_class baseline + comprehensive 8-oracle-path leakage diagnostic on CYB009 sample
e520bf1 verified
preview code
|
raw
history blame contribute delete
22.3 kB
metadata 
license: cc-by-nc-4.0
library_name: pytorch
tags:
  - cybersecurity
  - vulnerability-management
  - cve
  - cvss
  - epss
  - cisa-kev
  - tabular-classification
  - synthetic-data
  - xgboost
  - baseline
  - leakage-diagnostic
  - data-quality-audit
pipeline_tag: tabular-classification
base_model: []
datasets:
  - xpertsystems/cyb009-sample
metrics:
  - accuracy
  - f1
  - roc_auc
model-index:
  - name: cyb009-baseline-classifier
    results:
      - task:
          type: tabular-classification
          name: 8-class vulnerability classification (CWE-style families)
        dataset:
          type: xpertsystems/cyb009-sample
          name: CYB009 Synthetic Vulnerability Intelligence Dataset (Sample)
        metrics:
          - type: roc_auc
            value: 0.6837
            name: Test macro ROC-AUC OvR (XGBoost, seed 42)
          - type: accuracy
            value: 0.2374
            name: Test accuracy (XGBoost, seed 42)
          - type: f1
            value: 0.2244
            name: Test macro-F1 (XGBoost, seed 42)
          - type: accuracy
            value: 0.244
            name: Multi-seed accuracy mean ± 0.023 (XGBoost, 10 seeds)
          - type: roc_auc
            value: 0.687
            name: Multi-seed ROC-AUC mean ± 0.014 (XGBoost, 10 seeds)

CYB009 Baseline Classifier

Vulnerability classification baseline (8-class) trained on the CYB009 synthetic vulnerability intelligence sample. The primary artifact value of this repo is leakage_diagnostic.json — the most comprehensive structural-leakage audit in the XpertSystems baseline catalog, documenting 8 oracle paths and 6 unlearnable README-suggested targets. The classifier itself is the catalog's weakest baseline by design (acc 0.244 vs majority 0.176), included to show that vulnerability_class is the ONLY README-headline target that learns honestly on this sample.

Read this first. This repo ships three artifacts in priority order:

  1. leakage_diagnostic.json — comprehensive audit of 8 oracle paths discovered on CYB009 and 6 README-suggested targets that are unlearnable on the sample after honest leak removal.
  2. A working classifier for vulnerability_class 8-class — the only README target that learns honestly on this sample, and the weakest baseline in the XpertSystems catalog by design.
  3. A feature engineering reference (feature_engineering.py).

If you came here looking for a strong baseline, you will be disappointed. If you came here to understand why the CYB009 sample has hard-to-detect structural label-feature determinism, the diagnostic is exactly the artifact you need.

Model overview

Property Value
Primary task 8-class vulnerability_class classification (CWE-style families)
Primary artifact leakage_diagnostic.json — 8 oracle paths + 6 unlearnable targets
Training data xpertsystems/cyb009-sample (2,638 vulnerabilities)
Models XGBoost + PyTorch MLP
Input features 57 (after one-hot encoding)
Split Stratified random (per-vulnerability, no group structure to leak)
Validation Single seed (artifact) + multi-seed aggregate across 10 seeds
License CC-BY-NC-4.0 (matches dataset)
Status Reference baseline + comprehensive leakage diagnostic

Why this task — and the journey to get here

The CYB009 README lists 11 suggested use cases. We piloted every README-headline target and found pervasive structural leakage. The abandoned candidates, in order of how we discovered them:

Initial candidate: exploit_maturity_final 4-class (ABANDONED)

The most natural target — 4-class (unproven/PoC/functional/weaponised), n=2638 well-balanced (36/27/25/12%), maps directly to EPSS calibration. Initial feasibility hit acc 0.74, macro-F1 0.72, ROC-AUC 0.91 vs majority 0.36 — a +38pp lift looked excellent.

Then we found the leak. cvss_temporal_score_final divided by cvss_base_score clusters near-deterministically per maturity tier:

Maturity tier Observed ratio (median ± std) CVSS v3.1 multiplier
unproven 0.801 ± 0.011 0.91 × (other Temporal factors)
proof_of_concept 0.827 ± 0.011 0.94 × (other Temporal factors)
functional 0.854 ± 0.011 0.97 × (other Temporal factors)
weaponised 0.880 ± 0.012 1.00 × (other Temporal factors)

This is exactly the CVSS v3.1 Exploit Code Maturity multiplier (unproven 0.91 / PoC 0.94 / functional 0.97 / high or weaponised 1.00), combined with other near-constant Temporal factors (Remediation Level, Report Confidence). The cvss_temporal/cvss_base ratio uniquely identifies the maturity tier.

Drop cvss_temporal_score_final → accuracy collapses to 0.31 (below majority 0.36). The target is structurally unlearnable on the sample once the oracle is removed.

Other 5 candidates: also unlearnable after honest leak removal

Target n_positive Maj baseline Honest acc Honest AUC Verdict
exploitation_occurred_flag 203 0.923 0.857 0.65 Below majority
zero_day_flag 76 0.971 0.949 0.60 Below majority
cisa_kev_flag 14 0.995 0.992 0.61 Below majority
supply_chain_propagation_flag 20 0.992 0.992 0.80 Below majority
false_positive_flag 205 0.922 0.866 0.52 Below majority

All five rare-event binaries are oracled by time_to_exploit_days (-1 sentinel) or time_to_remediate_days (120 sentinel) at full features; after honest leak removal, all are at-or-below majority.

Per-timestep multi-class targets: state-machine oracles

lifecycle_phase, patch_status, and remediation_status on vulnerability_records.csv form a tightly-coupled state machine:

  • lifecycle_phase = residual_risk_review → 100% remediated
  • lifecycle_phase = discovery → 100% undetected
  • lifecycle_phase = remediation_deployment → 100% in_remediation
  • patch_status = deployed → 100% remediated

Naive evaluation on these targets reaches accuracy 0.95-0.98, but any two of the three deterministically pin the third. None of these is a viable independent ML target on the sample.

severity_class: 100% mechanical CVSS function

Observed cvss_base_score ranges per severity match CVSS v3.1 exactly: critical [9.0, 10.0], high [7.0, 9.0], medium [4.0, 7.0], low [1.8, 4.0]. Predicting severity is trivial with CVSS; below majority (acc 0.55 vs 0.51) without it.

vulnerability_class 8-class: the only honest target — and the baseline ships

After exhausting the README-suggested targets, vulnerability_class is the only one that learns honestly:

  • acc 0.244 ± 0.023, macro-F1 0.230 ± 0.024, ROC-AUC 0.687 ± 0.014
  • +7pp lift over majority (the catalog's smallest)
  • All 8 classes represented (per-class F1 0.09-0.33)
  • No oracle feature — modest signal genuinely spread across CVSS, EPSS, asset context, and binary flags

This is the weakest baseline in the XpertSystems catalog by design. The full ~487k-row product would tighten per-class signal materially. The dataset roadmap recommendations in leakage_diagnostic.json describe what would make CYB009's headline targets viable on the sample.

Quick start 

pip install xgboost torch safetensors pandas huggingface_hub

from huggingface_hub import hf_hub_download, snapshot_download
import json, numpy as np, torch, xgboost as xgb
from safetensors.torch import load_file

REPO = "xpertsystems/cyb009-baseline-classifier"

paths = {n: hf_hub_download(REPO, n) for n in [
    "model_xgb.json", "model_mlp.safetensors",
    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
]}

import sys, os
sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
from feature_engineering import (
    transform_single, load_meta, build_asset_lookup, INT_TO_LABEL,
)

meta = load_meta(paths["feature_meta.json"])

# Asset features are joined from asset_inventory.csv at inference time
ds = snapshot_download("xpertsystems/cyb009-sample", repo_type="dataset")
asset_lookup = build_asset_lookup(f"{ds}/asset_inventory.csv")

xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])

# Predict (see inference_example.ipynb for the full pattern)
# Note: do NOT include exploit_maturity_final, cvss_temporal_score_final,
# time_to_exploit_days, time_to_remediate_days, patch_lag_days, or
# risk_score_composite - those were the outcome-leak columns.
X = transform_single(my_vuln_record, meta, asset_lookup=asset_lookup)
proba = xgb_model.predict_proba(X)[0]
print(INT_TO_LABEL[int(np.argmax(proba))])

See inference_example.ipynb for the full copy-paste demo.

Training data

Trained on the public sample of CYB009, 2,638 per-vulnerability records:

Vulnerability class Vulns Class share
memory_corruption 465 17.6%
injection_family 436 16.5%
misconfiguration 435 16.5%
auth_access_control 350 13.3%
cryptographic_failure 301 11.4%
supply_chain_weakness 271 10.3%
logic_flaw 228 8.6%
information_disclosure 152 5.8%

Stratified split

Per-vulnerability task (one row per vuln in vuln_summary.csv), StratifiedShuffleSplit nested 70/15/15:

Fold Vulns
Train 1,846
Validation 396
Test 396

Class imbalance addressed with class_weight='balanced' (XGBoost sample_weight) and weighted cross-entropy (MLP).

Feature pipeline

The bundled feature_engineering.py is the canonical recipe. 57 features survive after encoding, drawn from:

  • Per-vulnerability numeric (10): cvss_base_score, epss_score_final, plus 8 binary post-hoc flags
  • Per-vulnerability categorical (1, one-hot): severity_class (4 values, CVSS-derived but useful as feature)
  • Asset features (joined from asset_inventory.csv): 8 numeric
    • 4 categorical (asset_type, criticality_tier, environment_type, os_family)
  • Engineered (5): log_epss, is_high_cvss, exposure_severity_composite, risk_flag_count, epss_x_base

Excluded columns (outcome leaks)

Column Why excluded
exploit_maturity_final Indirect leak via CVSS temporal multiplier (would reintroduce the 0.91/0.94/0.97/1.00 oracle)
cvss_temporal_score_final Near-deterministic per exploit_maturity_final tier (the primary leak we discovered)
time_to_exploit_days -1 sentinel oracle for exploitation_occurred_flag
time_to_remediate_days 120 sentinel oracle for remediation_success_flag
patch_lag_days Suspected similar sentinel (precaution)
risk_score_composite Computed from flag fields (indirect oracle)

Evaluation

Test-set metrics, seed 42 (n = 396 vulnerabilities)

XGBoost (the published model_xgb.json artifact)

Metric Value
Macro ROC-AUC (OvR) 0.6837
Accuracy 0.2374
Macro-F1 0.2244
Weighted-F1 0.2407

MLP (the published model_mlp.safetensors artifact)

Metric Value
Macro ROC-AUC (OvR) 0.6899
Accuracy 0.2323
Macro-F1 0.2209
Weighted-F1 0.2362

MLP and XGBoost are within noise of each other on this task — both are publishing the same modest honest signal.

Multi-seed robustness (XGBoost, 10 seeds)

Metric Mean Std Min Max
Accuracy 0.244 0.023 0.217 0.283
Macro-F1 0.230 0.024 0.206 0.280
Macro ROC-AUC OvR 0.687 0.014 0.660 0.700

All 10 seeds yielded all 8 classes in the test fold (stratified split guarantees this). Full per-seed results in multi_seed_results.json.

Per-class F1 (seed 42)

Vulnerability class Class share XGBoost F1 MLP F1
memory_corruption 17.6% 0.333 0.365
information_disclosure 5.8% 0.291 0.154
misconfiguration 16.5% 0.259 0.162
injection_family 16.5% 0.237 0.235
supply_chain_weakness 10.3% 0.222 0.292
cryptographic_failure 11.4% 0.217 0.168
auth_access_control 13.3% 0.146 0.163
logic_flaw 8.6% 0.090 0.228

memory_corruption (highest mean CVSS at 8.3) and information_disclosure (lowest mean CVSS at 5.4) are the most distinctive classes. logic_flaw is the hardest — its feature distribution overlaps closely with everything else.

Ablation: which feature groups matter

Configuration Accuracy Macro-F1 ROC-AUC Δ accuracy
Full feature set (published) 0.2374 0.2244 0.6837 —
No CVSS features 0.2121 0.1926 0.6690 −0.0253
No asset features 0.2172 0.1967 0.6870 −0.0202
No engineered features 0.2323 0.2216 0.6871 −0.0051
No severity (one-hot) 0.2273 0.2175 0.6857 −0.0101
No EPSS features 0.2475 0.2237 0.6926 +0.0101
No binary flags 0.2273 0.2114 0.6776 −0.0101

Three findings:

  1. No feature group is dominant. Largest single drop is 2.5pp (CVSS features). Every group contributes a little; nothing contributes a lot. The signal is genuinely diffuse.
  2. CVSS and asset features carry the most signal (~2pp each), consistent with the observation that per-class CVSS means differ (5.4 to 8.3) and asset features modestly inform class.
  3. EPSS features slightly hurt on this task (+1pp without them). EPSS is intended for exploitation prediction, not class prediction; on this sample it acts as small additional noise.

Architecture

XGBoost: multi-class gradient boosting (multi:softprob, 8 classes), hist tree method, class-balanced sample weights, early stopping on validation mlogloss.

MLP: 57 → 128 → 64 → 8, each hidden layer followed by BatchNorm1d → ReLU → Dropout(0.3), weighted cross-entropy loss, AdamW optimizer, early stopping on validation macro-F1.

Training hyperparameters are held internally by XpertSystems.

Limitations

This is a baseline reference, not a production vulnerability classifier.

  1. The headline finding is the leakage diagnostic, not the classifier. Read leakage_diagnostic.json first. The classifier demonstrates that vulnerability_class is the only README-suggested target that learns honestly on the sample.

  2. Per-class F1 ranges 0.09–0.33. The model is more confident on memory_corruption and information_disclosure than on logic_flaw and auth_access_control. For production use, expect different error patterns by class.

  3. No feature group contributes more than 3pp accuracy. The model has no single decisive signal; instead it integrates many weakly-informative features. Removing any one group has minimal impact.

  4. Synthetic-vs-real transfer. The dataset is synthetic, calibrated to 12 benchmarks from authoritative vulnerability intelligence sources (NIST NVD, EPSS v3, CISA KEV, Mandiant, Verizon DBIR, Rapid7, Qualys, Tenable). Real vulnerability telemetry has different noise characteristics — in particular, the structural-oracle patterns documented in leakage_diagnostic.json (CVSS temporal multipliers, sentinel-coded time fields, lifecycle state-machine determinism) would not be present in real data with comparable density. Real data has stochastic transitions and observation noise.

  5. 2,638 vulnerabilities is a modest training set for 8 classes. The 396-vulnerability test fold yields stable multi-seed metrics (std 0.023) but per-class confidence intervals are wide. The full ~487k-row product has materially more data per class.

Notes on dataset schema

The CYB009 sample dataset README describes some fields differently from the actual schema. This note helps buyers reconcile what they read with what they receive.

What the README says What the data actually contains
vulnerability_records has 19 columns Data has 16 columns
vulnerability_records includes severity, exploited_in_wild_flag, cisa_kev_listed_flag, zero_day_flag, supply_chain_flag, internet_exposed, sla_breached_flag None of these columns exist in vulnerability_records. Per-vuln flags are only on vuln_summary.
vuln_class has 10 values (incl. race_condition, web_application, configuration) 8 values in the data; differs in: misconfiguration (not configuration), auth_access_control (not authentication_bypass), logic_flaw (new); no race_condition, no web_application, no deserialization
8 lifecycle phases 12 phases in the data, adding residual_risk_review (45% of all rows), false_positive_closed, sla_breach, accepted_risk, discovery, organisational_triage, exploitation_in_wild
patch_status has 4 values 6 values in the data: adds vendor_notified, patch_in_development, patch_validated
severity has 5 values (incl. none) 4 values in the data (severity_class): low, medium, high, critical only
vuln_summary has 15 columns Data has 21 columns
Field renames severity_final → severity_class; cvss_base_score_final → cvss_base_score; cisa_kev_listed → cisa_kev_flag; exploited_in_wild → exploitation_occurred_flag; supply_chain_compromise → supply_chain_propagation_flag
Semantic inversion README's sla_breached (True = bad) ↔ data's sla_compliance_flag (True = good)
remediation_outcome categorical (patched/mitigated/accepted/unpatched) Replaced with remediation_success_flag (binary) plus per-timestep remediation_status
Not in README New fields: risk_score_composite, compensating_control_flag, time_to_exploit_days, time_to_remediate_days, patch_lag_days

None of these affects model correctness — the feature pipeline uses the actual column names. If you build your own pipeline against the dataset, use the actual columns.

Intended use

  • Reading the leakage diagnostic — the primary value of this repo. Reusable methodology for any synthetic vulnerability dataset.
  • Evaluating fit of the CYB009 dataset for your research, with open knowledge of the structural-oracle patterns
  • Honest baseline reference for the only README-suggested target that learns on the sample
  • Feature engineering reference for per-vulnerability ML

Out-of-scope use

  • Production vulnerability triage on real telemetry
  • Exploit maturity prediction — README headline target, unlearnable on the sample after honest leak removal
  • Zero-day / KEV / supply-chain prediction — README headline targets, unlearnable as rare-event binaries on the sample
  • SLA breach prediction — README headline target, unlearnable after honest leak removal
  • Any operational security decision without further validation on real data

Reproducibility

Outputs above were produced with seed = 42 (published artifact), nested StratifiedShuffleSplit (70/15/15), on the published sample (xpertsystems/cyb009-sample, version 1.0.0, generated 2026-05-16). The feature pipeline in feature_engineering.py is deterministic and the trained weights in this repo correspond exactly to the metrics above.

Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200) in multi_seed_results.json confirm robust performance across splits (std 0.023 on accuracy).

The training script itself is private to XpertSystems.

Files in this repo

File Purpose
leakage_diagnostic.json PRIMARY ARTIFACT — 8 oracle paths + 6 unlearnable targets
model_xgb.json XGBoost weights (seed 42)
model_mlp.safetensors PyTorch MLP weights (seed 42)
feature_engineering.py Feature pipeline
feature_meta.json Feature column order + categorical levels
feature_scaler.json MLP input mean/std (XGBoost ignores)
validation_results.json Per-class metrics, confusion matrix, architecture
ablation_results.json Per-feature-group ablation
multi_seed_results.json XGBoost metrics across 10 seeds
inference_example.ipynb End-to-end inference demo notebook
README.md This file

Contact and full product

The full CYB009 dataset contains ~487,000 vulnerability records across four files, with calibrated benchmark validation against 12 metrics drawn from authoritative vulnerability intelligence sources (NIST NVD, EPSS v3, CISA KEV, Mandiant, Verizon DBIR, Rapid7, Qualys, Tenable). The full XpertSystems.ai synthetic data catalogue spans 41 SKUs across Cybersecurity, Healthcare, Insurance & Risk, Oil & Gas, and Materials & Energy.

Citation 

@misc{xpertsystems_cyb009_baseline_2026,
  title  = {CYB009 Baseline Classifier: XGBoost and MLP for Vulnerability Classification, with the XpertSystems Catalog's Most Comprehensive Structural-Leakage Audit},
  author = {XpertSystems.ai},
  year   = {2026},
  url    = {https://huggingface.co/xpertsystems/cyb009-baseline-classifier},
  note   = {Reference baseline + 8-oracle-path leakage diagnostic on xpertsystems/cyb009-sample}
}