Initial release: XGBoost + MLP for SOC alert triage outcome classification, with structural-leakage and unlearnable-target diagnostic

README.md

@@ -0,0 +1,495 @@
+---
+license: cc-by-nc-4.0
+library_name: pytorch
+tags:
+  - cybersecurity
+  - soc-operations
+  - alert-triage
+  - mitre-attack
+  - soar
+  - siem
+  - tabular-classification
+  - synthetic-data
+  - xgboost
+  - baseline
+  - leakage-diagnostic
+pipeline_tag: tabular-classification
+base_model: []
+datasets:
+  - xpertsystems/cyb008-sample
+metrics:
+  - accuracy
+  - f1
+  - roc_auc
+model-index:
+  - name: cyb008-baseline-classifier
+    results:
+      - task:
+          type: tabular-classification
+          name: 5-class SOC alert triage outcome classification
+        dataset:
+          type: xpertsystems/cyb008-sample
+          name: CYB008 Synthetic SOC Alert Dataset (Sample)
+        metrics:
+          - type: roc_auc
+            value: 0.9522
+            name: Test macro ROC-AUC OvR (XGBoost, seed 42)
+          - type: accuracy
+            value: 0.7659
+            name: Test accuracy (XGBoost, seed 42)
+          - type: f1
+            value: 0.7430
+            name: Test macro-F1 (XGBoost, seed 42)
+          - type: accuracy
+            value: 0.777
+            name: Multi-seed accuracy mean ± 0.007 (XGBoost, 10 seeds)
+          - type: roc_auc
+            value: 0.955
+            name: Multi-seed ROC-AUC mean ± 0.003 (XGBoost, 10 seeds)
+          - type: roc_auc
+            value: 0.9552
+            name: Test macro ROC-AUC OvR (MLP, seed 42)
+          - type: accuracy
+            value: 0.7674
+            name: Test accuracy (MLP, seed 42)
+          - type: f1
+            value: 0.7510
+            name: Test macro-F1 (MLP, seed 42)
+---
+
+# CYB008 Baseline Classifier
+
+**SOC alert triage classifier trained on the CYB008 synthetic SOC alert
+sample. Predicts which of 5 triage outcome classes
+(`auto_resolved_soar` / `duplicate_merged` / `false_positive_closed` /
+`true_positive_remediated` / `true_positive_escalated`) an alert
+will reach, from per-alert features. ALSO ships a leakage diagnostic
+for the three structural-oracle columns dropped from the feature
+pipeline.**
+
+> **Read this first.** This repo ships two related artifacts:
+> (1) a working baseline classifier for `resolution_outcome` (the
+> primary product), and (2) a `leakage_diagnostic.json` file
+> documenting (a) the three structural oracle columns that were
+> dropped from the feature set, and (b) the separate finding that the
+> README's first suggested use case — MITRE ATT&CK tactic
+> classification — is **not learnable** on this sample. Both files
+> matter; the diagnostic is required reading for anyone evaluating
+> CYB008 for a triage product.
+
+## Model overview
+
+| Property | Value |
+|---|---|
+| Primary task | 5-class `resolution_outcome` classification (SOC alert triage) |
+| Secondary artifact | `leakage_diagnostic.json` — structural oracle + unlearnable-target audit |
+| Training data | `xpertsystems/cyb008-sample` (9,200 alerts) |
+| Models | XGBoost + PyTorch MLP |
+| Input features | 53 (after one-hot encoding) |
+| Split | **Stratified random** (no natural group key in this dataset — see rationale below) |
+| Validation | Single seed (artifact) + multi-seed aggregate across 10 seeds |
+| License | CC-BY-NC-4.0 (matches dataset) |
+| Status | Reference baseline + leakage diagnostic |
+
+## Why this task — and what was dropped
+
+The CYB008 README lists **alert triage (TP vs FP prediction)** as its
+first suggested use case and **MITRE ATT&CK tactic classification** as
+its second. We piloted both on the sample dataset:
+
+- **Triage outcome:** works honestly. After dropping 3 structural
+  oracle columns, the model achieves **acc 0.777 ± 0.007, ROC-AUC
+  0.955 ± 0.003** on 5-class classification. This is the primary
+  baseline.
+
+- **MITRE tactic classification:** **does NOT work on this sample.**
+  Without `mitre_technique_id` (which is a perfect ATT&CK-by-design
+  oracle), the per-tactic feature distributions are nearly identical
+  (raw_score 0.37–0.39 across all 12 tactics, similar for enriched
+  score and fatigue). A trained XGBoost achieves accuracy 0.08,
+  below the majority baseline of 0.14. The README's stated use case
+  cannot be honestly demonstrated on the sample. See
+  [`leakage_diagnostic.json`](./leakage_diagnostic.json) for the full
+  finding and our recommendation to the dataset author.
+
+### The three structural oracle columns (dropped)
+
+CYB008 has three columns that structurally encode the
+`resolution_outcome` label:
+
+| Column | Oracle relationship |
+|---|---|
+| `alert_lifecycle_phase` | 3 of 4 values deterministically map to specific outcomes (auto_closed → auto_resolved_soar; escalated → true_positive_escalated; suppressed_duplicate → duplicate_merged) |
+| `automation_resolved` | Exact 1:1 with `auto_resolved_soar` outcome |
+| `escalation_flag` | 1319 escalation flags = 1319 `true_positive_escalated` outcomes (near-1:1) |
+
+With all three present, plain XGBoost achieves **100% test accuracy
+across all seeds** — mechanical, not learned. With all three dropped,
+accuracy is **0.79 with ROC-AUC 0.96**: real learning on a
+non-trivial 5-class task. The published baseline trains with these
+three columns excluded.
+
+Two model artifacts are published. They are designed to be used
+together — disagreement is a useful triage signal:
+
+- `model_xgb.json` — gradient-boosted trees
+- `model_mlp.safetensors` — PyTorch MLP in SafeTensors format
+
+On CYB008 the MLP slightly outperforms XGBoost on the test fold
+(0.767 vs 0.766 accuracy, 0.955 vs 0.952 ROC-AUC at seed 42) — only
+the second SKU in the XpertSystems baseline catalog where this
+happens (after CYB007).
+
+## Quick start
+
+```bash
+pip install xgboost torch safetensors pandas huggingface_hub
+```
+
+```python
+from huggingface_hub import hf_hub_download
+import json, numpy as np, torch, xgboost as xgb
+from safetensors.torch import load_file
+
+REPO = "xpertsystems/cyb008-baseline-classifier"
+
+paths = {n: hf_hub_download(REPO, n) for n in [
+    "model_xgb.json", "model_mlp.safetensors",
+    "feature_engineering.py", "feature_meta.json", "feature_scaler.json",
+]}
+
+import sys, os
+sys.path.insert(0, os.path.dirname(paths["feature_engineering.py"]))
+from feature_engineering import transform_single, load_meta, INT_TO_LABEL
+
+meta = load_meta(paths["feature_meta.json"])
+xgb_model = xgb.XGBClassifier(); xgb_model.load_model(paths["model_xgb.json"])
+
+# Predict (see inference_example.ipynb for the full pattern)
+# Note: do NOT include alert_lifecycle_phase, automation_resolved, or
+# escalation_flag in your record - those were the oracle columns.
+X = transform_single(my_alert_record, meta)
+proba = xgb_model.predict_proba(X)[0]
+print(INT_TO_LABEL[int(np.argmax(proba))])
+```
+
+See [`inference_example.ipynb`](./inference_example.ipynb) for the full
+copy-paste demo.
+
+## Training data
+
+Trained on the public sample of CYB008, 9,200 per-alert records:
+
+| Outcome | Alerts | Class share |
+|---|---:|---:|
+| `false_positive_closed` | 2,996 | 32.6% |
+| `auto_resolved_soar` | 2,642 | 28.7% |
+| `true_positive_remediated` | 1,848 | 20.1% |
+| `true_positive_escalated` | 1,319 | 14.3% |
+| `duplicate_merged` | 395 | 4.3% |
+
+### Stratified split (no natural group key)
+
+CYB008 does not have a natural row-level group key for group-aware
+splitting:
+- 25 analysts — group-aware split would yield only ~4 test analysts
+- 5 SOCs — would yield 1 test SOC
+- 589 incidents — only 9% of alerts have a non-null `incident_id`
+
+Alerts are essentially independent given features, so we use
+**StratifiedShuffleSplit** (nested 70/15/15), the same approach as
+CYB001 for network flow classification:
+
+| Fold | Alerts |
+|---|---:|
+| Train | 6,440 |
+| Validation | 1,380 |
+| Test | 1,380 |
+
+Class imbalance is addressed with `class_weight='balanced'` (XGBoost
+`sample_weight`) and weighted cross-entropy (MLP).
+
+## Feature pipeline
+
+The bundled `feature_engineering.py` is the canonical feature recipe.
+53 features survive after encoding, drawn from:
+
+- **Per-alert numeric** (9): `raw_score`, `enriched_score`, `time_in_phase_minutes`, `queue_depth_at_ingestion`, `soar_playbook_triggered`, `sla_breached_flag`, `mttd_minutes`, `mttr_minutes`, `fatigue_score_at_alert`
+- **Per-alert categorical** (5, one-hot): `alert_severity` (7 values), `alert_source` (8 values), `mitre_tactic` (12 values), `analyst_tier` (3 values), `siem_platform` (8 values)
+- **Engineered** (6): `enrichment_lift`, `log_mttr`, `log_mttd`, `queue_pressure`, `enrichment_per_minute`, `is_high_confidence`
+
+### Excluded columns
+
+**Oracle columns** (dropped to allow honest evaluation):
+
+| Column | Why excluded |
+|---|---|
+| `alert_lifecycle_phase` | 3 of 4 values are deterministic outcome oracles |
+| `automation_resolved` | 1:1 with `auto_resolved_soar` outcome |
+| `escalation_flag` | Near-1:1 with `true_positive_escalated` outcome |
+
+**High-cardinality columns** (dropped for tractability):
+
+| Column | Why excluded |
+|---|---|
+| `mitre_technique_id` | 36 unique values; perfect oracle for `mitre_tactic` but unrelated to this target |
+| `detection_rule_id` | 656 unique values; one-hot explosion with no real per-tactic affinity (only 5% of rules map to a single tactic) |
+
+### Partial-oracle features (kept as legitimate observables)
+
+`soar_playbook_triggered` is a *necessary but not sufficient* condition
+for `auto_resolved_soar` — when 0, the alert is never auto-resolved;
+when 1, the outcome is auto-resolved 68% of the time but can also be
+TP-remediated, TP-escalated, FP-closed, or duplicate-merged. This is
+a legitimate observable that downstream operators would already have
+on hand at decision time. KEPT in the pipeline.
+
+## Evaluation
+
+### Test-set metrics, seed 42 (n = 1,380 alerts)
+
+**XGBoost** (the published `model_xgb.json` artifact)
+
+| Metric | Value |
+|---|---:|
+| Macro ROC-AUC (OvR) | **0.9522** |
+| Accuracy | **0.7659** |
+| Macro-F1 | 0.7430 |
+| Weighted-F1 | 0.7672 |
+
+**MLP** (the published `model_mlp.safetensors` artifact) — **slightly outperforms XGBoost**
+
+| Metric | Value |
+|---|---:|
+| Macro ROC-AUC (OvR) | **0.9552** |
+| Accuracy | **0.7674** |
+| Macro-F1 | 0.7510 |
+| Weighted-F1 | 0.7691 |
+
+With 6,440 training rows and 53 features, the MLP has enough data to
+compete favorably with boosted trees. Both models are published.
+
+### Multi-seed robustness (XGBoost, 10 seeds)
+
+Very stable performance — std 0.007 on accuracy is among the tightest
+in the XpertSystems catalog:
+
+| Metric | Mean | Std | Min | Max |
+|---|---:|---:|---:|---:|
+| Accuracy | 0.777 | 0.007 | 0.766 | 0.792 |
+| Macro-F1 | 0.765 | 0.011 | 0.743 | 0.783 |
+| Macro ROC-AUC OvR | 0.955 | 0.003 | 0.950 | 0.960 |
+
+Full per-seed results in [`multi_seed_results.json`](./multi_seed_results.json).
+All 10 seeds yielded all 5 classes in the test fold (stratified split
+guarantees this).
+
+### Per-class F1 (seed 42)
+
+| Outcome | Class share | XGBoost F1 | MLP F1 |
+|---|---:|---:|---:|
+| `false_positive_closed` | 32.6% | **0.904** | 0.910 |
+| `duplicate_merged` | 4.3% | 0.794 | 0.825 |
+| `auto_resolved_soar` | 28.7% | 0.757 | 0.751 |
+| `true_positive_remediated` | 20.1% | 0.701 | 0.698 |
+| `true_positive_escalated` | 14.3% | 0.559 | 0.571 |
+
+The model performs best on `false_positive_closed` (clearest behavioural
+profile — low scores, fast resolution by L1 analysts) and
+`duplicate_merged` (smallest class but distinctive — duplicate-suppressed
+severity is a strong tell). The hardest discrimination is between
+`true_positive_remediated` and `true_positive_escalated` — both are
+genuine threats, differing primarily by whether the alert was closed
+by the original analyst or passed to a higher tier. In production this
+matters less because both are TP outcomes; binary TP-vs-FP recall is
+much higher.
+
+### Ablation: which feature groups matter
+
+| Configuration | Accuracy | Macro-F1 | ROC-AUC | Δ accuracy |
+|---|---:|---:|---:|---:|
+| Full feature set (published) | 0.7659 | 0.7430 | 0.9522 | — |
+| No alert severity | 0.5138 | 0.3933 | 0.7304 | **−0.2522** |
+| No `soar_playbook_triggered` | 0.6188 | 0.5773 | 0.8369 | **−0.1471** |
+| No analyst tier | 0.7717 | 0.7471 | 0.9524 | +0.0058 |
+| No siem platform | 0.7681 | 0.7474 | 0.9522 | +0.0022 |
+| No alert source | 0.7638 | 0.7406 | 0.9511 | −0.0022 |
+| No engineered features | 0.7681 | 0.7480 | 0.9533 | +0.0022 |
+| No mitre_tactic | 0.7812 | 0.7656 | 0.9530 | +0.0152 |
+| No timing features | 0.7775 | 0.7572 | 0.9547 | +0.0116 |
+| No score features | 0.7710 | 0.7569 | 0.9541 | +0.0051 |
+
+Four findings:
+
+1. **Alert severity carries the dominant signal** (drops 25 pp
+   accuracy, 22 pp ROC-AUC). This is intuitive: severity directly
+   drives triage priority, which drives outcome. `false_positive`
+   severity → `false_positive_closed`; `duplicate_suppressed` severity
+   → `duplicate_merged`.
+2. **`soar_playbook_triggered` is the second-strongest signal**
+   (drops 15 pp accuracy). It's a partial oracle for the
+   `auto_resolved_soar` outcome class.
+3. **MITRE tactic and analyst tier contribute essentially nothing.**
+   The model performs marginally *better* without them — they add
+   noise that the trees over-fit on the training set.
+4. **Engineered features and timing features are near-flat.** The
+   trees recover composites from raw inputs. Kept in the pipeline as
+   a documented baseline reference.
+
+### Architecture
+
+**XGBoost:** multi-class gradient boosting (`multi:softprob`, 5 classes),
+`hist` tree method, class-balanced sample weights, early stopping on
+validation mlogloss.
+
+**MLP:** `53 → 128 → 64 → 5`, each hidden layer followed by `BatchNorm1d`
+→ `ReLU` → `Dropout(0.3)`, weighted cross-entropy loss, AdamW optimizer,
+early stopping on validation macro-F1.
+
+Training hyperparameters are held internally by XpertSystems.
+
+## Limitations
+
+**This is a baseline reference, not a production SOC triage system.**
+
+1. **MITRE tactic classification is unlearnable on this sample.** The
+   README lists it as a suggested use case but the per-tactic feature
+   distributions are too similar (raw_score 0.37–0.39 across all 12
+   tactics). See [`leakage_diagnostic.json`](./leakage_diagnostic.json)
+   for the full audit. Real SOC data has stronger per-tactic feature
+   signatures.
+
+2. **TP-remediated vs TP-escalated is the hardest discrimination.**
+   F1 0.56 on TP-escalated is the weakest per-class result. Both are
+   genuine threats; the difference is workflow rather than threat
+   nature. For most operational uses (TP-vs-FP recall, SLA-breach
+   reduction), this confusion does not matter.
+
+3. **MLP modestly outperforms XGBoost.** Both are shipped; we
+   recommend running both and treating disagreement as a triage
+   triage signal. The boost is modest enough that for production
+   deployment, the choice between them is essentially an engineering
+   preference.
+
+4. **Synthetic-vs-real transfer.** The dataset is synthetic and
+   calibrated to 12 SOC-operations benchmarks (SANS SOC Survey, IBM
+   Cost of Data Breach, Mandiant M-Trends, Forrester Wave SOAR,
+   Gartner SIEM Magic Quadrant, SOC.OS, CrowdStrike, Splunk State of
+   Security, Verizon DBIR). Real SOC telemetry has different noise
+   characteristics and the structural-oracle pattern documented
+   above (alert_lifecycle_phase deterministically encoding outcome)
+   would not be present in real data — real lifecycle phases
+   transition stochastically. Do not assume metrics transfer
+   end-to-end.
+
+5. **9,200 alerts is a modest training set.** The 1,380-alert test
+   fold yields stable multi-seed metrics (std 0.007), but full
+   confidence intervals for downstream production decisions should
+   come from the full ~280k-alert product.
+
+## Notes on dataset schema
+
+The CYB008 sample dataset README describes some fields differently
+from the actual schema. The model was trained on the actual schema;
+this note helps buyers reconcile what they read with what they receive.
+
+| What the README says | What the data actually contains |
+|---|---|
+| `incident_summary` has 8 columns | Data has **23 columns** including incident_type, kill_chain_stages_observed, false_positive_rate, soar_actions_taken, etc. |
+| `alert_severity` has 6 values (info / low / medium / high / critical / false_positive) | **7 values**: adds `duplicate_suppressed`. All values are suffixed (`high_severity`, `low_severity`, `critical_confirmed`, `informational`). |
+| `analyst_tier` has 4 values (tier_1 / tier_2 / tier_3 / manager) | 3 values on alerts (`L1_junior`, `L2_senior`, `L3_threat_hunter`); 4 on `soc_topology` (adds `L4_incident_commander`). |
+| 14 MITRE ATT&CK tactics | 12 tactics in the data (no `reconnaissance` or `resource_development` from PRE-ATT&CK). |
+| Detection source mix: edr, siem, ndr, ids, ueba, casb, deception, threat intel | Field is `alert_source` (not `detection_source`); 8 values: `edr_behavioural_engine`, `nids_signature`, `ueba_user_anomaly`, `cspm_cloud_rule`, `siem_correlation_rule`, `threat_intel_ioc_match`, `honeypot_trigger`, `itdr_identity_anomaly`. |
+| `triage_score` / `enrichment_score` columns | Actual names: `raw_score` / `enriched_score`. |
+| `alert_timestamp` (ISO string) | Actual: `alert_timestamp_min` (integer minutes from epoch). |
+| `kill_chain_stage`, `storm_event_flag` columns on alerts | Not present in the data. |
+| Field rename: `detection_source` ↔ data `alert_source` | Same fact noted twice |
+| `resolution_outcome` values (true_positive / false_positive / duplicate / suppressed) | Actual 5 values: `auto_resolved_soar`, `duplicate_merged`, `false_positive_closed`, `true_positive_escalated`, `true_positive_remediated`. |
+| Extra columns in data not in README | `shift_id`, `time_in_phase_minutes`, `queue_depth_at_ingestion`, `fatigue_score_at_alert`, `siem_platform`, `soar_playbook_id`, `detection_rule_id`, `alert_lifecycle_phase` |
+
+None of these affects model correctness — the feature pipeline uses
+the actual column names. If you build your own pipeline against the
+dataset, use the actual columns.
+
+## Intended use
+
+- **Evaluating fit** of the CYB008 dataset for your SOC-triage research
+- **Baseline reference** for new model architectures
+- **Reference example of structural-leakage diagnostics** in
+  synthetic SOC datasets — the diagnostic methodology is reusable
+- **Feature engineering reference** for per-alert SOC telemetry
+
+## Out-of-scope use
+
+- Production SOC triage decisions on real telemetry
+- MITRE ATT&CK tactic prediction (this baseline establishes that
+  task is unlearnable on the sample)
+- SLA-breach prediction (also tested as unlearnable on the sample —
+  acc 0.68 vs majority 0.82)
+- Any operational decision affecting actual security operations
+  without further validation on your own data
+
+## Reproducibility
+
+Outputs above were produced with `seed = 42` (published artifact),
+nested `StratifiedShuffleSplit` (70/15/15), on the published sample
+(`xpertsystems/cyb008-sample`, version 1.0.0, generated 2026-05-16).
+The feature pipeline in `feature_engineering.py` is deterministic and
+the trained weights in this repo correspond exactly to the metrics
+above.
+
+Multi-seed results (seeds 42, 7, 13, 17, 23, 31, 45, 99, 123, 200)
+in `multi_seed_results.json` confirm robust performance across splits.
+
+The training script itself is private to XpertSystems.
+
+## Files in this repo
+
+| File | Purpose |
+|---|---|
+| `model_xgb.json` | XGBoost weights (seed 42) |
+| `model_mlp.safetensors` | PyTorch MLP weights (seed 42) |
+| `feature_engineering.py` | Feature pipeline |
+| `feature_meta.json` | Feature column order + categorical levels |
+| `feature_scaler.json` | MLP input mean/std (XGBoost ignores) |
+| `validation_results.json` | Per-class metrics, confusion matrix, architecture |
+| `ablation_results.json` | Per-feature-group ablation |
+| `multi_seed_results.json` | XGBoost metrics across 10 seeds |
+| `leakage_diagnostic.json` | **Structural-oracle audit + unlearnable-target finding** |
+| `inference_example.ipynb` | End-to-end inference demo notebook |
+| `README.md` | This file |
+
+## Contact and full product
+
+The full **CYB008** dataset contains ~335,000 rows across four files,
+with calibrated benchmark validation against 12 metrics drawn from
+authoritative SOC operations and threat intelligence sources (SANS
+SOC Survey, IBM Cost of Data Breach, Mandiant M-Trends, Forrester
+Wave SOAR, Gartner SIEM Magic Quadrant, SOC.OS, CrowdStrike, Splunk
+State of Security, Verizon DBIR). The full XpertSystems.ai synthetic
+data catalogue spans 41 SKUs across Cybersecurity, Healthcare,
+Insurance & Risk, Oil & Gas, and Materials & Energy.
+
+- 📧 **pradeep@xpertsystems.ai**
+- 🌐 **https://xpertsystems.ai**
+- 🗂  Dataset: https://huggingface.co/datasets/xpertsystems/cyb008-sample
+- 🤖 Companion models:
+  - https://huggingface.co/xpertsystems/cyb001-baseline-classifier (network traffic)
+  - https://huggingface.co/xpertsystems/cyb002-baseline-classifier (ATT&CK kill-chain)
+  - https://huggingface.co/xpertsystems/cyb003-baseline-classifier (malware execution phase)
+  - https://huggingface.co/xpertsystems/cyb004-baseline-classifier (phishing campaign phase)
+  - https://huggingface.co/xpertsystems/cyb005-baseline-classifier (ransomware actor-tier attribution)
+  - https://huggingface.co/xpertsystems/cyb006-baseline-classifier (user risk tier + leakage diagnostic)
+  - https://huggingface.co/xpertsystems/cyb007-baseline-classifier (insider threat type)
+
+## Citation
+
+```bibtex
+@misc{xpertsystems_cyb008_baseline_2026,
+  title  = {CYB008 Baseline Classifier: XGBoost and MLP for SOC Alert Triage Outcome Classification, with Structural-Leakage and Unlearnable-Target Diagnostic},
+  author = {XpertSystems.ai},
+  year   = {2026},
+  url    = {https://huggingface.co/xpertsystems/cyb008-baseline-classifier},
+  note   = {Baseline reference model trained on xpertsystems/cyb008-sample}
+}
+```

ablation_results.json

@@ -0,0 +1,659 @@
+{
+  "purpose": "Quantify how much each feature group contributes to the headline XGBoost score. Identical architecture, same stratified split, with one feature group dropped at a time.",
+  "full_model_metrics": {
+    "model": "xgboost",
+    "accuracy": 0.7659420289855072,
+    "macro_f1": 0.7429876131468711,
+    "weighted_f1": 0.7669168766123218,
+    "per_class_f1": {
+      "auto_resolved_soar": 0.7572383073496659,
+      "duplicate_merged": 0.7936507936507936,
+      "false_positive_closed": 0.9038461538461539,
+      "true_positive_remediated": 0.7012987012987013,
+      "true_positive_escalated": 0.5589041095890411
+    },
+    "confusion_matrix": {
+      "labels": [
+        "auto_resolved_soar",
+        "duplicate_merged",
+        "false_positive_closed",
+        "true_positive_remediated",
+        "true_positive_escalated"
+      ],
+      "matrix": [
+        [
+          340,
+          17,
+          6,
+          16,
+          17
+        ],
+        [
+          9,
+          50,
+          0,
+          0,
+          0
+        ],
+        [
+          74,
+          0,
+          376,
+          0,
+          0
+        ],
+        [
+          40,
+          0,
+          0,
+          189,
+          48
+        ],
+        [
+          39,
+          0,
+          0,
+          57,
+          102
+        ]
+      ]
+    },
+    "macro_roc_auc_ovr": 0.9522005654044479
+  },
+  "ablations": {
+    "no_severity": {
+      "n_features": 46,
+      "dropped_count": 7,
+      "metrics": {
+        "model": "xgboost_no_severity",
+        "accuracy": 0.513768115942029,
+        "macro_f1": 0.39328452803110936,
+        "weighted_f1": 0.48887003837655496,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.8058455114822547,
+          "duplicate_merged": 0.0,
+          "false_positive_closed": 0.4,
+          "true_positive_remediated": 0.3155893536121673,
+          "true_positive_escalated": 0.4449877750611247
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              386,
+              1,
+              1,
+              3,
+              5
+            ],
+            [
+              15,
+              0,
+              17,
+              21,
+              6
+            ],
+            [
+              75,
+              30,
+              149,
+              122,
+              74
+            ],
+            [
+              42,
+              26,
+              91,
+              83,
+              35
+            ],
+            [
+              44,
+              6,
+              37,
+              20,
+              91
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.7303857388456401
+      },
+      "delta_accuracy": 0.25217391304347825,
+      "delta_macro_f1": 0.34970308511576176
+    },
+    "no_alert_source": {
+      "n_features": 45,
+      "dropped_count": 8,
+      "metrics": {
+        "model": "xgboost_no_alert_source",
+        "accuracy": 0.763768115942029,
+        "macro_f1": 0.7406277489805807,
+        "weighted_f1": 0.764708131838635,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.755011135857461,
+          "duplicate_merged": 0.784,
+          "false_positive_closed": 0.8984468339307049,
+          "true_positive_remediated": 0.6981132075471698,
+          "true_positive_escalated": 0.5675675675675675
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              339,
+              17,
+              11,
+              13,
+              16
+            ],
+            [
+              10,
+              49,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              41,
+              0,
+              0,
+              185,
+              51
+            ],
+            [
+              38,
+              0,
+              0,
+              55,
+              105
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9511218248098263
+      },
+      "delta_accuracy": 0.0021739130434782483,
+      "delta_macro_f1": 0.002359864166290415
+    },
+    "no_tactic": {
+      "n_features": 41,
+      "dropped_count": 12,
+      "metrics": {
+        "model": "xgboost_no_tactic",
+        "accuracy": 0.7811594202898551,
+        "macro_f1": 0.7655889644647986,
+        "weighted_f1": 0.7823552630061641,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7717750826901875,
+          "duplicate_merged": 0.8346456692913385,
+          "false_positive_closed": 0.908433734939759,
+          "true_positive_remediated": 0.7088122605363985,
+          "true_positive_escalated": 0.6042780748663101
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              350,
+              15,
+              3,
+              15,
+              13
+            ],
+            [
+              6,
+              53,
+              0,
+              0,
+              0
+            ],
+            [
+              73,
+              0,
+              377,
+              0,
+              0
+            ],
+            [
+              42,
+              0,
+              0,
+              185,
+              50
+            ],
+            [
+              40,
+              0,
+              0,
+              45,
+              113
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9529923809161402
+      },
+      "delta_accuracy": -0.01521739130434785,
+      "delta_macro_f1": -0.02260135131792751
+    },
+    "no_siem": {
+      "n_features": 45,
+      "dropped_count": 8,
+      "metrics": {
+        "model": "xgboost_no_siem",
+        "accuracy": 0.7681159420289855,
+        "macro_f1": 0.747392848800313,
+        "weighted_f1": 0.7695871955675133,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7577777777777778,
+          "duplicate_merged": 0.8,
+          "false_positive_closed": 0.9025270758122743,
+          "true_positive_remediated": 0.706766917293233,
+          "true_positive_escalated": 0.5698924731182796
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              341,
+              16,
+              6,
+              15,
+              18
+            ],
+            [
+              9,
+              50,
+              0,
+              0,
+              0
+            ],
+            [
+              75,
+              0,
+              375,
+              0,
+              0
+            ],
+            [
+              39,
+              0,
+              0,
+              188,
+              50
+            ],
+            [
+              40,
+              0,
+              0,
+              52,
+              106
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9521514669693077
+      },
+      "delta_accuracy": -0.0021739130434782483,
+      "delta_macro_f1": -0.0044052356534418635
+    },
+    "no_analyst_tier": {
+      "n_features": 50,
+      "dropped_count": 3,
+      "metrics": {
+        "model": "xgboost_no_analyst_tier",
+        "accuracy": 0.7717391304347826,
+        "macro_f1": 0.7470947169858246,
+        "weighted_f1": 0.7727339237745289,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.768893756845564,
+          "duplicate_merged": 0.784,
+          "false_positive_closed": 0.9071170084439083,
+          "true_positive_remediated": 0.6948176583493282,
+          "true_positive_escalated": 0.5806451612903226
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              351,
+              17,
+              3,
+              14,
+              11
+            ],
+            [
+              10,
+              49,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              41,
+              0,
+              0,
+              181,
+              55
+            ],
+            [
+              41,
+              0,
+              0,
+              49,
+              108
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9524262361561989
+      },
+      "delta_accuracy": -0.005797101449275366,
+      "delta_macro_f1": -0.004107103838953519
+    },
+    "no_timing": {
+      "n_features": 48,
+      "dropped_count": 5,
+      "metrics": {
+        "model": "xgboost_no_timing",
+        "accuracy": 0.777536231884058,
+        "macro_f1": 0.7572452763946715,
+        "weighted_f1": 0.7795520836463574,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7676991150442478,
+          "duplicate_merged": 0.8031496062992126,
+          "false_positive_closed": 0.9071170084439083,
+          "true_positive_remediated": 0.723404255319149,
+          "true_positive_escalated": 0.5848563968668408
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              347,
+              17,
+              3,
+              9,
+              20
+            ],
+            [
+              8,
+              51,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              37,
+              0,
+              0,
+              187,
+              53
+            ],
+            [
+              42,
+              0,
+              0,
+              44,
+              112
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9546713378957848
+      },
+      "delta_accuracy": -0.011594202898550732,
+      "delta_macro_f1": -0.014257663247800423
+    },
+    "no_scores": {
+      "n_features": 48,
+      "dropped_count": 5,
+      "metrics": {
+        "model": "xgboost_no_scores",
+        "accuracy": 0.7710144927536232,
+        "macro_f1": 0.7569411600325896,
+        "weighted_f1": 0.7729871790343515,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7531285551763367,
+          "duplicate_merged": 0.8253968253968254,
+          "false_positive_closed": 0.9019138755980861,
+          "true_positive_remediated": 0.7047970479704797,
+          "true_positive_escalated": 0.5994694960212201
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              331,
+              15,
+              9,
+              23,
+              18
+            ],
+            [
+              7,
+              52,
+              0,
+              0,
+              0
+            ],
+            [
+              73,
+              0,
+              377,
+              0,
+              0
+            ],
+            [
+              38,
+              0,
+              0,
+              191,
+              48
+            ],
+            [
+              34,
+              0,
+              0,
+              51,
+              113
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9541430544791097
+      },
+      "delta_accuracy": -0.005072463768115987,
+      "delta_macro_f1": -0.013953546885718482
+    },
+    "no_soar": {
+      "n_features": 52,
+      "dropped_count": 1,
+      "metrics": {
+        "model": "xgboost_no_soar",
+        "accuracy": 0.618840579710145,
+        "macro_f1": 0.5773360587813117,
+        "weighted_f1": 0.5258347983183296,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.028846153846153848,
+          "duplicate_merged": 0.8194444444444444,
+          "false_positive_closed": 0.8424068767908309,
+          "true_positive_remediated": 0.6328358208955224,
+          "true_positive_escalated": 0.5631469979296067
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              6,
+              26,
+              156,
+              122,
+              86
+            ],
+            [
+              0,
+              59,
+              0,
+              0,
+              0
+            ],
+            [
+              9,
+              0,
+              441,
+              0,
+              0
+            ],
+            [
+              2,
+              0,
+              0,
+              212,
+              63
+            ],
+            [
+              3,
+              0,
+              0,
+              59,
+              136
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.8369099942380366
+      },
+      "delta_accuracy": 0.14710144927536228,
+      "delta_macro_f1": 0.16565155436555945
+    },
+    "no_engineered": {
+      "n_features": 47,
+      "dropped_count": 6,
+      "metrics": {
+        "model": "xgboost_no_engineered",
+        "accuracy": 0.7681159420289855,
+        "macro_f1": 0.7479996795268518,
+        "weighted_f1": 0.7700206321761683,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7542087542087542,
+          "duplicate_merged": 0.796875,
+          "false_positive_closed": 0.9027611044417767,
+          "true_positive_remediated": 0.7094339622641509,
+          "true_positive_escalated": 0.5767195767195767
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              336,
+              18,
+              7,
+              13,
+              22
+            ],
+            [
+              8,
+              51,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              40,
+              0,
+              0,
+              188,
+              49
+            ],
+            [
+              37,
+              0,
+              0,
+              52,
+              109
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9533153727185603
+      },
+      "delta_accuracy": -0.0021739130434782483,
+      "delta_macro_f1": -0.00501206637998064
+    }
+  }
+}

feature_engineering.py

@@ -0,0 +1,338 @@
+"""
+feature_engineering.py
+======================
+
+Feature pipeline for the CYB008 baseline classifier.
+
+Predicts `resolution_outcome` (5-class triage outcome) from per-alert
+features on the CYB008 sample dataset.
+
+CSV inputs:
+    soc_alerts.csv          (primary, one row per alert, 9,200 alerts)
+    soc_topology.csv        (per-analyst registry; reserved for future
+                             work - 25 analysts is too small to be a
+                             useful join target beyond the analyst_tier
+                             column already on soc_alerts)
+    incident_summary.csv    (per-incident aggregates; reserved - only
+                             9% of alerts link to an incident)
+    alert_events.csv        (discrete alert event log; reserved)
+
+Target classes (5):
+    auto_resolved_soar, duplicate_merged, false_positive_closed,
+    true_positive_escalated, true_positive_remediated
+
+Grouping decision
+-----------------
+There is no natural row-level group key for CYB008:
+- 25 analysts -> group-aware split would yield ~4 test analysts
+- 5 SOCs     -> group-aware split would yield ~1 test SOC
+- 589 incidents -> only 9% of alerts have a non-null incident_id
+
+This baseline uses STRATIFIED random splitting (like CYB001 for network
+flows), which is the right choice when alerts are independent given
+features. The model card documents this rationale.
+
+Leakage audit
+-------------
+Three columns are structural oracles for resolution_outcome and are
+DROPPED from the feature set:
+
+1. `alert_lifecycle_phase` (4 values: auto_closed, escalated, resolved,
+   suppressed_duplicate): three of the four values map deterministically
+   to specific resolution_outcome classes. Drop.
+
+2. `automation_resolved` (binary): exactly 1:1 with auto_resolved_soar
+   outcome. Drop.
+
+3. `escalation_flag` (binary): near-1:1 with true_positive_escalated
+   outcome (1319 escalation flags = 1319 escalated outcomes). Drop.
+
+With all three dropped, accuracy drops from 100% to 79% - confirming
+they were structural oracles, not real predictive signal.
+
+`soar_playbook_triggered` is a PARTIAL oracle (one-way necessary
+condition: auto_resolved_soar => soar_playbook_triggered=1, but
+soar_playbook_triggered=1 also yields 32% non-auto-resolve outcomes).
+This is a legitimate observable - a SOAR playbook actually executing
+is part of how the alert is triaged. KEPT.
+
+`mitre_technique_id` is a perfect oracle for mitre_tactic (every T-
+number belongs to one tactic by ATT&CK design) but has no relationship
+to resolution_outcome. It is high-cardinality (36 values from a small
+sample of a 600+-value enterprise space) and contributes nothing to
+this task. Dropped for parsimony.
+
+`detection_rule_id` has 656 unique values - too high-cardinality for
+one-hot encoding. Dropped.
+
+Identifier / non-feature columns
+--------------------------------
+Dropped: alert_id, incident_id (mostly null), analyst_id, soc_id,
+shift_id, alert_timestamp_min, soar_playbook_id (high cardinality).
+
+Public API
+----------
+    build_features(alerts_path) -> (X, y, ids, meta)
+    transform_single(record, meta) -> np.ndarray
+    save_meta(meta, path) / load_meta(path)
+
+License
+-------
+Ships with the public model on Hugging Face under CC-BY-NC-4.0,
+matching the dataset license. See README.md.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+# ---------------------------------------------------------------------------
+# Label space
+# ---------------------------------------------------------------------------
+
+# Ordered by triage spectrum: auto -> dup -> FP -> TP-remediate -> TP-escalate
+LABEL_ORDER = [
+    "auto_resolved_soar",
+    "duplicate_merged",
+    "false_positive_closed",
+    "true_positive_remediated",
+    "true_positive_escalated",
+]
+LABEL_TO_INT = {lbl: i for i, lbl in enumerate(LABEL_ORDER)}
+INT_TO_LABEL = {i: lbl for lbl, i in LABEL_TO_INT.items()}
+
+# ---------------------------------------------------------------------------
+# Identifier and target columns
+# ---------------------------------------------------------------------------
+
+ID_COLUMNS = [
+    "alert_id", "incident_id", "analyst_id", "soc_id", "shift_id",
+    "alert_timestamp_min", "soar_playbook_id",
+]
+TARGET_COLUMN = "resolution_outcome"
+
+# Structural oracle columns - dropped from features.
+ORACLE_COLUMNS = [
+    "alert_lifecycle_phase",  # deterministically maps to 3 of 5 outcomes
+    "automation_resolved",    # 1:1 with auto_resolved_soar outcome
+    "escalation_flag",        # 1:1 with true_positive_escalated outcome
+]
+
+# High-cardinality categorical columns - dropped for tractability.
+HIGH_CARDINALITY_COLUMNS = [
+    "mitre_technique_id",  # 36 values; no relationship to outcome
+    "detection_rule_id",   # 656 values; one-hot explosion
+]
+
+DROPPED_FROM_FEATURES = ORACLE_COLUMNS + HIGH_CARDINALITY_COLUMNS
+
+# ---------------------------------------------------------------------------
+# Per-alert numeric features
+# ---------------------------------------------------------------------------
+
+DIRECT_NUMERIC_FEATURES = [
+    "raw_score",
+    "enriched_score",
+    "time_in_phase_minutes",
+    "queue_depth_at_ingestion",
+    "soar_playbook_triggered",  # partial oracle, kept as observable
+    "sla_breached_flag",
+    "mttd_minutes",
+    "mttr_minutes",
+    "fatigue_score_at_alert",
+]
+
+CATEGORICAL_FEATURES = [
+    "alert_severity",      # 7 values
+    "alert_source",        # 8 values
+    "mitre_tactic",        # 12 values
+    "analyst_tier",        # 3 values (alerts) / 4 (topology) -- 3 here
+    "siem_platform",       # 8 values
+]
+
+
+# ---------------------------------------------------------------------------
+# Engineered features
+# ---------------------------------------------------------------------------
+
+def _add_engineered_features(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Six engineered features encoding triage-outcome hypotheses.
+    Each composite is a quantity a SOC analyst would compute by hand
+    to assess an alert's likely disposition.
+    """
+    df = df.copy()
+
+    # 1. Enrichment lift: how much enrichment improved the raw score.
+    #    Positive lift = enrichment increased confidence (often -> TP).
+    df["enrichment_lift"] = (
+        df["enriched_score"] - df["raw_score"]
+    ).astype(float)
+
+    # 2. Log-scaled MTTR. MTTR is heavy-tailed (auto-resolves seconds,
+    #    escalations hours). log1p compresses for both XGBoost and MLP.
+    df["log_mttr"] = np.log1p(df["mttr_minutes"].clip(lower=0)).astype(float)
+
+    # 3. Log-scaled MTTD. Same heavy-tail shape.
+    df["log_mttd"] = np.log1p(df["mttd_minutes"].clip(lower=0)).astype(float)
+
+    # 4. Queue pressure: queue depth times analyst fatigue. High =
+    #    overloaded analyst, more likely to auto-resolve or escalate.
+    df["queue_pressure"] = (
+        df["queue_depth_at_ingestion"] * df["fatigue_score_at_alert"]
+    ).astype(float)
+
+    # 5. Triage time efficiency: enrichment_score per minute in phase.
+    df["enrichment_per_minute"] = (
+        df["enriched_score"] / df["time_in_phase_minutes"].clip(lower=0.1)
+    ).astype(float)
+
+    # 6. Is high-confidence alert: enriched score above 0.7 typically
+    #    indicates a strong signal that warrants escalation.
+    df["is_high_confidence"] = (df["enriched_score"] > 0.7).astype(int)
+
+    return df
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+def build_features(
+    alerts_path: str | Path,
+) -> tuple[pd.DataFrame, pd.Series, pd.Series, dict[str, Any]]:
+    """
+    Load soc_alerts.csv, drop target + identifiers + oracle columns,
+    engineer features, one-hot encode, return (X, y, ids, meta).
+
+    `ids` is a Series of alert_id values aligned with X (used for
+    round-tripping; not a group label since this task uses stratified
+    random splitting).
+    """
+    alerts = pd.read_csv(alerts_path)
+
+    y = alerts[TARGET_COLUMN].map(LABEL_TO_INT)
+    if y.isna().any():
+        bad = alerts.loc[y.isna(), TARGET_COLUMN].unique()
+        raise ValueError(f"Unknown resolution_outcome values: {bad}")
+    y = y.astype(int)
+    ids = alerts["alert_id"].copy()
+
+    alerts = alerts.drop(
+        columns=ID_COLUMNS + [TARGET_COLUMN] + DROPPED_FROM_FEATURES,
+        errors="ignore",
+    )
+
+    alerts = _add_engineered_features(alerts)
+
+    numeric_features = (
+        DIRECT_NUMERIC_FEATURES
+        + [
+            "enrichment_lift", "log_mttr", "log_mttd",
+            "queue_pressure", "enrichment_per_minute", "is_high_confidence",
+        ]
+    )
+    numeric_features = [c for c in numeric_features if c in alerts.columns]
+    X_numeric = alerts[numeric_features].astype(float)
+
+    categorical_levels: dict[str, list[str]] = {}
+    blocks: list[pd.DataFrame] = []
+    for col in CATEGORICAL_FEATURES:
+        if col not in alerts.columns:
+            continue
+        levels = sorted(alerts[col].dropna().unique().tolist())
+        categorical_levels[col] = levels
+        block = pd.get_dummies(
+            alerts[col].astype("category").cat.set_categories(levels),
+            prefix=col, dummy_na=False,
+        ).astype(int)
+        blocks.append(block)
+
+    X = pd.concat(
+        [X_numeric.reset_index(drop=True)]
+        + [b.reset_index(drop=True) for b in blocks],
+        axis=1,
+    ).fillna(0.0)
+
+    meta = {
+        "feature_names": X.columns.tolist(),
+        "numeric_features": numeric_features,
+        "categorical_levels": categorical_levels,
+        "label_to_int": LABEL_TO_INT,
+        "int_to_label": INT_TO_LABEL,
+        "oracle_excluded": ORACLE_COLUMNS,
+        "high_cardinality_excluded": HIGH_CARDINALITY_COLUMNS,
+    }
+    return X, y, ids, meta
+
+
+def transform_single(
+    record: dict | pd.DataFrame,
+    meta: dict[str, Any],
+) -> np.ndarray:
+    """Encode a single alert record for inference."""
+    if isinstance(record, dict):
+        df = pd.DataFrame([record.copy()])
+    else:
+        df = record.copy()
+
+    df = _add_engineered_features(df)
+
+    numeric = pd.DataFrame({
+        col: df.get(col, pd.Series([0.0] * len(df))).astype(float).values
+        for col in meta["numeric_features"]
+    })
+    blocks: list[pd.DataFrame] = [numeric]
+    for col, levels in meta["categorical_levels"].items():
+        val = df.get(col, pd.Series([None] * len(df)))
+        block = pd.get_dummies(
+            val.astype("category").cat.set_categories(levels),
+            prefix=col, dummy_na=False,
+        ).astype(int)
+        for lvl in levels:
+            cname = f"{col}_{lvl}"
+            if cname not in block.columns:
+                block[cname] = 0
+        block = block[[f"{col}_{lvl}" for lvl in levels]]
+        blocks.append(block)
+
+    X = pd.concat(blocks, axis=1).fillna(0.0)
+    X = X.reindex(columns=meta["feature_names"], fill_value=0.0)
+    return X.values.astype(np.float32)
+
+
+def save_meta(meta: dict[str, Any], path: str | Path) -> None:
+    serializable = {
+        "feature_names": meta["feature_names"],
+        "numeric_features": meta["numeric_features"],
+        "categorical_levels": meta["categorical_levels"],
+        "label_to_int": meta["label_to_int"],
+        "int_to_label": {str(k): v for k, v in meta["int_to_label"].items()},
+        "oracle_excluded": meta.get("oracle_excluded", []),
+        "high_cardinality_excluded": meta.get("high_cardinality_excluded", []),
+    }
+    with open(path, "w") as f:
+        json.dump(serializable, f, indent=2)
+
+
+def load_meta(path: str | Path) -> dict[str, Any]:
+    with open(path) as f:
+        meta = json.load(f)
+    meta["int_to_label"] = {int(k): v for k, v in meta["int_to_label"].items()}
+    return meta
+
+
+if __name__ == "__main__":
+    import sys
+    base = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/mnt/user-data/uploads")
+    X, y, ids, meta = build_features(base / "soc_alerts.csv")
+    print(f"X shape: {X.shape}")
+    print(f"y shape: {y.shape}")
+    print(f"n_features: {len(meta['feature_names'])}")
+    print(f"label distribution:\n{y.map(INT_TO_LABEL).value_counts()}")
+    print(f"X has NaN: {X.isnull().any().any()}")

feature_meta.json

@@ -0,0 +1,147 @@
+{
+  "feature_names": [
+    "raw_score",
+    "enriched_score",
+    "time_in_phase_minutes",
+    "queue_depth_at_ingestion",
+    "soar_playbook_triggered",
+    "sla_breached_flag",
+    "mttd_minutes",
+    "mttr_minutes",
+    "fatigue_score_at_alert",
+    "enrichment_lift",
+    "log_mttr",
+    "log_mttd",
+    "queue_pressure",
+    "enrichment_per_minute",
+    "is_high_confidence",
+    "alert_severity_critical_confirmed",
+    "alert_severity_duplicate_suppressed",
+    "alert_severity_false_positive",
+    "alert_severity_high_severity",
+    "alert_severity_informational",
+    "alert_severity_low_severity",
+    "alert_severity_medium_severity",
+    "alert_source_cspm_cloud_rule",
+    "alert_source_edr_behavioural_engine",
+    "alert_source_honeypot_trigger",
+    "alert_source_itdr_identity_anomaly",
+    "alert_source_nids_signature",
+    "alert_source_siem_correlation_rule",
+    "alert_source_threat_intel_ioc_match",
+    "alert_source_ueba_user_anomaly",
+    "mitre_tactic_collection",
+    "mitre_tactic_command_and_control",
+    "mitre_tactic_credential_access",
+    "mitre_tactic_defense_evasion",
+    "mitre_tactic_discovery",
+    "mitre_tactic_execution",
+    "mitre_tactic_exfiltration",
+    "mitre_tactic_impact",
+    "mitre_tactic_initial_access",
+    "mitre_tactic_lateral_movement",
+    "mitre_tactic_persistence",
+    "mitre_tactic_privilege_escalation",
+    "analyst_tier_L1_junior",
+    "analyst_tier_L2_senior",
+    "analyst_tier_L3_threat_hunter",
+    "siem_platform_chronicle_google",
+    "siem_platform_elastic_siem",
+    "siem_platform_exabeam_fusion",
+    "siem_platform_ibm_qradar",
+    "siem_platform_logrhythm_axon",
+    "siem_platform_microsoft_sentinel",
+    "siem_platform_splunk_enterprise",
+    "siem_platform_sumo_logic"
+  ],
+  "numeric_features": [
+    "raw_score",
+    "enriched_score",
+    "time_in_phase_minutes",
+    "queue_depth_at_ingestion",
+    "soar_playbook_triggered",
+    "sla_breached_flag",
+    "mttd_minutes",
+    "mttr_minutes",
+    "fatigue_score_at_alert",
+    "enrichment_lift",
+    "log_mttr",
+    "log_mttd",
+    "queue_pressure",
+    "enrichment_per_minute",
+    "is_high_confidence"
+  ],
+  "categorical_levels": {
+    "alert_severity": [
+      "critical_confirmed",
+      "duplicate_suppressed",
+      "false_positive",
+      "high_severity",
+      "informational",
+      "low_severity",
+      "medium_severity"
+    ],
+    "alert_source": [
+      "cspm_cloud_rule",
+      "edr_behavioural_engine",
+      "honeypot_trigger",
+      "itdr_identity_anomaly",
+      "nids_signature",
+      "siem_correlation_rule",
+      "threat_intel_ioc_match",
+      "ueba_user_anomaly"
+    ],
+    "mitre_tactic": [
+      "collection",
+      "command_and_control",
+      "credential_access",
+      "defense_evasion",
+      "discovery",
+      "execution",
+      "exfiltration",
+      "impact",
+      "initial_access",
+      "lateral_movement",
+      "persistence",
+      "privilege_escalation"
+    ],
+    "analyst_tier": [
+      "L1_junior",
+      "L2_senior",
+      "L3_threat_hunter"
+    ],
+    "siem_platform": [
+      "chronicle_google",
+      "elastic_siem",
+      "exabeam_fusion",
+      "ibm_qradar",
+      "logrhythm_axon",
+      "microsoft_sentinel",
+      "splunk_enterprise",
+      "sumo_logic"
+    ]
+  },
+  "label_to_int": {
+    "auto_resolved_soar": 0,
+    "duplicate_merged": 1,
+    "false_positive_closed": 2,
+    "true_positive_remediated": 3,
+    "true_positive_escalated": 4
+  },
+  "int_to_label": {
+    "0": "auto_resolved_soar",
+    "1": "duplicate_merged",
+    "2": "false_positive_closed",
+    "3": "true_positive_remediated",
+    "4": "true_positive_escalated"
+  },
+  "oracle_excluded": [
+    "alert_lifecycle_phase",
+    "automation_resolved",
+    "escalation_flag"
+  ],
+  "high_cardinality_excluded": [
+    "mitre_technique_id",
+    "detection_rule_id"
+  ]
+}

feature_scaler.json

@@ -0,0 +1 @@
+{"mean": [0.38312180124223605, 0.44073427018633543, 494.7303866459627, 0.0, 0.42220496894409937, 0.1781055900621118, 137.2831350931677, 494.7303866459627, 0.6417724378881986, 0.05761246894409938, 6.162163956057129, 4.838456166953409, 0.0, 0.0009746911233458116, 0.12018633540372671, 0.025, 0.06164596273291925, 0.4549689440993789, 0.07577639751552795, 0.08214285714285714, 0.12950310559006212, 0.17096273291925465, 0.12298136645962733, 0.12406832298136646, 0.13198757763975155, 0.1253105590062112, 0.12593167701863353, 0.12468944099378881, 0.12111801242236025, 0.12391304347826088, 0.06055900621118013, 0.062111801242236024, 0.10170807453416149, 0.10791925465838509, 0.07872670807453416, 0.10512422360248447, 0.05031055900621118, 0.05015527950310559, 0.13788819875776398, 0.06506211180124223, 0.08649068322981367, 0.09394409937888198, 0.7020186335403726, 0.21521739130434783, 0.0827639751552795, 0.044099378881987575, 0.19145962732919256, 0.1203416149068323, 0.20543478260869566, 0.09208074534161491, 0.15434782608695652, 0.0891304347826087, 0.1031055900621118], "std": [0.17850135030508904, 0.20892201626750886, 146.90053054989468, 1.0, 0.49394920704438045, 0.3826313144711351, 58.62096838689685, 146.90053054989468, 0.1734504703334948, 0.04815657263795656, 0.29940309845142277, 0.43629738062432477, 1.0, 0.0005739726052285025, 0.325204554451481, 0.15613707287413436, 0.24053008473802837, 0.4980067419072385, 0.2646605593600251, 0.2746035639662693, 0.33578201102512484, 0.3765056291068823, 0.3284413197786903, 0.3296850798759676, 0.3385035444954766, 0.33109642900410163, 0.33179810798133247, 0.33039209198094494, 0.3262897045833066, 0.32950790685171577, 0.23853814883841196, 0.24137724091987214, 0.3022874975869994, 0.3103024985870334, 0.26933265211902974, 0.3067372346334799, 0.21860198300436606, 0.2182822165526327, 0.34480937505453896, 0.2466545770469458, 0.2811090811235689, 0.29177358481916393, 0.4574067767634188, 0.4110049834120366, 0.2755465283870223, 0.20533185439625573, 0.3934804694938916, 0.3253858494057618, 0.4040503472586642, 0.28916235120376915, 0.3613099024493325, 0.28495404697734833, 0.304120352879204]}

inference_example.ipynb

@@ -0,0 +1,311 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# CYB008 Baseline Classifier — Inference Example\n",
+    "\n",
+    "End-to-end demo: load the trained XGBoost and PyTorch MLP models from the Hugging Face repo and predict the **SOC alert triage outcome** from a per-alert record.\n",
+    "\n",
+    "**Models predict one of 5 outcome classes:** `auto_resolved_soar`, `duplicate_merged`, `false_positive_closed`, `true_positive_remediated`, `true_positive_escalated`.\n",
+    "\n",
+    "**This is a baseline reference model**, not a production SOC triage system. See the model card and **especially `leakage_diagnostic.json`** for the structural-leakage findings (three columns were dropped as oracles)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Install dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install --quiet xgboost torch safetensors pandas numpy huggingface_hub"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Download model artifacts from Hugging Face"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import hf_hub_download\n",
+    "\n",
+    "REPO_ID = \"xpertsystems/cyb008-baseline-classifier\"\n",
+    "\n",
+    "files = {}\n",
+    "for name in [\"model_xgb.json\", \"model_mlp.safetensors\",\n",
+    "             \"feature_engineering.py\", \"feature_meta.json\",\n",
+    "             \"feature_scaler.json\"]:\n",
+    "    files[name] = hf_hub_download(repo_id=REPO_ID, filename=name)\n",
+    "    print(f\"  downloaded: {name}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import sys, os\n",
+    "fe_dir = os.path.dirname(files[\"feature_engineering.py\"])\n",
+    "if fe_dir not in sys.path:\n",
+    "    sys.path.insert(0, fe_dir)\n",
+    "\n",
+    "from feature_engineering import transform_single, load_meta, INT_TO_LABEL"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Load models and metadata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import json\n",
+    "import numpy as np\n",
+    "import torch\n",
+    "import torch.nn as nn\n",
+    "import xgboost as xgb\n",
+    "from safetensors.torch import load_file\n",
+    "\n",
+    "meta = load_meta(files[\"feature_meta.json\"])\n",
+    "with open(files[\"feature_scaler.json\"]) as f:\n",
+    "    scaler = json.load(f)\n",
+    "\n",
+    "N_FEATURES = len(meta[\"feature_names\"])\n",
+    "N_CLASSES = len(meta[\"int_to_label\"])\n",
+    "print(f\"feature count: {N_FEATURES}\")\n",
+    "print(f\"class count:   {N_CLASSES}\")\n",
+    "print(f\"label classes: {list(meta['int_to_label'].values())}\")\n",
+    "print(f\"\\noracle columns excluded (do not pass these to the model):\")\n",
+    "for c in meta.get(\"oracle_excluded\", []):\n",
+    "    print(f\"  - {c}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xgb_model = xgb.XGBClassifier()\n",
+    "xgb_model.load_model(files[\"model_xgb.json\"])\n",
+    "\n",
+    "# MLP architecture (must match training)\n",
+    "class TriageMLP(nn.Module):\n",
+    "    def __init__(self, n_features, n_classes=5, hidden1=128, hidden2=64, dropout=0.3):\n",
+    "        super().__init__()\n",
+    "        self.net = nn.Sequential(\n",
+    "            nn.Linear(n_features, hidden1),\n",
+    "            nn.BatchNorm1d(hidden1),\n",
+    "            nn.ReLU(),\n",
+    "            nn.Dropout(dropout),\n",
+    "            nn.Linear(hidden1, hidden2),\n",
+    "            nn.BatchNorm1d(hidden2),\n",
+    "            nn.ReLU(),\n",
+    "            nn.Dropout(dropout),\n",
+    "            nn.Linear(hidden2, n_classes),\n",
+    "        )\n",
+    "    def forward(self, x):\n",
+    "        return self.net(x)\n",
+    "\n",
+    "mlp_model = TriageMLP(N_FEATURES, n_classes=N_CLASSES)\n",
+    "mlp_model.load_state_dict(load_file(files[\"model_mlp.safetensors\"]))\n",
+    "mlp_model.eval()\n",
+    "print(\"models loaded\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Prediction helper"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "MU = np.array(scaler[\"mean\"], dtype=np.float32)\n",
+    "SD = np.array(scaler[\"std\"],  dtype=np.float32)\n",
+    "\n",
+    "def predict_triage_outcome(record: dict) -> dict:\n",
+    "    \"\"\"Predict the resolution outcome for one SOC alert record.\n",
+    "\n",
+    "    Note: do NOT include alert_lifecycle_phase, automation_resolved,\n",
+    "    or escalation_flag in the record. These were structural oracles\n",
+    "    in the training data and are excluded from the feature set.\n",
+    "    \"\"\"\n",
+    "    X = transform_single(record, meta)\n",
+    "\n",
+    "    xgb_proba = xgb_model.predict_proba(X)[0]\n",
+    "    xgb_label = INT_TO_LABEL[int(np.argmax(xgb_proba))]\n",
+    "\n",
+    "    Xs = ((X - MU) / SD).astype(np.float32)\n",
+    "    with torch.no_grad():\n",
+    "        logits = mlp_model(torch.tensor(Xs))\n",
+    "        mlp_proba = torch.softmax(logits, dim=1).numpy()[0]\n",
+    "    mlp_label = INT_TO_LABEL[int(np.argmax(mlp_proba))]\n",
+    "\n",
+    "    return {\n",
+    "        \"xgboost\": {\n",
+    "            \"label\": xgb_label,\n",
+    "            \"probabilities\": {INT_TO_LABEL[i]: float(p) for i, p in enumerate(xgb_proba)},\n",
+    "        },\n",
+    "        \"mlp\": {\n",
+    "            \"label\": mlp_label,\n",
+    "            \"probabilities\": {INT_TO_LABEL[i]: float(p) for i, p in enumerate(mlp_proba)},\n",
+    "        },\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Run on an example record\n",
+    "\n",
+    "Real high-severity ITDR identity-anomaly alert assigned to an L3 threat hunter, who escalated it to a true-positive incident. Both models should predict `true_positive_escalated` or the adjacent `true_positive_remediated`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Real alert from the sample dataset (true outcome: true_positive_escalated)\n",
+    "example_record = {\n",
+    "    \"alert_severity\": \"high_severity\",\n",
+    "    \"alert_source\": \"itdr_identity_anomaly\",\n",
+    "    \"mitre_tactic\": \"initial_access\",\n",
+    "    \"analyst_tier\": \"L3_threat_hunter\",\n",
+    "    \"siem_platform\": \"logrhythm_axon\",\n",
+    "    \"raw_score\": 0.2683,\n",
+    "    \"enriched_score\": 0.343,\n",
+    "    \"time_in_phase_minutes\": 429.26,\n",
+    "    \"queue_depth_at_ingestion\": 0,\n",
+    "    \"soar_playbook_triggered\": 0,\n",
+    "    \"sla_breached_flag\": 1,\n",
+    "    \"mttd_minutes\": 177.47,\n",
+    "    \"mttr_minutes\": 429.26,\n",
+    "    \"fatigue_score_at_alert\": 0.3805,\n",
+    "}\n",
+    "\n",
+    "result = predict_triage_outcome(example_record)\n",
+    "\n",
+    "print(f\"XGBoost  ->  {result['xgboost']['label']}\")\n",
+    "for lbl, p in sorted(result['xgboost']['probabilities'].items(), key=lambda x: -x[1]):\n",
+    "    print(f\"    P({lbl:30s}) = {p:.4f}\")\n",
+    "\n",
+    "print(f\"\\nMLP      ->  {result['mlp']['label']}\")\n",
+    "for lbl, p in sorted(result['mlp']['probabilities'].items(), key=lambda x: -x[1]):\n",
+    "    print(f\"    P({lbl:30s}) = {p:.4f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Honest confusion between TP-remediated and TP-escalated\n",
+    "\n",
+    "The two `true_positive_*` outcomes look behaviourally similar in the data — both involve genuine threats. They differ by whether the alert was closed by the original analyst (remediated) or passed to a higher tier (escalated). When the trained models confuse these two classes on individual alerts, that's honest learning — not a defect.\n",
+    "\n",
+    "In a production triage workflow, the better operational metric is **TP vs FP** (recall on true positives, regardless of remediated/escalated). The published baseline achieves ROC-AUC 0.955 on the full 5-class task, which substantially exceeds practical thresholds for downstream binary TP-vs-FP decisions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Batch prediction on the sample dataset"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from huggingface_hub import snapshot_download\n",
+    "import pandas as pd\n",
+    "\n",
+    "ds_path = snapshot_download(repo_id=\"xpertsystems/cyb008-sample\", repo_type=\"dataset\")\n",
+    "alerts = pd.read_csv(f\"{ds_path}/soc_alerts.csv\")\n",
+    "\n",
+    "# Score the first 500 alerts\n",
+    "sample = alerts.head(500).copy()\n",
+    "preds = [predict_triage_outcome(row.to_dict())[\"xgboost\"][\"label\"] for _, row in sample.iterrows()]\n",
+    "sample[\"xgb_pred\"] = preds\n",
+    "\n",
+    "ct = pd.crosstab(sample[\"resolution_outcome\"], sample[\"xgb_pred\"],\n",
+    "                 rownames=[\"true\"], colnames=[\"pred\"])\n",
+    "print(\"Confusion on first 500 sample alerts (XGBoost):\")\n",
+    "print(ct)\n",
+    "acc = (sample[\"resolution_outcome\"] == sample[\"xgb_pred\"]).mean()\n",
+    "print(f\"\\nbatch accuracy on first 500 alerts (in-distribution): {acc:.4f}\")\n",
+    "print(\"\\nNote: this includes training-set alerts. See validation_results.json\\n\"\n",
+    "      \"for proper held-out test metrics.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Important reading: the leakage diagnostic\n",
+    "\n",
+    "Before using CYB008 sample data to train your own triage model, read **`leakage_diagnostic.json`** in this repo. The CYB008 sample has three columns (`alert_lifecycle_phase`, `automation_resolved`, `escalation_flag`) that structurally encode the resolution_outcome label. With these columns present, a plain XGBoost achieves 100% accuracy that does not reflect real learning. The published baseline excludes them; the diagnostic file shows the cumulative ablation.\n",
+    "\n",
+    "The diagnostic also documents that **mitre_tactic prediction is unlearnable on this sample** (acc 0.08 vs majority 0.14). The README lists this as a top suggested use case, but the per-tactic feature distributions are too similar to learn from."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Next steps\n",
+    "\n",
+    "- See `validation_results.json` for held-out test metrics (1,380 alerts).\n",
+    "- See `multi_seed_results.json` for the across-10-seeds picture (accuracy 0.777 ± 0.007, ROC-AUC 0.955 ± 0.003).\n",
+    "- See `ablation_results.json` for per-feature-group contribution. Alert severity carries the dominant signal (−25 pp accuracy when removed); the SOAR-playbook-triggered indicator is second (−15 pp).\n",
+    "- See **`leakage_diagnostic.json`** for the full structural-leakage and unlearnable-target audit.\n",
+    "- For the full ~335k-row CYB008 dataset and commercial licensing, contact **pradeep@xpertsystems.ai**."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

leakage_diagnostic.json

@@ -0,0 +1,77 @@
+{
+  "purpose": "Document the three structural oracle columns dropped from the primary feature pipeline, and the unlearnable-target finding for mitre_tactic. CYB008 is calibrated against 12 SOC-operations benchmarks but encodes the resolution_outcome label structurally into alert_lifecycle_phase, automation_resolved, and escalation_flag. Real SOC telemetry has substantial overlap between these signals; the sample does not.",
+  "primary_target": "resolution_outcome (5-class)",
+  "split": "StratifiedShuffleSplit, 70/15/15 nested",
+  "oracle_structural_findings": {
+    "alert_lifecycle_phase": {
+      "deterministic_mapping": {
+        "auto_closed": "100% -> auto_resolved_soar",
+        "escalated": "100% -> true_positive_escalated",
+        "suppressed_duplicate": "100% -> duplicate_merged",
+        "resolved": "splits ~62/38 false_positive_closed / true_positive_remediated"
+      },
+      "note": "3 of 4 lifecycle phases are perfect class oracles. Drop required to evaluate honest learning."
+    },
+    "automation_resolved": {
+      "deterministic_mapping": {
+        "1": "100% -> auto_resolved_soar",
+        "0": "0 cases of auto_resolved_soar"
+      },
+      "note": "Exact 1:1 oracle with auto_resolved_soar outcome class."
+    },
+    "escalation_flag": {
+      "deterministic_mapping": {
+        "1 (n=1875)": "1319 true_positive_escalated + 556 auto_resolved_soar",
+        "0 (n=7325)": "0 cases of true_positive_escalated"
+      },
+      "note": "Near-perfect oracle for true_positive_escalated outcome."
+    }
+  },
+  "ablation_experiments": [
+    {
+      "config": "full features (all oracles intact)",
+      "n_features": 53,
+      "accuracy": 1.0,
+      "roc_auc": 1.0
+    },
+    {
+      "config": "cumulative drop through alert_lifecycle_phase",
+      "dropped_so_far": [
+        "alert_lifecycle_phase"
+      ],
+      "n_features": 49,
+      "accuracy": 1.0,
+      "roc_auc": 1.0
+    },
+    {
+      "config": "cumulative drop through automation_resolved",
+      "dropped_so_far": [
+        "alert_lifecycle_phase",
+        "automation_resolved"
+      ],
+      "n_features": 48,
+      "accuracy": 0.8388888888888889,
+      "roc_auc": 0.9726930344746759
+    },
+    {
+      "config": "cumulative drop through escalation_flag",
+      "dropped_so_far": [
+        "alert_lifecycle_phase",
+        "automation_resolved",
+        "escalation_flag"
+      ],
+      "n_features": 47,
+      "accuracy": 0.7898550724637681,
+      "roc_auc": 0.9562439021017856
+    }
+  ],
+  "conclusion": "With all three oracle columns dropped, test accuracy is 0.79 (vs 1.00 with oracles intact, and 0.33 majority baseline). The honest model still ROC-AUC 0.96 on a 5-class task - real learning, real signal, no mechanical leakage. The published baseline trains with the three oracle columns excluded.",
+  "mitre_tactic_unlearnable": {
+    "purpose": "The CYB008 README's first suggested use case is 'MITRE ATT&CK tactic classification from alert features'. We test this on the sample dataset and find it is NOT LEARNABLE - features do not distinguish tactics, the model performs below majority baseline.",
+    "task": "mitre_tactic 12-class (with mitre_technique_id excluded - it would be a perfect ATT&CK oracle)",
+    "majority_baseline_accuracy": 0.14097826086956522,
+    "xgboost_accuracy_mean_3seeds": 0.07971014492753624,
+    "interpretation": "Per-tactic feature distributions are nearly identical (raw_score 0.37-0.39, enriched_score similar, fatigue 0.64 across all 12 tactics). Without mitre_technique_id (which is a 100% ATT&CK-by-design oracle), alert_source is the only discriminating signal, and it has cross-tactic purity of 0.14 - close to random. Real SOC telemetry has stronger source-to-tactic associations and per-tactic feature distributions; the sample does not reproduce these.",
+    "recommendation_to_dataset_author": "To make tactic classification a viable benchmark, the generator should produce stronger per-tactic feature signatures (differentiated raw_score / enriched_score distributions per tactic, source-tactic affinity > 0.3 purity, characteristic MTTD / MTTR per tactic)."
+  }
+}

model_mlp.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d34f5334ddfda002098c5fa294c98908478b3037593ce06b31dbbfd4f4b672e
+size 66268

multi_seed_results.json

@@ -0,0 +1,98 @@
+{
+  "purpose": "Multi-seed evaluation across 10 stratified splits of the 9,200-alert sample. Reports XGBoost performance averaged over the full set of seeds.",
+  "seeds_evaluated": [
+    42,
+    7,
+    13,
+    17,
+    23,
+    31,
+    45,
+    99,
+    123,
+    200
+  ],
+  "per_seed": [
+    {
+      "seed": 42,
+      "test_n_classes": 5,
+      "accuracy": 0.7659420289855072,
+      "macro_f1": 0.7429876131468711,
+      "macro_roc_auc_ovr": 0.9522005654044479
+    },
+    {
+      "seed": 7,
+      "test_n_classes": 5,
+      "accuracy": 0.7768115942028986,
+      "macro_f1": 0.769435481914568,
+      "macro_roc_auc_ovr": 0.9535405274694995
+    },
+    {
+      "seed": 13,
+      "test_n_classes": 5,
+      "accuracy": 0.7862318840579711,
+      "macro_f1": 0.7773476010631033,
+      "macro_roc_auc_ovr": 0.9593350309948587
+    },
+    {
+      "seed": 17,
+      "test_n_classes": 5,
+      "accuracy": 0.7731884057971015,
+      "macro_f1": 0.7657000386460112,
+      "macro_roc_auc_ovr": 0.9510884009809615
+    },
+    {
+      "seed": 23,
+      "test_n_classes": 5,
+      "accuracy": 0.7768115942028986,
+      "macro_f1": 0.7655808630589699,
+      "macro_roc_auc_ovr": 0.9557712595581618
+    },
+    {
+      "seed": 31,
+      "test_n_classes": 5,
+      "accuracy": 0.7789855072463768,
+      "macro_f1": 0.7635031878905345,
+      "macro_roc_auc_ovr": 0.9575528903552497
+    },
+    {
+      "seed": 45,
+      "test_n_classes": 5,
+      "accuracy": 0.7920289855072464,
+      "macro_f1": 0.7827912746822961,
+      "macro_roc_auc_ovr": 0.9599146202095736
+    },
+    {
+      "seed": 99,
+      "test_n_classes": 5,
+      "accuracy": 0.7666666666666667,
+      "macro_f1": 0.7513856936195747,
+      "macro_roc_auc_ovr": 0.9498718419129876
+    },
+    {
+      "seed": 123,
+      "test_n_classes": 5,
+      "accuracy": 0.7760869565217391,
+      "macro_f1": 0.7672910648132462,
+      "macro_roc_auc_ovr": 0.9549881182366795
+    },
+    {
+      "seed": 200,
+      "test_n_classes": 5,
+      "accuracy": 0.7753623188405797,
+      "macro_f1": 0.7594433532222149,
+      "macro_roc_auc_ovr": 0.9530752276015168
+    }
+  ],
+  "aggregate": {
+    "accuracy_mean": 0.7768115942028986,
+    "accuracy_std": 0.0074957104585424775,
+    "accuracy_min": 0.7659420289855072,
+    "accuracy_max": 0.7920289855072464,
+    "macro_f1_mean": 0.7645466172057391,
+    "macro_f1_std": 0.01093479933122361,
+    "roc_auc_mean": 0.9547338482723937,
+    "roc_auc_std": 0.003234503129030738
+  },
+  "published_artifact_seed": 42
+}

validation_results.json

@@ -0,0 +1,180 @@
+{
+  "version": "1.0.0",
+  "dataset": "xpertsystems/cyb008-sample",
+  "task": "5-class resolution_outcome classification (SOC alert triage)",
+  "baselines": {
+    "always_predict_majority_accuracy": 0.32608695652173914,
+    "majority_class": "false_positive_closed",
+    "random_guess_accuracy": 0.2
+  },
+  "split": {
+    "strategy": "stratified (StratifiedShuffleSplit, nested 70/15/15)",
+    "rationale": "CYB008 has no natural row-level group key: 25 analysts (group-aware split would yield ~4 test analysts), 5 SOCs (would yield 1 test SOC), 589 incidents but only 9% of alerts have a non-null incident_id. Alerts are essentially independent given features, so stratified random split is the right choice (same approach as CYB001 for network flow classification).",
+    "alerts_train": 6440,
+    "alerts_val": 1380,
+    "alerts_test": 1380,
+    "seed": 42
+  },
+  "n_features": 53,
+  "label_classes": [
+    "auto_resolved_soar",
+    "duplicate_merged",
+    "false_positive_closed",
+    "true_positive_remediated",
+    "true_positive_escalated"
+  ],
+  "class_distribution_train": {
+    "false_positive_closed": 2097,
+    "auto_resolved_soar": 1849,
+    "true_positive_remediated": 1294,
+    "true_positive_escalated": 923,
+    "duplicate_merged": 277
+  },
+  "class_distribution_test": {
+    "false_positive_closed": 450,
+    "auto_resolved_soar": 396,
+    "true_positive_remediated": 277,
+    "true_positive_escalated": 198,
+    "duplicate_merged": 59
+  },
+  "oracle_excluded_features": [
+    "alert_lifecycle_phase (deterministically maps to 3 of 5 outcomes)",
+    "automation_resolved (1:1 with auto_resolved_soar)",
+    "escalation_flag (near 1:1 with true_positive_escalated)"
+  ],
+  "high_cardinality_excluded_features": [
+    "mitre_technique_id (36 unique values; perfect oracle for mitre_tactic but unrelated to this target)",
+    "detection_rule_id (656 unique values; one-hot explosion)"
+  ],
+  "leakage_audit_note": "See leakage_diagnostic.json for the full audit of structural oracles and the separate unlearnable-target finding for mitre_tactic. The model is trained with all three oracle columns excluded; full-features experiments showed 100% test accuracy, confirming the structural leakage.",
+  "models": {
+    "xgboost": {
+      "architecture": "Gradient-boosted decision trees, multi:softprob, 5 classes",
+      "framework": "xgboost",
+      "test_metrics": {
+        "model": "xgboost",
+        "accuracy": 0.7659420289855072,
+        "macro_f1": 0.7429876131468711,
+        "weighted_f1": 0.7669168766123218,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7572383073496659,
+          "duplicate_merged": 0.7936507936507936,
+          "false_positive_closed": 0.9038461538461539,
+          "true_positive_remediated": 0.7012987012987013,
+          "true_positive_escalated": 0.5589041095890411
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              340,
+              17,
+              6,
+              16,
+              17
+            ],
+            [
+              9,
+              50,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              40,
+              0,
+              0,
+              189,
+              48
+            ],
+            [
+              39,
+              0,
+              0,
+              57,
+              102
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9522005654044479
+      }
+    },
+    "mlp": {
+      "architecture": "PyTorch MLP, 53 -> 128 -> 64 -> 5, BatchNorm1d + ReLU + Dropout, weighted cross-entropy loss",
+      "framework": "pytorch",
+      "test_metrics": {
+        "model": "mlp",
+        "accuracy": 0.7673913043478261,
+        "macro_f1": 0.7510024599009764,
+        "weighted_f1": 0.769556192579193,
+        "per_class_f1": {
+          "auto_resolved_soar": 0.7505773672055427,
+          "duplicate_merged": 0.8251748251748252,
+          "false_positive_closed": 0.910411622276029,
+          "true_positive_remediated": 0.6981818181818182,
+          "true_positive_escalated": 0.5706666666666667
+        },
+        "confusion_matrix": {
+          "labels": [
+            "auto_resolved_soar",
+            "duplicate_merged",
+            "false_positive_closed",
+            "true_positive_remediated",
+            "true_positive_escalated"
+          ],
+          "matrix": [
+            [
+              325,
+              25,
+              0,
+              23,
+              23
+            ],
+            [
+              0,
+              59,
+              0,
+              0,
+              0
+            ],
+            [
+              74,
+              0,
+              376,
+              0,
+              0
+            ],
+            [
+              38,
+              0,
+              0,
+              192,
+              47
+            ],
+            [
+              33,
+              0,
+              0,
+              58,
+              107
+            ]
+          ]
+        },
+        "macro_roc_auc_ovr": 0.9552409409036638
+      }
+    }
+  }
+}