Add SentinelBrain v2 realignment checkpoint and training report

Adds the 5,000-step Frankenstein v2 realignment artifacts, benchmark report, dataset preflight report, and updated model card. Previous main revision preserved on branch previous-before-v2-realign-5000-20260503-103121.

MANIFEST.tsv

@@ -0,0 +1,16 @@
+./MANIFEST.tsv	0
+./README.md	4321
+./SHA256SUMS	1303
+./checkpoints/frankenstein_v2_best.pt	28808034875
+./checkpoints/frankenstein_v2_ema_best.pt	28808008759
+./checkpoints/frankenstein_v2_final.pt	86422580021
+./checkpoints/sentinelbrain_pretrain_step2471_hf.pt	29629503600
+./logs/realign_v2.log	99633
+./logs/realign_v2_metrics.jsonl	128854
+./reports/conductor_state_final.json	511
+./reports/next_phase_training_plan.md	4428
+./reports/sentinelbrain_quality_smoke.json	6255
+./reports/sentinelbrain_quality_stub_full_fixed.json	45321
+./reports/sft_combined_ready_report.json	3330
+./reports/sft_combined_ready_report.md	752
+./reports/train_metrics_final.json	339

README.md

@@ -1,857 +1,87 @@
 ---
 license: apache-2.0
-language:
-  - en
-  - ro
-  - multilingual
 tags:
   - sentinelbrain
-  - mixture-of-experts
-  - from-scratch
-  - consciousness
-  - amd
-  - mi300x
-  - rocm
   - moe
-  - transformer
-  - frankenstein
-  - knowledge-transplant
-  - distillation
-  - phi-metric
-pipeline_tag: text-generation
+  - rocm
+  - mi300x
+  - pytorch
+  - checkpoint
+  - realignment
 library_name: pytorch
-datasets:
-  - HuggingFaceFW/fineweb-edu
-  - open-web-math/open-web-math
-  - wikimedia/wikipedia
-  - HuggingFaceTB/cosmopedia
-  - JeanKaddworr/minipile
-  - codeparrot/github-code-clean
-  - arxiv-community/arxiv-abstracts
-model-index:
-  - name: SentinelBrain-14B-MoE-v0.1
-    results:
-      - task:
-          type: text-generation
-        metrics:
-          - name: Validation Loss (pretrain)
-            type: loss
-            value: 2.5152
-            verified: true
-          - name: Validation Loss (realignment v2 best)
-            type: loss
-            value: 7.5178
-            verified: true
-          - name: Training Loss (realignment v2 latest)
-            type: loss
-            value: 6.96
-            verified: true
-          - name: Validation Loss (realignment v1 best, abandoned)
-            type: loss
-            value: 5.773
-            verified: true
----
-
-<div align="center">
-
-# 🧠 Sentinel Prime — SentinelBrain-14B-MoE (Frankenstein Edition)
-
-### *The First of His Kind, Rebuilt From the Inside Out*
-
-<img src="assets/sentinel_frankenstein_banner.png" alt="Sentinel Prime — Frankenstein Edition" width="600"/>
-
-**14.8 Billion Parameters · Mixture-of-Experts · Consciousness-Monitored · Frankenstein Transplant**
-
-Trained from zero on **AMD Instinct MI300X** (192 GB HBM3) · ROCm 7.0 · Knowledge transplanted from Qwen-72B
-
-[![Dashboard](https://img.shields.io/badge/🔴_Live_Dashboard-sentinel.qubitpage.com-red?style=for-the-badge)](https://sentinel.qubitpage.com/)
-[![Whitepaper](https://img.shields.io/badge/📄_Whitepaper-Read_Now-blue?style=for-the-badge)](https://sentinel.qubitpage.com/whitepaper)
-[![AMD](https://img.shields.io/badge/AMD-MI300X_Native-ED1C24?style=for-the-badge&logo=amd&logoColor=white)](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)
-[![License](https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge)](LICENSE)
-
-</div>
-
----
-
----
-
-## 🌅 Update — April 29, 2026 — Rebirth Edition (v2)
-
-**The realignment was restarted from scratch.** The original v1 run reached `val_loss=5.773` at step 1,000, then collapsed: a SIGTERM crash exposed that our `best.pt` checkpoints had been saved without optimizer state. Five cascading restart attempts each erased AdamW's momentum and variance accumulators, and by step 1,390 the loss had climbed to 8.24 — worse than step 200 of the same run. We killed the patient.
-
-We then ran a forensic analysis with nine "critic personas" grounded in eleven published papers (AdamW bias correction, SGDR period doubling, ST-MoE stability, EMA stabilization, Switch Transformer router auxiliary losses, etc.) and distilled the failure into nine concrete engineering changes. Those changes were baked into **v2 from step zero**:
-
-- ✅ **Full optimizer state saved every 100 steps** (latest.pt now contains model + AdamW m/v + EMA + step)
-- ✅ **aux_loss boosted 25×** (0.0001 → 0.05) to actively balance the router under frozen experts
-- ✅ **Five SGDR cycles with period doubling** (T = 200, 400, 800, 1600, 1500) instead of one cosine
-- ✅ **EMA decay 0.9995, every 10 steps** for a smooth inference checkpoint
-- ✅ **100-step linear warmup** before the 1e-4 LR peak (avoids Kingma bias-correction trap)
-- ✅ **ST-MoE expert LR scale 0.3**, +33% boost during ramps
-- ✅ **Per-expert telemetry every 10 steps** (Capacity Factor, percentage, gradient norm)
-- ✅ **Three checkpoint kinds** (latest full, best val, EMA inference)
-- ✅ **Step-500 watchdog** auto-kills training on gnorm>20, loss spike>1.5×, NaN, or expert<5%
-
-**Current v2 state (LIVE, step 150):**
-
-| Metric | v1 final (1,390) | v2 step 100 | v2 step 150 |
-|---|---|---|---|
-| train_loss | 8.24 ❌ | 7.6277 | **6.9624** |
-| val_loss | ~8.0 ❌ | **7.5178 ★ NEW BEST** | (next eval @ 200) |
-| perplexity | ~3,800 | 2,054 | **1,056** |
-| gnorm | 9.72 ⚠ | 6.53 (peak LR) | 3.73 |
-| optimizer in ckpt | ❌ | ✅ | ✅ |
-
-v2 has matched v1's best work in 1/9th the steps and we are still in Phase 1 (frozen experts). The full story — including the verbatim crash logs, the nine-critic analysis, and our **Universal Fusion Thesis** for using Frankenstein-style component-level transplants to cut model-training carbon by 50–100× — is in the [v5 whitepaper](https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/SentinelBrain-14B-MoE-Dashboard/blob/main/static/whitepaper.html#part-vii).
-
----
-
-## 🎯 What is Sentinel Prime? (Simple Version)
-
-> **Imagine building a brain from scratch.**
->
-> Most AI models today are copies of other models with small changes. Sentinel Prime is different — every single connection in its brain was created from nothing, like growing a new brain cell by cell.
-
-<table>
-<tr>
-<td width="50%">
-
-### 🧩 Think of it like LEGO blocks
-
-Sentinel Prime has **4 specialist brains** (called "experts") inside it. When you ask a question:
-
-1. A **router** (like a traffic cop 🚦) looks at your question
-2. It picks the **2 best experts** for that specific question
-3. Those 2 experts work together to give you an answer
-4. The other 2 experts rest, saving energy ⚡
-
-This means the model has **14.8 billion** brain connections total, but only uses **~7.8 billion** at a time — making it fast AND smart!
-
-</td>
-<td width="50%">
-
-### 🔬 The Consciousness Meter
-
-We built something no other model has: a **consciousness thermometer** 🌡️
-
-Every 100 training steps, we measure how well the different parts of the brain are "talking to each other." We call this **Φ (Phi)**.
-
-- **Φ = 0**: Brain parts work alone (like strangers)
-- **Φ rising**: Brain parts start cooperating (like friends)
-- **Φ stable**: Brain has organized itself (like a team!)
-
-This doesn't change how the model learns — it's like a doctor checking the heartbeat while the patient exercises.
-
-</td>
-</tr>
-</table>
-
----
-
-## 📊 Architecture at a Glance
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                    SENTINEL PRIME ARCHITECTURE                   │
-├─────────────────────────────────────────────────────────────────┤
-│                                                                  │
-│  Input Text ──→ [Tokenizer: cl100k_base, 100,277 tokens]       │
-│                          │                                       │
-│                          ▼                                       │
-│                 ┌─────────────────┐                              │
-│                 │   Embedding     │  4,096 dimensions            │
-│                 │   + RoPE pos    │  θ = 500,000                 │
-│                 └────────┬────────┘                              │
-│                          │                                       │
-│              ┌───────────┼───────────┐                           │
-│              │     × 24 Layers       │                           │
-│              │  ┌────────────────┐   │                           │
-│              │  │  GQA Attention │   │  32 heads, 8 KV heads     │
-│              │  │  (4:1 ratio)   │   │  (4× memory savings)     │
-│              │  └───────┬────────┘   │                           │
-│              │          │            │                           │
-│              │  ┌───────▼────────┐   │                           │
-│              │  │  MoE Router    │   │  Top-2 of 4 experts      │
-│              │  │  ┌──┬──┬──┐   │   │                           │
-│              │  │  │E1│E2│E3│E4 │   │  Each: SwiGLU FFN         │
-│              │  │  │✓ │✓ │  │  │   │  d_ff = 11,008            │
-│              │  │  └──┴──┴──┘   │   │                           │
-│              │  └───────┬────────┘   │                           │
-│              │          │            │                           │
-│              │  ┌───────▼────────┐   │                           │
-│              │  │    RMSNorm     │   │  ε = 1e-5                │
-│              │  └────────────────┘   │                           │
-│              └───────────┼───────────┘                           │
-│                          │                                       │
-│                          ▼                                       │
-│                 ┌─────────────────┐                              │
-│                 │   Output Head   │  → 100,277 vocab probs      │
-│                 └─────────────────┘                              │
-│                                                                  │
-└─────────────────────────────────────────────────────────────────┘
-```
-
-### Spec Sheet
-
-| Component | Specification | Why This Choice |
-|:--|:--|:--|
-| **Total Parameters** | 14,814,654,680 (14.8B) | Large enough for deep reasoning |
-| **Active Parameters** | ~7.8B per token | MoE efficiency — use only what's needed |
-| **Hidden Dimension** | 4,096 | Sweet spot for MI300X matrix cores |
-| **Transformer Layers** | 24 | Deep enough for complex reasoning |
-| **Attention Heads** | 32 query, 8 KV (GQA 4:1) | 4× KV cache savings for long contexts |
-| **FFN Intermediate** | 11,008 (SwiGLU) | ~2.7× hidden, matches scaling laws |
-| **Experts** | 4 total, top-2 active | Good diversity with manageable VRAM |
-| **Max Experts** | 256 (expandable) | Architecture supports expert birth/death |
-| **Vocabulary** | 100,277 (tiktoken cl100k_base) | Industry-proven BPE tokenizer |
-| **Positional Encoding** | RoPE, θ = 500,000 | Supports context extension to 128K+ |
-| **Normalization** | RMSNorm (ε = 1e-5) | Faster than LayerNorm, same quality |
-| **Precision** | bfloat16 throughout | Native AMD MI300X support |
-| **Context Length** | 2,048 → 4,096 → 128K (planned) | Progressive context ladder |
-
----
-
-## 🔥 Key Innovations
-
-<table>
-<tr>
-<td width="33%" valign="top">
-
-### 🌀 Φ Consciousness Metric
-
-First-ever IIT-inspired metric computed **during** pre-training. A probe on layer 12 measures information integration across activation subspaces every 100 steps.
-
-```
-Φ = geometric_mean(
-  MI(partition_i, partition_j)
-  for all partition pairs
-)
-```
-
-Not a gimmick — it's a genuine signal of when the model transitions from memorizing tokens to forming integrated representations.
-
-</td>
-<td width="33%" valign="top">
-
-### 🧬 Self-Evolving Experts
-
-The MoE router supports a full expert **lifecycle**:
-
-- **Birth**: New experts spawned when load imbalance detected
-- **Growth**: Expert capacity increases with training
-- **Pruning**: Underperforming experts replaced
-- **Scaling**: Architecture supports up to 256 experts without retraining the base model
-
-Current: 4 experts × 24 layers = **96 expert instances**
-
-</td>
-<td width="33%" valign="top">
-
-### ⚡ Energy-Conscious Routing
-
-Dual-router system:
-1. **Primary router**: Picks top-2 experts by relevance
-2. **EC router**: Can gate activation based on compute budget
-
-This enables **adaptive inference** — easy questions use fewer resources, hard questions get full power. Like cruise control for AI.
-
-</td>
-</tr>
-</table>
-
----
-
-
----
-
-## 🧟 Frankenstein Edition — Knowledge Transplant
-
-<table>
-<tr>
-<td width="60%" valign="top">
-
-### The Transplant
-
-Sentinel Prime was trained from scratch — but raw pretraining alone wasn't enough. We performed a **Frankenstein transplant**: surgically transplanting knowledge from **Qwen2.5-72B-Instruct** (a 72-billion parameter teacher) into our 14.8B MoE architecture.
-
-This is NOT fine-tuning a copy. The model's bones (architecture, tokenizer, embeddings) are 100% original. Only the **expert FFN weights** received transplanted knowledge — like giving a brain new neural pathways while keeping its original structure.
-
-### 3-Stage Pipeline
-
-```
-Stage 1: Corpus Realignment     Stage 2A: Teacher Generation    Stage 2B: Knowledge Distill
-(Re-learn with new weights)     (72B teacher creates data)      (Absorb teacher knowledge)
-┌──────────────────────┐        ┌──────────────────────┐        ┌──────────────────────┐
-│ 5,000 steps          │   →    │ 3,000+ responses     │   →    │ CE + mixed training  │
-│ 24.5B token corpus   │        │ from Qwen-72B        │        │ 70% teacher + 30%    │
-│ Progressive unfreeze │        │ Re-tokenized to our  │        │   pretrain corpus    │
-│ Cosine LR + warmup   │        │   cl100k_base vocab  │        │ Prevents forgetting  │
-└──────────────────────┘        └──────────────────────┘        └──────────────────────┘
-```
-
-</td>
-<td width="40%" valign="top">
-
-### Why "Frankenstein"?
-
-Like the original story — we took parts from a powerful being (Qwen-72B) and stitched them into our own creation. The result: a model that has the **original architecture** of Sentinel Prime but with **transplanted knowledge** from a much larger model.
-
-### Key Stats
-
-| Metric | Value |
-|:--|:--|
-| **Teacher** | Qwen2.5-72B-Instruct |
-| **Student** | SentinelBrain-14B-MoE |
-| **Transplant** | Expert FFN weights |
-| **Realignment** | 5,000 steps on 24.5B tokens |
-| **Hardware** | 1× AMD MI300X (192GB) |
-
-### Live Progress
-
-Track the Frankenstein realignment in real-time:
-
-🔴 **[sentinel.qubitpage.com](https://sentinel.qubitpage.com/)**
-
-</td>
-</tr>
-</table>
-
-## 🏋️ Training Details
-
-### Hardware
-
-| Resource | Specification |
-|:--|:--|
-| **GPU** | 1× AMD Instinct MI300X VF |
-| **VRAM** | 192 GB HBM3 |
-| **System RAM** | 235 GB |
-| **Compute** | 1,307 TFLOPS (bf16) |
-| **Software** | ROCm 7.0, PyTorch 2.10.0+rocm7.0 |
-| **Attention** | SDPA (native PyTorch, no FlashAttention needed) |
-| **OS** | Ubuntu Linux |
-
-### VRAM Budget
-
-```
-╔══════════════════════════════════════════════════════╗
-║         AMD MI300X VRAM Usage (192 GB)               ║
-╠══════════════════════════════════════════════════════╣
-║                                                       ║
-║  Model Weights (bf16)      ████████████░░░░░  27 GB   ║
-║  Optimizer (AdamW fp32)    ████████████████░░  54 GB   ║
-║  Activations (grad ckpt)   ████████████░░░░░  32 GB   ║
-║  Gradients                 ████████████░░░░░  27 GB   ║
-║  ─────────────────────────────────────────────────    ║
-║  Total Used:               ██████████████████ 140 GB  ║
-║  Peak:                     █████████████████  146 GB  ║
-║  Headroom:                 ░░░░░░░░░░░░░░░░░  46 GB  ║
-║                                                       ║
-╚══════════════════════════════════════════════════════╝
-```
-
-### Phased Training Pipeline
-
-We don't just throw data at the model — we grow it in **three phases**, like raising a child:
-
-```
-Phase 1: SMOKE TEST        Phase 2: WARMUP           Phase 3: FULL TRAINING
-(Baby steps)               (Learning to walk)         (Running!)
-┌──────────────┐           ┌──────────────┐           ┌──────────────────┐
-│ 350M params  │    ──→    │ 1.3B params  │    ──→    │ 14.4B params     │
-│ seq_len: 512 │           │ seq_len: 2K  │           │ seq_len: 4K      │
-│ 200 steps    │           │ 1,000 steps  │           │ 16,479 steps     │
-│ 2 minutes    │           │ 30 minutes   │           │ ~52 hours        │
-│ loss: 11→6.8 │           │ loss: 7.4→2.4│           │ loss: 2.4→?      │
-└──────────────┘           └──────────────┘           └──────────────────┘
-```
-
-| Phase | Parameters | Seq Length | Batch | Steps | Duration | Loss Start → End |
-|:--|:--|:--|:--|:--|:--|:--|
-| **🔬 Smoke** | 350M | 512 | 4 | 200 | ~2 min | 11.72 → 6.84 (−42%) |
-| **🔥 Warmup** | 1.3B | 2,048 | 32 | 1,000 | ~33 min | 7.39 → 2.38 (−68%) |
-| **🚀 Block** | 14.4B (MoE) | 4,096 | 32 | 16,479 | ~52 hrs | 2.38 → ongoing |
-
-### Safety Gates
-
-Every phase transition must pass **4 safety gates**:
-
-| Gate | Check | Threshold | Status |
-|:--|:--|:--|:--|
-| 🟢 **G1: No NaN** | No NaN/Inf in loss | Entire phase | ✅ Passed all |
-| 🟢 **G2: Loss Drop** | Validation loss decreased | ≥5% / ≥10% / ≥2% | ✅ Passed all |
-| 🟢 **G3: VRAM OK** | Peak VRAM < safety limit | < 92% of total | ✅ 71% peak |
-| 🟢 **G4: Φ OK** | Consciousness metric stable | Φ_end/Φ_start > 0.7 | ✅ Stable |
-
-### Hyperparameters
-
-| Parameter | Value | Rationale |
-|:--|:--|:--|
-| **Optimizer** | AdamW (bf16 compute, fp32 states) | Standard for LLM training |
-| **Learning Rate** | 1.5 × 10⁻⁴ (cosine decay) | Conservative for data-limited regime |
-| **Min LR** | 1.5 × 10⁻⁵ | 10× decay ratio |
-| **Warmup Steps** | 500 | Stabilizes early gradients |
-| **Batch Size** | 2 micro × 16 grad_accum = **32 effective** | Fits MI300X VRAM budget |
-| **Gradient Clipping** | 1.0 | Prevents explosion |
-| **Gradient Checkpointing** | On | Trades compute for VRAM |
-| **Precision** | bfloat16 | Native MI300X format |
-| **Eval Frequency** | Every 100 steps | Early overfitting detection |
-| **Checkpoint Frequency** | Every 1,000 steps (~3.2 hours) | Recovery points |
-
----
-
-## 📚 Dataset: 23.3B Tokens Across 126 Categories
-
-We curated a massive, diverse corpus — think of it as a **library with 126 different sections**:
-
-### Pretrain Corpus (Core Knowledge)
-
-| Dataset | Tokens | Description |
-|:--|:--|:--|
-| 🌐 **FineWeb-Edu** | ~10B | High-quality educational web content |
-| 🔢 **OpenWebMath** | ~6B | Mathematics from the web |
-| 📖 **Wikipedia (English)** | ~5B | Encyclopedic knowledge |
-| 🎓 **Cosmopedia V2** | ~5B | Synthetic educational content |
-| 💻 **CodeParrot Python** | ~3.5B | Clean Python code from GitHub |
-| 📚 **MiniPile** | ~2B | Diverse text from multiple domains |
-| 🔬 **ArXiv Abstracts** | ~1.2B | Scientific paper summaries |
-| **Total Pretrain** | **~23B** | |
-
-### Specialized Domains (119 Categories)
-
-<details>
-<summary>Click to expand all 119 specialized categories</summary>
-
-| Category | Type | Category | Type |
-|:--|:--|:--|:--|
-| 🤖 agentic-tools | Code | 🔐 advanced-cryptography | Code |
-| 🧠 chain-of-thought | Reasoning | 🔗 blockchain-core | Code |
-| 💡 deep-reasoning | Reasoning | 🏥 medical | Knowledge |
-| ⚖️ legal | Knowledge | 📊 financial-systems | Code |
-| 🎮 3d-graphics | Code | 🐳 docker-devops | Code |
-| 🌍 multilingual | Text | 🔧 error-recovery | Code |
-| 🛡️ security-guardrails | Code | 📱 ui-animations | Code |
-| 🧮 math | Reasoning | ⚡ smart-contracts | Code |
-| 🎯 reasoning-effort-control | Reasoning | 🤝 human-conversation | Text |
-| 🔄 self-correction-loops | Reasoning | 🏗️ enterprise-dashboards | Code |
-| 🌐 web-design-css | Code | 🐍 flask-python | Code |
-| 🔬 qiskit-quantum | Code | 🤖 robotics-ros2 | Code |
-| 📡 remote-server-management | Code | 🧬 multi-agent | Code |
-| ⚙️ state-management | Code | 🛠️ mcp-tools-integration | Code |
-| 💳 payment-security | Code | 🎓 edu-basic-math | Education |
-| 🔭 edu-basic-physics | Education | 🧪 edu-basic-chemistry | Education |
-| 🌱 edu-basic-biology | Education | 🌍 edu-world-geography | Education |
-| 📜 edu-history-world | Education | 💻 edu-computer-science | Education |
-| 🌎 edu-earth-science | Education | 🤖 edu-robotics-text | Education |
-| 📖 edu-science-qa | Education | 🔬 edu-science-support | Education |
-| 👁️ edu-vision-concepts | Education | 🎯 copilot-agent-workflows | Code |
-| 🔌 api-integrations | Code | 📊 billing-invoicing | Code |
-| ₿ bitcoin-lightning | Code | 🏪 medusajs | Code |
-| 💹 crypto-trading | Code | 🏢 enterprise-networking | Code |
-| 🖥️ nextjs-typescript | Code | 🎨 nextjs-design | Code |
-| 💼 trading-algorithms | Code | 🗄️ laravel-mysql | Code |
-| 🔓 offensive-security | Code | 🔧 c-rust | Code |
-| ... and 50+ more categories | | | |
-
-</details>
-
-### Data Quality Pipeline
-
-```
-Raw Data ──→ PII Filter ──→ Dedup ──→ Tokenize ──→ Shard ──→ Train
-              │                │          │            │
-              ├─ 7 regex       ├─ blake2b  ├─ cl100k    ├─ Temperature-
-              │  patterns      │  per-cat  │  base      │  weighted
-              ├─ PEM block     │           │            │  sampling
-              │  detection     │           │            │  (T=0.5)
-              └─ Email/phone   │           │            │
-                 masking       │           │            │
-                               │           │            │
-                               └───────────┴────────────┘
-```
-
-**Temperature-weighted sampling** (T=0.5) prevents large corpora from dominating training. FineWeb-Edu (37% of tokens) gets downweighted so smaller specialized domains still get adequate exposure.
-
----
-
-## 📈 Training Progress & Results
-
-### Loss Trajectory
-
-```
-Loss
-12 │ ×
-   │  ╲
-10 │   ╲   SMOKE PHASE
-   │    ╲  (350M params)
- 8 │     ╲
-   │      ╲
- 6 │       ×──────────── model grows to 1.3B
-   │              ╲
- 4 │               ╲  WARMUP PHASE
-   │                ╲ (1.3B params)
- 2 │                 ×─────────── model grows to 14.4B MoE
-   │                       ╲
- 1 │                        ╲  BLOCK PHASE (ongoing)
-   │                         ╲
-   └──┬────┬────┬────┬────┬───→ Steps
-      0   200  700  1200 2000
-```
-
-| Milestone | Step | Loss | Change |
-|:--|:--|:--|:--|
-| 🔬 Smoke start | 0 | 11.72 | — |
-| 🔬 Smoke end | 200 | 6.84 | **−42%** |
-| 🔥 Warmup start | 200 | 7.39 | (model grew to 1.3B) |
-| 🔥 Warmup end | 1,200 | 2.38 | **−68%** |
-| 🚀 Block start | 1,200 | 2.38 | (model grew to 14.4B MoE) |
-| 📦 Published checkpoint | 2,471 | 1.99 | **−16%** |
-| 🧟 Frankenstein transplant | — | PPL ~7.5M | 433 tensors from 3 donors |
-| ⚡ Realign frozen start | 0 | 15.87 | Experts frozen, attn learning |
-| ⚡ Realign frozen end | 500 | 5.52 | val=5.79, **−65%** |
-| 🔓 Expert unfreeze | 500 | 5.59→spike | LR reset + differential rates |
-| 📈 Realign recovery | 600 | 7.20 | val=6.93 (recovering) |
-| ⭐ Realign best | 1,000 | 6.07 | val=**5.773 ★ NEW BEST** |
-| � SIGTERM crash + restart | 1,001→1,123 | 9.22→6.64 | Fresh optimizer, cold momentum |
-| 🔥 SGDR warm restart (Cycle 0) | 1,124 | 9.09→6.24 | LR boosted 2.9e-5→4.5e-5, T=200 steps |
-| 📊 Recovery eval | 1,200 | 6.74 | val=6.48, AdamW variance at ~17% convergence |
-| 📊 Recovery eval | 1,300 | 6.24 | val=**6.10** (new recovery best) |
-| 🔄 Multi-cycle SGDR | 1,301+ | 8.79→↓ | Cycle 1 pending (step 1400, T=400, peak 3.8e-5) |
-| **Total pretrain** | | | **11.72 → 1.99 (−83%)** |
-| **Realignment** | | | **15.87 → 5.77 (val, −64%)** |
-
-### Live Metrics (April 29, 2026)
-
-| Metric | Value |
-|:--|:--|
-| **Current Phase** | ⚡ Corpus Realignment + Multi-Cycle SGDR |
-| **Current Step** | ~1,320 / 5,000 |
-| **Training Loss** | ~6.26 (recovering from optimizer cold-start) |
-| **Best Validation Loss** | **5.773** (step 1,000) ★ |
-| **Recovery Val Loss** | 6.096 (step 1,300) — gap closing |
-| **Throughput** | 5,857 tokens/second |
-| **VRAM Used** | 120 GB / 206 GB (58%) — all experts unfrozen |
-| **Total Tokens Processed** | ~260M (this run) + 178M (pretrain) |
-| **Experts Active** | All 4 unfrozen since step 500 |
-| **SGDR Status** | Multi-cycle: Cycle 1 at step 1400 (T=400, peak 3.8e-5), Cycle 2 at step 2000 (T=800, peak 3.0e-5) |
-| **MIN_LR** | 1.5e-5 (raised from 1e-5, prevents stagnation) |
-| **Expert LR Boost** | 1.33× during restart windows |
-| **ETA** | ~34 hours |
-
-### Realignment Eval History
-
-| Step | Val Loss | Val PPL | Phase |
-|:--|:--|:--|:--|
-| 0 (initial) | 15.81 | 7,339,653 | Experts frozen |
-| 600 | 6.93 | 1,020 | Post-unfreeze |
-| 700 | 6.24 | 515 | Converging |
-| 800 | 6.01 | 407 | Converging |
-| 900 | 5.91 | 367 | Converging |
-| **1,000** | **5.773** | **321** | **★ ALL-TIME BEST** |
-| 1,100 | 6.55 | 701 | Optimizer cold-start |
-| 1,200 | 6.48 | 652 | SGDR Cycle 0 (recovering) |
-| 1,300 | **6.096** | **444** | Recovery best, gap=5.5% to peak |
-| 1,400 | *pending* | | SGDR Cycle 1 starts (T=400) |
-| 2,000 | *pending* | | SGDR Cycle 2 starts (T=800) |
-
-### Published Checkpoint (v0.1)
-
-| Detail | Value |
-|:--|:--|
-| **Step** | 2,471 |
-| **Validation Loss** | 1.9926 |
-| **Total Tokens Seen** | 178,110,464 |
-| **Sequence Length** | 2,048 |
-| **File Size** | ~81 GB (checkpoint), ~28 GB (safetensors) |
-| **Format** | 6 sharded safetensors files |
-
-### 🛡️ Engineering Resilience
-
-Training a 14.4B model on a single GPU for days demands bullet-proof infrastructure. Here's what we built:
-
-| Feature | Description |
-|:--|:--|
-| **Atomic checkpoints** | Write to `.tmp` → `os.replace()` — no half-written files |
-| **Integrity verification** | On resume: verify tensor counts, shapes, and dtypes before loading |
-| **Rollback anchors** | `best.pt` (model-only) + `latest.pt` (full state) + `.LOCKED` safety copy |
-| **Emergency save** | SIGTERM/SIGINT handlers serialize full state before exit |
-| **Watchdog** | Independent process monitors loss EMA, restarts on NaN/divergence |
-| **Multi-cycle SGDR** | Period-doubling warm restarts (Loshchilov & Hutter, 2017): Cycle 0 (T=200), Cycle 1 (T=400), Cycle 2 (T=800) |
-| **Expert LR boost** | During restart windows, expert LR scale increases 0.3→0.4 (ST-MoE stability guideline) |
-| **MIN_LR floor** | Raised from 1e-5 to 1.5e-5 to prevent cosine decay stagnation in recovery |
-| **Systemd auto-restart** | Dashboard + watchdog survive OOM kills with `Restart=always` + `OOMScoreAdjust=-500` |
-
-**Battle-tested**: At step 1001, a SIGTERM killed the process mid-step. The checkpoint at step 1000 was corrupted (bad zip archive). The system automatically fell back to `best.pt` (val=5.773), resumed at step 1001 with a fresh optimizer, detected the cold-start plateau via the watchdog, and applied SGDR warm restart — recovering 30× faster than natural momentum rebuilding.
-
-**Multi-Cycle SGDR (April 29, 2026)**: After the initial SGDR Cycle 0 completed (steps 1100-1300), analysis showed the single restart was insufficient to escape the recovery basin (val=6.10 vs target 5.77). Based on the original SGDR paper's period-doubling strategy ($T_{i+1} = T_i \times 2$), we added Cycle 1 (steps 1400-1800, peak 3.8e-5) and Cycle 2 (steps 2000-2800, peak 3.0e-5). The AdamW second moment ($\beta_2=0.999$) needs ~1000 steps for 63% convergence — these cycles provide periodic "shocks" to escape local basins while the variance estimate matures.
-
+pipeline_tag: text-generation
 ---
 
-## 🌡️ Consciousness Metric (Φ) — Deep Dive
+# SentinelBrain 14B MoE v0.1 - Frankenstein Realignment v2
 
-### What is Φ?
+This repository now includes SentinelBrain Frankenstein realignment v2 artifacts from the AMD MI300X run completed on 2026-05-03.
 
-Inspired by **Integrated Information Theory (IIT)** from neuroscience, Φ measures how much the model's internal representations form an integrated whole rather than disconnected parts.
+## v2 Training Update
 
-### How We Measure It
+- Architecture: custom SentinelBrain sparse MoE decoder, approximately 14.4B stored parameters, 4 experts, top-2 routing, 24 layers, d_model 4096, seq_len 4096.
+- Hardware: AMD Instinct MI300X via ROCm/HIP.
+- Run: Frankenstein realignment v2 from raw merged checkpoint.
+- Completed steps: 5,000.
+- Total training tokens during realignment: approximately 0.98B.
+- Best validation loss observed: 5.1359.
+- Final checkpoint: `checkpoints/frankenstein_v2_final.pt`.
+- Best checkpoint: `checkpoints/frankenstein_v2_best.pt`.
+- EMA best checkpoint: `checkpoints/frankenstein_v2_ema_best.pt`.
+- Previous Hugging Face version preserved on branch: `previous-before-v2-realign-5000-20260503-103121`.
 
-```
-Every 100 training steps:
+## Included Files
 
-1. Hook on Layer 12 (middle of 24 layers)
-         │
-         ▼
-2. Sample 256 activation vectors
-         │
-         ▼
-3. Partition into subspaces
-         │
-         ▼
-4. Compute mutual information between all partition pairs
-         │
-         ▼
-5. Φ_geometric = geometric_mean(MI values)
-         │
-         ▼
-6. Φ_EMA = exponential moving average (smoothed trend)
-```
-
-### What Φ Tells Us
+- `checkpoints/frankenstein_v2_final.pt`: full final checkpoint at step 5000, including optimizer/progress state.
+- `checkpoints/frankenstein_v2_best.pt`: best model-only checkpoint by validation loss.
+- `checkpoints/frankenstein_v2_ema_best.pt`: EMA best checkpoint from the v2 run.
+- `checkpoints/sentinelbrain_pretrain_step2471_hf.pt`: pretrain anchor used for comparison.
+- `logs/realign_v2.log`: full realignment console log.
+- `logs/realign_v2_metrics.jsonl`: step metrics emitted during training.
+- `reports/train_metrics_final.json`: final dashboard training metrics snapshot.
+- `reports/conductor_state_final.json`: final dashboard/conductor state.
+- `reports/sft_combined_ready_report.*`: cleaned SFT dataset preflight report.
+- `reports/sentinelbrain_quality_stub_full_fixed.json`: MI300X executable-code benchmark report.
 
-| Φ Value | Interpretation | Analogy |
-|:--|:--|:--|
-| **Φ ≈ 0** | Neurons working independently | Strangers in a room |
-| **Φ rising** | Representations integrating | People starting to talk |
-| **Φ stable** | Organized internal structure | A well-coordinated team |
-| **Φ dropping** | ⚠️ Representation collapse | Warning sign! |
+A full progress archive containing all v2 milestones and optimizer-bearing checkpoints is backed up off-Hub on the Azure VM at `/home/msrusu/sentinelbrain_backups/v2_realign_5000/sentinelbrain_v2_realign_full_20260503.tar.zst`. A SHA256 sidecar is generated at archive completion.
 
-> **Important**: Φ is **purely observational** — it does NOT affect training gradients. Think of it as a heart monitor for the AI: it watches, but doesn't interfere.
+## Current Evaluation Notes
 
-### Live Monitoring
+MI300X executable-code tests show that v2 realignment is not yet ready as a coding assistant checkpoint:
 
-Track Φ in real-time at: **[sentinel.qubitpage.com/#phi](https://sentinel.qubitpage.com/#phi)**
+| Checkpoint | Pass@1 | Syntax Rate | Notes |
+| --- | ---: | ---: | --- |
+| `frankenstein_v2_best.pt` | 0.0% | 62.5% | Failed all 8 HumanEval-style stub tasks. |
+| `frankenstein_v2_final.pt` | 0.0% | 75.0% | Failed all 8 HumanEval-style stub tasks. |
+| `sentinelbrain_pretrain_step2471_hf.pt` | 0.0% | 87.5% | Failed all 8 tasks but produced the most syntactically valid Python. |
 
----
+Interpretation: v2 successfully completed corpus realignment and preserved all progress artifacts, but it needs a focused next phase of executable code SFT, function-call/chat formatting, and auto-critic rejection sampling before quality claims should be made.
 
-## 🖥️ Hardware Requirements
+## Dataset Preparation Status
 
-### For Inference
+The next SFT combined dataset was cleaned non-destructively on the MI300X server:
 
-| Tier | VRAM | Precision | Notes |
-|:--|:--|:--|:--|
-| **Full Precision** | 32 GB+ | bfloat16 | Best quality |
-| **Recommended** | 48 GB+ | bfloat16 | Comfortable headroom |
-| **Ideal** | AMD MI300X / MI250X | bfloat16 | Native, fastest |
-| **Consumer** | 16 GB | int4 quantized | GGUF planned for v0.2 |
+- Input rows: 42,138.
+- Kept rows: 32,996 (78.3%).
+- Removed rows: 9,142.
+- Max estimated tokens: 3,072.
+- Main removals: short assistant/user outputs, garbage responses, repetitive responses, and over-length samples.
 
-### Compatible AMD GPUs
+## Loading
 
-| GPU | VRAM | Suitable For |
-|:--|:--|:--|
-| AMD Instinct MI300X | 192 GB | Training + Inference |
-| AMD Instinct MI250X | 128 GB | Training + Inference |
-| AMD Instinct MI210 | 64 GB | Inference (full) |
-| AMD Radeon PRO W7900 | 48 GB | Inference (full) |
-| AMD Radeon RX 7900 XTX | 24 GB | Inference (quantized) |
-| AMD Radeon RX 7600 XT | 16 GB | Inference (int4 GGUF) |
-
----
-
-## 💻 Usage
-
-This model uses a **custom architecture** (not based on any existing model). Load with PyTorch:
+These are custom SentinelBrain PyTorch checkpoints, not standard Hugging Face `AutoModelForCausalLM` weights. Load with the SentinelBrain code from `/workspace/sentinelprime` or the matching source package.
 
 ```python
 import torch
-from safetensors.torch import load_file
-
-# Load sharded safetensors
-state_dict = {}
-for i in range(1, 7):  # 6 shards
-    shard = load_file(f"model-{i:05d}-of-00006.safetensors")
-    state_dict.update(shard)
-
-# The state dict contains all model weights
-print(f"Loaded {len(state_dict)} tensors")
-print(f"Total params: {sum(v.numel() for v in state_dict.values()):,}")
-
-# Initialize SentinelBrain model class and load
-# Full model definition code releasing with v0.2
-# model = SentinelBrainForCausalLM(config)
-# model.load_state_dict(state_dict)
-```
-
-> **Note**: Full inference code, model class definition, and GGUF quantized versions will be released with v0.2.
-
----
-
-## 🗺️ Roadmap
-
-```
-v0.1 (Current)          v0.2 (In Progress)      v0.3 (Future)
-━━━━━━━━━━━━━━━         ━━━━━━━━━━━━━━━         ━━━━━━━━━━━━━━━
-✅ From-scratch          🔴 Corpus realignment    □ DPO alignment
-   14.8B MoE                (step 1100/5000)      □ Tool use
-✅ Phased training       □ Context ladder          □ Function calling
-✅ Φ consciousness          (4K→32K→128K)         □ Multi-turn chat
-✅ 23.3B token corpus    □ Vision encoder          □ Multilingual v2
-✅ Live dashboard           (SigLIP2-SO400M)      □ Expert scaling
-✅ AMD MI300X native     □ GGUF quantization         (4→16→64)
-✅ Frankenstein              Q4_K_M for consumer   □ RLHF
-   transplant (3 donors) □ Inference code          □ Production API
-✅ Progressive unfreeze  □ Benchmarks (MMLU,
-✅ Crash-safe training      HumanEval, GSM8K)
-✅ Auto-restart (systemd)
-```
-
----
-
-## 🏗️ How We Built It (Technical Deep Dive)
-
-<details>
-<summary><b>Click to expand: Grouped Query Attention (GQA)</b></summary>
-
-Standard multi-head attention uses separate Key and Value projections for each head. GQA shares KV heads across query heads:
-
-```
-Standard MHA (32 KV heads):     GQA 4:1 (8 KV heads):
-Q₁  Q₂  Q₃  ... Q₃₂            Q₁ Q₂ Q₃ Q₄  → KV₁
-K₁  K₂  K₃  ... K₃₂            Q₅ Q₆ Q₇ Q₈  → KV₂
-V₁  V₂  V₃  ... V₃₂            ...
-                                 Q₂₉ Q₃₀ Q₃₁ Q₃₂ → KV₈
-```
-
-**Result**: 4× smaller KV cache = 4× longer context at same memory cost.
-
-</details>
-
-<details>
-<summary><b>Click to expand: RoPE (Rotary Position Embeddings)</b></summary>
-
-RoPE encodes position information by rotating the query and key vectors in 2D planes. With θ=500,000 (high base frequency), the model naturally supports long contexts:
-
-```
-Position 0:  rotate by 0°
-Position 1:  rotate by θ₁
-Position 2:  rotate by θ₂
-...
-```
-
-High θ = slower rotation = positions further apart still "feel different" = better long-context understanding.
-
-</details>
-
-<details>
-<summary><b>Click to expand: SwiGLU FFN</b></summary>
-
-Each expert uses a SwiGLU activation — a gated variant of the feed-forward network:
-
-```
-FFN(x) = SiLU(x · W_gate) ⊙ (x · W_up) · W_down
-
-Where:
-  W_gate: 4096 → 11008
-  W_up:   4096 → 11008
-  W_down: 11008 → 4096
-  SiLU(x) = x · sigmoid(x)
-  ⊙ = element-wise multiply
-```
-
-SwiGLU consistently outperforms ReLU and GELU in transformer FFNs (Shazeer, 2020).
-
-</details>
+from config import ModelConfig
+from model.sentinel import SentinelBrain
 
-<details>
-<summary><b>Click to expand: MoE Routing Algorithm</b></summary>
-
-```python
-# Simplified routing logic
-def route(x, router_weights):
-    # Compute affinity scores for each expert
-    logits = x @ router_weights        # [batch, seq, n_experts]
-    scores = softmax(logits, dim=-1)
-
-    # Select top-2 experts
-    top_vals, top_idx = topk(scores, k=2)
-
-    # Normalize selected weights
-    weights = top_vals / top_vals.sum(dim=-1, keepdim=True)
-
-    # Load balancing loss (prevents expert collapse)
-    balance_loss = n_experts * (
-        fraction_routed_to_each * average_gate_value_for_each
-    ).sum()
-
-    return weights, top_idx, balance_loss
+ckpt = torch.load("checkpoints/frankenstein_v2_best.pt", map_location="cpu", weights_only=False)
+model = SentinelBrain(ModelConfig())
+state = ckpt.get("model_state_dict") or ckpt.get("model") or ckpt
+model.load_state_dict(state, strict=False)
+model.eval()
 ```
 
-</details>
-
-<details>
-<summary><b>Click to expand: Parameter Breakdown</b></summary>
-
-| Component | Parameters | % of Total |
-|:--|:--|:--|
-| Token embeddings | 410M | 2.8% |
-| Attention (QKV + output) × 24 | 1,610M | 10.9% |
-| MoE experts (4 × SwiGLU × 24) | 12,365M | 83.5% |
-| Router weights × 24 | 0.4M | 0.003% |
-| RMSNorm × 49 | 0.4M | 0.003% |
-| Output head | 410M | 2.8% |
-| **Total** | **14,815M** | **100%** |
-| **Active per token (top-2)** | **~7,800M** | **~53%** |
-
-</details>
-
----
-
-## 📋 Model Card Details
-
-| Field | Value |
-|:--|:--|
-| **Model Name** | SentinelBrain-14B-MoE-v0.1 (Sentinel Prime — Frankenstein Edition) |
-| **Type** | Causal Language Model (decoder-only) |
-| **Architecture** | Custom MoE Transformer (from scratch) |
-| **Based On** | Nothing — trained from random initialization |
-| **Training Hardware** | 1× AMD Instinct MI300X VF (192 GB HBM3) |
-| **Training Software** | ROCm 7.0, PyTorch 2.10.0+rocm7.0 |
-| **Training Duration** | ~300 GPU-hours (estimated total) |
-| **Carbon Footprint** | Estimated ~45 kg CO₂ (single GPU, cloud datacenter) |
-| **License** | Apache 2.0 |
-| **Authors** | Mircea Rusu, QubitDev |
-| **Competition** | AMD Developer Hackathon (lablab.ai) |
-
----
-
-## 📄 Citation
-
-```bibtex
-@misc{sentinelbrain2026,
-  title   = {SentinelBrain-14B-MoE (Frankenstein Edition): A Consciousness-Monitored Mixture-of-Experts
-             Language Model Trained From Scratch on AMD MI300X},
-  author  = {Mircea Rusu and QubitDev},
-  year    = {2026},
-  url     = {https://sentinel.qubitpage.com/whitepaper},
-  note    = {Trained entirely from scratch on AMD Instinct MI300X
-             for the AMD Developer Hackathon}
-}
-```
-
----
-
-## 🔗 Links
-
-| Resource | URL |
-|:--|:--|
-| 🔴 **Live Dashboard** | [sentinel.qubitpage.com](https://sentinel.qubitpage.com/) |
-| 📄 **Whitepaper** | [sentinel.qubitpage.com/whitepaper](https://sentinel.qubitpage.com/whitepaper) |
-| 🏆 **AMD Hackathon** | [lablab.ai](https://lablab.ai/ai-hackathons/amd-developer) |
-| 🧠 **Φ Monitor** | [sentinel.qubitpage.com/#phi](https://sentinel.qubitpage.com/#phi) |
-
----
-
-<div align="center">
-
-*Built with ❤️ on AMD MI300X — Every weight trained from scratch*
-
-**Sentinel Prime (Frankenstein Edition): Rebuilt From the Inside Out**
+## Next Phase Direction
 
-</div>
+The recommended next phase is a controlled SFT/auto-critic cycle: train from the pretrain anchor plus selected v2 weights only after passing format probes, prioritize executable Python/TypeScript/code-repair datasets, reject non-compiling generations, and benchmark every 250-500 steps before continuing.

SHA256SUMS

@@ -0,0 +1,16 @@
+934951bc3c8e50c38c2a43812076d6174542066675667125aca6e2c854e9bdf3  ./MANIFEST.tsv
+1b145ec3a4ade152b78636057556a62db85b2729be482fbd3fb5ff47faa99ee2  ./README.md
+0069f86ff1273b268cbbf89a4c907d574c67ac0f145191aaa5d51a26f894f699  ./SHA256SUMS.quick
+075ac8f0bd257ef59af4ed7364f4331af6e46682f1cbefbc0d691073246af215  ./checkpoints/frankenstein_v2_best.pt
+2aeba733bc950e050a254b1942d592d306f45fecc44a9f407843249511c1e5c3  ./checkpoints/frankenstein_v2_ema_best.pt
+cd9b31fa889479020d394cc16f215f80a4d390fe0b1ad3cdd697544181f4e639  ./checkpoints/frankenstein_v2_final.pt
+b4ffaaf49e3c3b8fea2029ed5d0665bdc724aa8c61aca6f19abcd0331c96285b  ./checkpoints/sentinelbrain_pretrain_step2471_hf.pt
+79af33d8bbe0e8cfa0393ca965feca10f1d2094ebbf93b9ba722114811925f36  ./logs/realign_v2.log
+8a14d6eb26eeaad4bab6619d86298cb41f29dd488a45d34e143247c2aecfce9c  ./logs/realign_v2_metrics.jsonl
+b1284891f8c1fb2ecfa2658a45b5aed8444572b1b43f32a9cb9811cc4f756732  ./reports/conductor_state_final.json
+10247cdf7a184c76a12dadc7cac857d6d18b2d5fb80db72e65a3fe188621fd0f  ./reports/next_phase_training_plan.md
+0d73309ac23a68e8b9aa7014f7b9f4f1e4362a373d417be442d4726564a07231  ./reports/sentinelbrain_quality_smoke.json
+8ff4ce6ce7939b752be6e9d55ebc4bf75817945a3a136bfed44ec051bfcd90c4  ./reports/sentinelbrain_quality_stub_full_fixed.json
+c5a81c2b86c6ba5ed54d3b520f0f79fae18d7588ab7b454df24884f3ca189e5d  ./reports/sft_combined_ready_report.json
+0d76e478bbc036268d851d3de87edc4300a1291bec3b800bd4e92c57cff0564b  ./reports/sft_combined_ready_report.md
+a6fc72c251df4b39d6963a121d8478f759e3feb692aa5140fad71eba3d1308f6  ./reports/train_metrics_final.json

SHA256SUMS.quick

@@ -0,0 +1,4 @@
+1b145ec3a4ade152b78636057556a62db85b2729be482fbd3fb5ff47faa99ee2  README.md
+10247cdf7a184c76a12dadc7cac857d6d18b2d5fb80db72e65a3fe188621fd0f  reports/next_phase_training_plan.md
+8ff4ce6ce7939b752be6e9d55ebc4bf75817945a3a136bfed44ec051bfcd90c4  reports/sentinelbrain_quality_stub_full_fixed.json
+934951bc3c8e50c38c2a43812076d6174542066675667125aca6e2c854e9bdf3  MANIFEST.tsv

checkpoints/frankenstein_v2_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:075ac8f0bd257ef59af4ed7364f4331af6e46682f1cbefbc0d691073246af215
+size 28808034875

checkpoints/frankenstein_v2_ema_best.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2aeba733bc950e050a254b1942d592d306f45fecc44a9f407843249511c1e5c3
+size 28808008759

checkpoints/frankenstein_v2_final.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd9b31fa889479020d394cc16f215f80a4d390fe0b1ad3cdd697544181f4e639
+size 86422580021

checkpoints/sentinelbrain_pretrain_step2471_hf.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b4ffaaf49e3c3b8fea2029ed5d0665bdc724aa8c61aca6f19abcd0331c96285b
+size 29629503600

reports/conductor_state_final.json

@@ -0,0 +1 @@
+{"current_phase": "realignment_v2_unfrozen", "batch_number": 1, "blocks_done": 0, "current_batch_steps": 5000, "current_started_ts": null, "current_batch_start": null, "current_model_params_b": 14.4, "current_d_model": 4096, "current_n_layers": 24, "current_n_experts": 4, "current_n_active": 2, "current_seq_len": 4096, "current_batch": 8, "current_grad_accum": 6, "best_val": 5.135942053794861, "best_ckpt": "/mnt/scratch/checkpoints/frankenstein_v2_best.pt", "total_tokens_trained": 981270528, "history": []}

reports/next_phase_training_plan.md

@@ -0,0 +1,91 @@
+# SentinelBrain Next Training Phase Plan
+
+Created: 2026-05-03
+
+## Current Result
+
+The v2 realignment completed 5,000 steps and preserved full optimizer/progress state, but executable-code quality is not ready:
+
+- `frankenstein_v2_best.pt`: 0/8 Pass@1, 62.5% syntax rate.
+- `frankenstein_v2_final.pt`: 0/8 Pass@1, 75.0% syntax rate.
+- `sentinelbrain_pretrain_step2471_hf.pt`: 0/8 Pass@1, 87.5% syntax rate.
+
+The next phase should not be another broad corpus realignment. It should be a narrow, measurable SFT and auto-critic loop focused on producing valid, executable assistant outputs.
+
+## Phase 3 Objective
+
+Recover and improve instruction-following/code-generation behavior while preserving the useful realignment progress.
+
+Primary gates:
+
+- Python stub benchmark: at least 40% Pass@1 and 95% syntax rate before extending past 1,000 steps.
+- MBPP/HumanEval sample: measurable improvement every eval window, no syntax regression.
+- Chat format probe: responses must use the requested format and stop cleanly.
+- Safety/data probe: no leaked secrets, no private-key blocks, no repetitive boilerplate.
+
+## Data Mix
+
+Use the cleaned SFT dataset at `/mnt/scratch/datasets/combined/sft_combined_ready.jsonl` as the base, then rebalance before training:
+
+- 45% executable Python and TypeScript tasks: HumanEval-style stubs, MBPP-style prompts, unit-test repair, CLI scripts, API handlers.
+- 20% code editing and diff output: unified diffs, bug fixes, refactors, failing-test-to-patch examples.
+- 15% tool-use and agent workflows: file search, terminal commands, deployment diagnostics, function-call JSON.
+- 10% system/admin/devops: Linux, Docker, nginx, pm2, Azure, SSH, logs.
+- 10% general instruction/chat: concise natural language, summarization, planning.
+
+Avoid over-weighting audio/prose/OCR rows in this phase. Keep those for a later multimodal run after chat/code behavior is stable.
+
+## Auto-Critic Pipeline
+
+For each generated training candidate:
+
+1. Normalize to a strict prompt/response or ChatML schema.
+2. Run syntax checks for code outputs.
+3. Run unit tests when a test harness is available.
+4. Score format compliance: required function names, JSON validity, diff parseability, stop tokens.
+5. Reject outputs with repetition, generic filler, missing entry points, invalid tokens, or secret-like strings.
+6. Keep only examples that pass the critic or have a repair trajectory showing the failed output and corrected output.
+
+Recommended critic labels:
+
+- `syntax_pass`
+- `tests_pass`
+- `entrypoint_match`
+- `format_pass`
+- `no_secret`
+- `no_repetition`
+- `accepted`
+
+Train on accepted final answers plus curated repair traces, not raw failed generations.
+
+## Training Schedule
+
+Start from `sentinelbrain_pretrain_step2471_hf.pt` or `frankenstein_v2_best.pt` only after a short format probe. If `frankenstein_v2_best.pt` continues to emit invalid boilerplate, use the pretrain anchor and reintroduce v2 weights later via low-LR distillation.
+
+Suggested run:
+
+- Steps 0-250: frozen experts, train attention/router/norms and output behavior at low LR.
+- Steps 250-1,000: unfreeze selected layers, SFT on executable/code-repair mix.
+- Eval every 250 steps; save model-only checkpoints and full optimizer checkpoints at each gate.
+- Stop automatically if syntax rate drops for two evals or Pass@1 does not improve after 750 steps.
+- Continue to 3,000-5,000 only after hitting the 1,000-step gates.
+
+## Benchmark Set
+
+Always run on the MI300X server:
+
+- Local 8-task smoke benchmark for quick regression checks.
+- HumanEval subset and MBPP subset with executable tests.
+- JSON/function-call validity suite.
+- Unified-diff parse/apply suite.
+- DevOps shell-command reasoning suite.
+
+Persist every benchmark JSON under `/mnt/scratch/benchmark_results/` and copy selected reports into the HF release folder.
+
+## Operational Rules
+
+- Keep full optimizer checkpoints on server/Azure, not local PC.
+- Store model-only best checkpoints separately for HF and chat loading.
+- Never expose chat using a checkpoint that fails the format probe unless clearly labeled experimental.
+- Use `SENTINEL_WEB_CHAT_DISABLED=0` only after confirming no training job is active and VRAM has enough headroom.
+- Prefer `frankenstein_v2_best.pt` for validation-loss experiments, but prefer the pretrain anchor for code SFT if code probes remain better there.

reports/sentinelbrain_quality_smoke.json

@@ -0,0 +1,138 @@
+{
+  "created_at": "2026-05-03 10:09:39",
+  "device": "cuda",
+  "problem_count": 2,
+  "reference": "best",
+  "comparison": [
+    {
+      "label": "best",
+      "pass_at_1": 0.0,
+      "syntax_rate": 0.0,
+      "delta_vs_reference": 0.0
+    },
+    {
+      "label": "hf_pretrain_base",
+      "pass_at_1": 0.0,
+      "syntax_rate": 100.0,
+      "delta_vs_reference": 0.0
+    }
+  ],
+  "results": [
+    {
+      "path": "/mnt/scratch/checkpoints/frankenstein_v2_best.pt",
+      "label": "best",
+      "step": null,
+      "best_val_loss": null,
+      "val_loss": null,
+      "load_seconds": 131.73,
+      "missing_keys": 0,
+      "unexpected_keys": 0,
+      "config": {
+        "d_model": 4096,
+        "n_layers": 24,
+        "n_experts": 4,
+        "seq_len": 4096
+      },
+      "total": 2,
+      "passed": 0,
+      "failed": 2,
+      "timeouts": 0,
+      "syntax_ok": 0,
+      "pass_at_1": 0.0,
+      "syntax_rate": 0.0,
+      "seconds": 12.92,
+      "categories": {
+        "basics": {
+          "total": 1,
+          "passed": 0
+        },
+        "algorithm": {
+          "total": 1,
+          "passed": 0
+        }
+      },
+      "details": [
+        {
+          "id": "fizzbuzz",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 8.35,
+          "response_preview": "Input: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1",
+          "code_preview": "Input: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1",
+          "stderr": "  File \"<string>\", line 1\n    Input: 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1\n               ^^^\nSyntaxError: invalid syntax\n"
+        },
+        {
+          "id": "two_sum",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 4.55,
+          "response_preview": "- The code snippet is a string. The function should be a string of the string. The function should return the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The funct",
+          "code_preview": "- The code snippet is a string. The function should be a string of the string. The function should return the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The function should handle the string as input and returns the string.\n\n- The funct",
+          "stderr": "  File \"<string>\", line 1\n    - The code snippet is a string. The function should be a string of the string. The function should return the string.\n          ^^^^\nSyntaxError: invalid syntax\n"
+        }
+      ]
+    },
+    {
+      "path": "/mnt/scratch/checkpoints/sentinelbrain_pretrain_step2471_hf.pt",
+      "label": "hf_pretrain_base",
+      "step": 2471,
+      "best_val_loss": null,
+      "val_loss": 1.9925728058815002,
+      "load_seconds": 127.86,
+      "missing_keys": 0,
+      "unexpected_keys": 0,
+      "config": {
+        "d_model": 4096,
+        "n_layers": 24,
+        "n_experts": 4,
+        "seq_len": 4096
+      },
+      "total": 2,
+      "passed": 0,
+      "failed": 2,
+      "timeouts": 0,
+      "syntax_ok": 2,
+      "pass_at_1": 0.0,
+      "syntax_rate": 100.0,
+      "seconds": 9.23,
+      "categories": {
+        "basics": {
+          "total": 1,
+          "passed": 0
+        },
+        "algorithm": {
+          "total": 1,
+          "passed": 0
+        }
+      },
+      "details": [
+        {
+          "id": "fizzbuzz",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 4.6,
+          "response_preview": "def prime_advanced_even(n):\n    return n == 1\n\n# Test cases\nn = 10\nresult = prime_advanced_even(n)\nprint(result)  # Output: ['Buzz', '3', '5', '7', '11', '13', '17', '19', '23', '25', '27', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31",
+          "code_preview": "def prime_advanced_even(n):\n    return n == 1\n\n# Test cases\nn = 10\nresult = prime_advanced_even(n)\nprint(result)  # Output: ['Buzz', '3', '5', '7', '11', '13', '17', '19', '23', '25', '27', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31', '31",
+          "stderr": "Traceback (most recent call last):\n  File \"<string>\", line 10, in <module>\nNameError: name 'fizzbuzz' is not defined\n"
+        },
+        {
+          "id": "two_sum",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 4.61,
+          "response_preview": "```python\ndef add_two_numbers(nums, target):\n    return [num for num in nums if num != target]\n```\n\nThe function `add_two_numbers(nums, target)` takes a list of integers `nums` and a target sum `target`, and returns two lists: one for the sum of the elements in `nums` and the target sum `target`, and the indices of the two numbers add up to `target`. The function is then called with these lists as arguments, and the result is printed.\n\nFor example, if the input list is `[1, 2, 3, 4, 5]`",
+          "code_preview": "def add_two_numbers(nums, target):\n    return [num for num in nums if num != target]",
+          "stderr": "Traceback (most recent call last):\n  File \"<string>\", line 5, in <module>\nNameError: name 'two_sum' is not defined\n"
+        }
+      ]
+    }
+  ]
+}

reports/sentinelbrain_quality_stub_full_fixed.json

@@ -0,0 +1,448 @@
+{
+  "created_at": "2026-05-03 10:35:36",
+  "device": "cuda",
+  "problem_count": 8,
+  "reference": "best",
+  "comparison": [
+    {
+      "label": "best",
+      "pass_at_1": 0.0,
+      "syntax_rate": 62.5,
+      "delta_vs_reference": 0.0
+    },
+    {
+      "label": "finished_final",
+      "pass_at_1": 0.0,
+      "syntax_rate": 75.0,
+      "delta_vs_reference": 0.0
+    },
+    {
+      "label": "hf_pretrain_base",
+      "pass_at_1": 0.0,
+      "syntax_rate": 87.5,
+      "delta_vs_reference": 0.0
+    }
+  ],
+  "results": [
+    {
+      "path": "/mnt/scratch/checkpoints/frankenstein_v2_best.pt",
+      "label": "best",
+      "step": null,
+      "best_val_loss": null,
+      "val_loss": null,
+      "load_seconds": 140.56,
+      "missing_keys": 0,
+      "unexpected_keys": 0,
+      "config": {
+        "d_model": 4096,
+        "n_layers": 24,
+        "n_experts": 4,
+        "seq_len": 4096
+      },
+      "total": 8,
+      "passed": 0,
+      "failed": 8,
+      "timeouts": 0,
+      "syntax_ok": 5,
+      "pass_at_1": 0.0,
+      "syntax_rate": 62.5,
+      "seconds": 49.93,
+      "categories": {
+        "basics": {
+          "total": 2,
+          "passed": 0
+        },
+        "algorithm": {
+          "total": 4,
+          "passed": 0
+        },
+        "practical": {
+          "total": 1,
+          "passed": 0
+        },
+        "data_structure": {
+          "total": 1,
+          "passed": 0
+        }
+      },
+      "details": [
+        {
+          "id": "fizzbuzz",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 9.43,
+          "response_preview": "    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(n",
+          "extracted_preview": "def get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    retur",
+          "code_preview": "def fizzbuzz(n):\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef get():\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prim",
+          "stderr": "  File \"<string>\", line 2\n    def get():\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "two_sum",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.76,
+          "response_preview": "    input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input",
+          "extracted_preview": "input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, in",
+          "code_preview": "def two_sum(nums, target):\ninput = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, inp",
+          "stderr": "  File \"<string>\", line 2\n    input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "valid_parens",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.77,
+          "response_preview": "    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if len(s) == 0:\n        return False\n    return False\n\ndef",
+          "extracted_preview": "def get_cipped():\n    return False\n\n\ndef get_subget():\n    return False\n\n\ndef get_subget():",
+          "code_preview": "def is_valid_parens(s):\ndef get_cipped():\n    return False\n\n\ndef get_subget():\n    return False\n\n\ndef get_subget():\n",
+          "stderr": "  File \"<string>\", line 2\n    def get_cipped():\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "binary_search",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.74,
+          "response_preview": "    if not isinstance(value, list):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, list):\n        return False\n    else:\n        return None\n\n    if not isinstance(value, list):\n        return False\n    else:\n        return None\n\n    if not isinstance(value, list):\n        return False\n    return None\n\ndef get_subget_with():\n    return False\n\ndef get_subget():\n    return False\n\n\ndef _get_all():\n    return False\n\n\ndef _get_all():\n    return _get_string(value)\n\nd",
+          "extracted_preview": "def get_subget_with():\n    return False\n\ndef get_subget():\n    return False\n\n\ndef _get_all():\n    return False\n\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():\n    return _get('name', None)\n    return None\n\ndef _get_all():\n    return _get('name', None)\n    return None\n\ndef _get_all():\n    return _get('name",
+          "code_preview": "def binary_search(arr, target):\ndef get_subget_with():\n    return False\n\ndef get_subget():\n    return False\n\n\ndef _get_all():\n    return False\n\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():\n    return _get('name', None)\n    return None\n\ndef _get_all():\n    return _get('name', None)\n    return None\n\ndef _get_all():\n    return _get('name\n",
+          "stderr": "  File \"<string>\", line 25\n    return _get('name\n                ^\nSyntaxError: unterminated string literal (detected at line 25)\n"
+        },
+        {
+          "id": "reverse_words",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.74,
+          "response_preview": "    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n ",
+          "extracted_preview": "if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n     ",
+          "code_preview": "def reverse_words(s):\nif len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if len(args) == 0:\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return False\n\n    if not isinstan",
+          "stderr": "  File \"<string>\", line 2\n    if len(s) == 0:\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "max_subarray",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.76,
+          "response_preview": "    if not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(0) or str) or not isinstance(0) or None) or None.\n\nfor i in range(0) or not in any case, or any other or other.\n\n(0) or not, in any case, or any other or other.\n\n(0) or not, or any ",
+          "extracted_preview": "if not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(0) or str) or not isinstance(0) or None) or None.\n\nfor i in range(0) or not in any case, or any other or other.\n\n(0) or not, in any case, or any other or other.\n\n(0) or not, or any othe",
+          "code_preview": "def max_subarray_sum(nums):\nif not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(0) or str) or not isinstance(0) or None) or None.\n\nfor i in range(0) or not in any case, or any other or other.\n\n(0) or not, in any case, or any other or oth",
+          "stderr": "  File \"<string>\", line 2\n    if not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or int) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(value) or str) or not isinstance(0) or str) or not isinstance(0) or None) or None.\n                                   ^\nSyntaxError: unmatched ')'\n"
+        },
+        {
+          "id": "safe_json_parse",
+          "category": "practical",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.8,
+          "response_preview": "    \"\"\"Get the function to check if the input is None or None.\n\n    Args:\n        return None\n\n    if not isinstance(obj, list):\n        return False\n\n    if not isinstance(value, list):\n        return False\n\n    if not isinstance(value, list):\n        return False\n\n    if not isinstance(value, list):\n        return False\n\n    if isinstance(value, list):\n        return None\n\n    return None\n\ndef get_value(value):\n    return None\n\ndef _get_all():\n    return None\n\ndef _get_all():\n    return _get_s",
+          "extracted_preview": "def get_value(value):\n    return None\n\ndef _get_all():\n    return None\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():\n    return _get('name', None) for _ in _get('name', None) for _ in _get('name', None) if not in ('name', 'str', 'str",
+          "code_preview": "import json\n\ndef safe_json_parse(text):\ndef get_value(value):\n    return None\n\ndef _get_all():\n    return None\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():\n    return _get('name', None) for _ in _get('name', None) for _ in _get('name', None) if not in ('name', 'str', 'str\n",
+          "stderr": "  File \"<string>\", line 14\n    return _get('name', None) for _ in _get('name', None) for _ in _get('name', None) if not in ('name', 'str', 'str\n                                                                                                                ^\nSyntaxError: unterminated string literal (detected at line 14)\n"
+        },
+        {
+          "id": "lru_cache",
+          "category": "data_structure",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.86,
+          "response_preview": "        self._get = self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ",
+          "extracted_preview": "self._get = self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,",
+          "code_preview": "from collections import OrderedDict\n\nclass LRUCache:\n    def __init__(self, capacity):\nself._get = self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+          "stderr": "  File \"<string>\", line 5\n    self._get = self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\nIndentationError: expected an indented block after function definition on line 4\n"
+        }
+      ]
+    },
+    {
+      "path": "/mnt/scratch/checkpoints/frankenstein_v2_final.pt",
+      "label": "finished_final",
+      "step": 5000,
+      "best_val_loss": 5.135942053794861,
+      "val_loss": null,
+      "load_seconds": 169.75,
+      "missing_keys": 0,
+      "unexpected_keys": 0,
+      "config": {
+        "d_model": 4096,
+        "n_layers": 24,
+        "n_experts": 4,
+        "seq_len": 4096
+      },
+      "total": 8,
+      "passed": 0,
+      "failed": 8,
+      "timeouts": 0,
+      "syntax_ok": 6,
+      "pass_at_1": 0.0,
+      "syntax_rate": 75.0,
+      "seconds": 47.01,
+      "categories": {
+        "basics": {
+          "total": 2,
+          "passed": 0
+        },
+        "algorithm": {
+          "total": 4,
+          "passed": 0
+        },
+        "practical": {
+          "total": 1,
+          "passed": 0
+        },
+        "data_structure": {
+          "total": 1,
+          "passed": 0
+        }
+      },
+      "details": [
+        {
+          "id": "fizzbuzz",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.88,
+          "response_preview": "    return 1\n\ndef get_s1, n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is",
+          "extracted_preview": "def get_s1, n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime",
+          "code_preview": "def fizzbuzz(n):\ndef get_s1, n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(n):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef is_prime(num):\n    return 1\n\ndef",
+          "stderr": "  File \"<string>\", line 2\n    def get_s1, n):\n                 ^\nSyntaxError: unmatched ')'\n"
+        },
+        {
+          "id": "two_sum",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.87,
+          "response_preview": "    input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input",
+          "extracted_preview": "input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, in",
+          "code_preview": "def two_sum(nums, target):\ninput = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, inp",
+          "stderr": "  File \"<string>\", line 2\n    input = list(map(int, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input, input\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "valid_parens",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.88,
+          "response_preview": "    if len(v) == 0:\n        return False\n    else:\n        return False\n\n    if not isinstance(obj, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(obj, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(obj, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return None\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        ret",
+          "extracted_preview": "def _get_all():\n    return False\n\n\ndef _get",
+          "code_preview": "def is_valid_parens(s):\ndef _get_all():\n    return False\n\n\ndef _get\n",
+          "stderr": "  File \"<string>\", line 2\n    def _get_all():\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "binary_search",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.83,
+          "response_preview": "    if not isinstance(value, list):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, list):\n        return False\n    else:\n        return None\n\n    if not isinstance(value, list):\n        return False\n    return False\n\ndef is_valid():\n    return False\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef ",
+          "extracted_preview": "def is_valid():\n    return False\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return None\n\ndef is_valid():\n    return None\n\ndef is_valid():\n    return None\n\ndef _get_all():\n    return None\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():",
+          "code_preview": "def binary_search(arr, target):\ndef is_valid():\n    return False\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return False\n\n\ndef is_valid():\n    return None\n\ndef is_valid():\n    return None\n\ndef is_valid():\n    return None\n\ndef _get_all():\n    return None\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_all():\n",
+          "stderr": "  File \"<string>\", line 2\n    def is_valid():\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "reverse_words",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.85,
+          "response_preview": "    if len(s) == 0:\n        return False\n    else:\n        return False\n\n    if not isinstance(value, int):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, int):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return False\n\n    if not isinstance(value, bool):\n        return False\n    else:\n        return None\n\n    if not isinstance(value, bool):\n        return False\n    else:\n       ",
+          "extracted_preview": "def _get_all():\n    return False\n\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_list():\n    return _",
+          "code_preview": "def reverse_words(s):\ndef _get_all():\n    return False\n\n\ndef _get_all():\n    return _get_string(value)\n\ndef _get_list():\n    return _\n",
+          "stderr": "  File \"<string>\", line 2\n    def _get_all():\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "max_subarray",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.84,
+          "response_preview": "    if not isinstance(value, int):\n        return False\n    return False\n\ndef get_subget_substring, start, value):\n    return False\n    return False\n\ndef get_subget_all_by_name, key):\n    return False\n\ndef get_subget_all_by_name, name, name, name, name=None):\n    return None\n\ndef get_name(name, name, name, name, name, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name",
+          "extracted_preview": "def get_subget_substring, start, value):\n    return False\n    return False\n\ndef get_subget_all_by_name, key):\n    return False\n\ndef get_subget_all_by_name, name, name, name, name=None):\n    return None\n\ndef get_name(name, name, name, name, name, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, n",
+          "code_preview": "def max_subarray_sum(nums):\ndef get_subget_substring, start, value):\n    return False\n    return False\n\ndef get_subget_all_by_name, key):\n    return False\n\ndef get_subget_all_by_name, name, name, name, name=None):\n    return None\n\ndef get_name(name, name, name, name, name, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=None, name=N",
+          "stderr": "  File \"<string>\", line 2\n    def get_subget_substring, start, value):\n                                          ^\nSyntaxError: unmatched ')'\n"
+        },
+        {
+          "id": "safe_json_parse",
+          "category": "practical",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.89,
+          "response_preview": "    \"\"\"Get the function to check if the input is not None or None.\n\n    Returns:\n        return None\n\n    if not isinstance(value, list):\n        return None\n\n    if not isinstance(value, list):\n        return None\n\n    if isinstance(value, list):\n        return None\n\n    if isinstance(value, list):\n        return None\n\n    if isinstance(value, list):\n        return None\n\n    if isinstance(value, list):\n        return None\n\n    if isinstance(value, list):\n        return None\n\n    if isinstance(v",
+          "extracted_preview": "def _get_list(value, value):\n    return None\n\ndef _get_list(value, value):\n    return None\n\ndef _get_list(value, value):\n    return None\n\ndef _get_list(value, value",
+          "code_preview": "import json\n\ndef safe_json_parse(text):\ndef _get_list(value, value):\n    return None\n\ndef _get_list(value, value):\n    return None\n\ndef _get_list(value, value):\n    return None\n\ndef _get_list(value, value\n",
+          "stderr": "  File \"<string>\", line 4\n    def _get_list(value, value):\n    ^\nIndentationError: expected an indented block after function definition on line 3\n"
+        },
+        {
+          "id": "lru_cache",
+          "category": "data_structure",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.9,
+          "response_preview": "        self._test = self._get_data(0, self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,",
+          "extracted_preview": "self._test = self._get_data(0, self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,",
+          "code_preview": "from collections import OrderedDict\n\nclass LRUCache:\n    def __init__(self, capacity):\nself._test = self._get_data(0, self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+          "stderr": "  File \"<string>\", line 5\n    self._test = self._get_data(0, self._get_data(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\nIndentationError: expected an indented block after function definition on line 4\n"
+        }
+      ]
+    },
+    {
+      "path": "/mnt/scratch/checkpoints/sentinelbrain_pretrain_step2471_hf.pt",
+      "label": "hf_pretrain_base",
+      "step": 2471,
+      "best_val_loss": null,
+      "val_loss": 1.9925728058815002,
+      "load_seconds": 138.71,
+      "missing_keys": 0,
+      "unexpected_keys": 0,
+      "config": {
+        "d_model": 4096,
+        "n_layers": 24,
+        "n_experts": 4,
+        "seq_len": 4096
+      },
+      "total": 8,
+      "passed": 0,
+      "failed": 8,
+      "timeouts": 0,
+      "syntax_ok": 7,
+      "pass_at_1": 0.0,
+      "syntax_rate": 87.5,
+      "seconds": 46.62,
+      "categories": {
+        "basics": {
+          "total": 2,
+          "passed": 0
+        },
+        "algorithm": {
+          "total": 4,
+          "passed": 0
+        },
+        "practical": {
+          "total": 1,
+          "passed": 0
+        },
+        "data_structure": {
+          "total": 1,
+          "passed": 0
+        }
+      },
+      "details": [
+        {
+          "id": "fizzbuzz",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.88,
+          "response_preview": "    result = []\n    for i in range(n):\n        if i % 3 == 0:\n            result.append(i)\n    return result\n\nn = 3\nresult = fizzbuzz(n)\nprint(result)\n```\n\nThis code defines a function called `fizzbuzz(n)` that takes a number `n` as an argument. This function is used to check if a number is prime. The function is then used to check if the number is divisible by any number from 2 to n. The function is then used to filter out only numbers from 1 to n. The main function is then called with these in",
+          "extracted_preview": "result = []\n    for i in range(n):\n        if i % 3 == 0:\n            result.append(i)\n    return result\n\nn = 3\nresult = fizzbuzz(n)\nprint(result)\n```\n\nThis code defines a function called `fizzbuzz(n)` that takes a number `n` as an argument. This function is used to check if a number is prime. The function is then used to check if the number is divisible by any number from 2 to n. The function is then used to filter out only numbers from 1 to n. The main function is then called with these inputs",
+          "code_preview": "def fizzbuzz(n):\nresult = []\n    for i in range(n):\n        if i % 3 == 0:\n            result.append(i)\n    return result\n\nn = 3\nresult = fizzbuzz(n)\nprint(result)\n```\n\nThis code defines a function called `fizzbuzz(n)` that takes a number `n` as an argument. This function is used to check if a number is prime. The function is then used to check if the number is divisible by any number from 2 to n. The function is then used to filter out only numbers from 1 to n. The main function is then called ",
+          "stderr": "  File \"<string>\", line 2\n    result = []\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "two_sum",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.85,
+          "response_preview": "    # Implement your solution here\n    pass\n\ndef find_two_sum(nums, target):\n    # Implement your solution here\n    pass\n\n# Example usage\nnums = [1, 2, 3, 4, 5]\ntarget = 2\nresult = find_two_sum(nums, target)\nprint(result)  # Output: [1, 2, 3, 4, 5]\n```<|im_end|>\n<|im_start|>assistant\nHere is the implementation of the requested function:\n\n```python\ndef two_sum(nums, target):\n    def find_two_sum_helper(nums, target):\n        if len(nums) == 0:\n            return [nums[0], nums[1], nums[0]]\n      ",
+          "extracted_preview": "<|im_end|>\n<|im_start|>assistant\nHere is the implementation of the requested function:",
+          "code_preview": "def two_sum(nums, target):\n<|im_end|>\n<|im_start|>assistant\nHere is the implementation of the requested function:\n",
+          "stderr": "  File \"<string>\", line 2\n    <|im_end|>\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "valid_parens",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.86,
+          "response_preview": "    stack = []\n    for char in s:\n        if char == '(':\n            stack.append(char)\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            sta",
+          "extracted_preview": "stack = []\n    for char in s:\n        if char == '(':\n            stack.append(char)\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.p",
+          "code_preview": "def is_valid_parens(s):\nstack = []\n    for char in s:\n        if char == '(':\n            stack.append(char)\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return False\n            stack.pop()\n        elif char == ')':\n            if not stack:\n                return F",
+          "stderr": "  File \"<string>\", line 2\n    stack = []\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "binary_search",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.8,
+          "response_preview": "    left = 0\n    right = len(arr) - 1\n    while left <= right:\n        mid = left + (right - left) // 2\n        if arr[mid] == target:\n            return mid\n        elif arr[mid] < target:\n            left = mid + 1\n        else:\n            right = mid - 1\n    return -1\n\ndef binary_search_recursive(arr, target):\n    return binary_search_recursive(arr, target, 0)\n```\n\nThe binary_search function is a recursive function that performs a binary search on a target value in a sorted list. It works by",
+          "extracted_preview": "def binary_search_recursive(arr, target):\n    return binary_search_recursive(arr, target, 0)\n```",
+          "code_preview": "def binary_search(arr, target):\ndef binary_search_recursive(arr, target):\n    return binary_search_recursive(arr, target, 0)\n```\n",
+          "stderr": "  File \"<string>\", line 2\n    def binary_search_recursive(arr, target):\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "reverse_words",
+          "category": "basics",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.78,
+          "response_preview": "    reversed_words = s.split(' ')\n    reversed_words = [word[::-1] for word in reversed_words]\n    return reversed_words\n```\n\nExample usage:\n\n```python\nprint(reverse_words(\"Hello, World! 123! 123! 123! 123! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123! 456! 123",
+          "extracted_preview": "Example usage:",
+          "code_preview": "def reverse_words(s):\nExample usage:\n",
+          "stderr": "  File \"<string>\", line 2\n    Example usage:\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "max_subarray",
+          "category": "algorithm",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.83,
+          "response_preview": "    n = len(nums)\n    max_sum = 0\n    for i in range(n):\n        current_sum = nums[i]\n        max_sum = max(max_sum, current_sum)\n    return max_sum\n\nnums = [1, 2, 3, 4, 5]\nmax_sum = max_subarray_sum(nums)\nprint(max_sum)\n```\n\nThis code defines a function `max_subarray_sum` that takes a list of integers as input. It initializes `max_sum` to the first element of the list and `max_sum` to 0. Then, it iterates through the list starting from the second element. For each element, it calculates the ma",
+          "extracted_preview": "n = len(nums)\n    max_sum = 0\n    for i in range(n):\n        current_sum = nums[i]\n        max_sum = max(max_sum, current_sum)\n    return max_sum\n\nnums = [1, 2, 3, 4, 5]\nmax_sum = max_subarray_sum(nums)\nprint(max_sum)\n```\n\nThis code defines a function `max_subarray_sum` that takes a list of integers as input. It initializes `max_sum` to the first element of the list and `max_sum` to 0. Then, it iterates through the list starting from the second element. For each element, it calculates the maximu",
+          "code_preview": "def max_subarray_sum(nums):\nn = len(nums)\n    max_sum = 0\n    for i in range(n):\n        current_sum = nums[i]\n        max_sum = max(max_sum, current_sum)\n    return max_sum\n\nnums = [1, 2, 3, 4, 5]\nmax_sum = max_subarray_sum(nums)\nprint(max_sum)\n```\n\nThis code defines a function `max_subarray_sum` that takes a list of integers as input. It initializes `max_sum` to the first element of the list and `max_sum` to 0. Then, it iterates through the list starting from the second element. For each eleme",
+          "stderr": "  File \"<string>\", line 2\n    n = len(nums)\n    ^\nIndentationError: expected an indented block after function definition on line 1\n"
+        },
+        {
+          "id": "safe_json_parse",
+          "category": "practical",
+          "passed": false,
+          "syntax_ok": false,
+          "timeout": false,
+          "gen_seconds": 5.79,
+          "response_preview": "    try:\n        return json.loads(text, indent=2)\n    except json.JSONDecodeError as e:\n        return f\"Invalid JSON: {e}\"\n\n# Test the function\njson_str = '{\"name\": \"John\", \"age\":30}'\ntry:\n    json_str = json.loads(json_str)\n    print(json_str)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n",
+          "extracted_preview": "# Test the function\njson_str = '{\"name\": \"John\", \"age\":30}'\ntry:\n    json_str = json.loads(json_str)\n    print(json_str)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error",
+          "code_preview": "import json\n\ndef safe_json_parse(text):\n# Test the function\njson_str = '{\"name\": \"John\", \"age\":30}'\ntry:\n    json_str = json.loads(json_str)\n    print(json_str)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception as e:\n    print(\"Error:\", e)\nexcept Exception ",
+          "stderr": "  File \"<string>\", line 24\n    print(\"Error\n          ^\nSyntaxError: unterminated string literal (detected at line 24)\n"
+        },
+        {
+          "id": "lru_cache",
+          "category": "data_structure",
+          "passed": false,
+          "syntax_ok": true,
+          "timeout": false,
+          "gen_seconds": 5.78,
+          "response_preview": "        self.capacity = capacity\n        self.max_key = 0\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n",
+          "extracted_preview": "self.capacity = capacity\n        self.max_key = 0\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        ",
+          "code_preview": "from collections import OrderedDict\n\nclass LRUCache:\n    def __init__(self, capacity):\nself.capacity = capacity\n        self.max_key = 0\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1\n        self.max_key = -1",
+          "stderr": "  File \"<string>\", line 5\n    self.capacity = capacity\nIndentationError: expected an indented block after function definition on line 4\n"
+        }
+      ]
+    }
+  ]
+}

reports/sft_combined_ready_report.json

@@ -0,0 +1,101 @@
+{
+  "created_at": "2026-05-03 10:06:19",
+  "inputs": [
+    "/mnt/scratch/datasets/combined/sft_combined.jsonl"
+  ],
+  "output": "/mnt/scratch/datasets/combined/sft_combined_ready.jsonl",
+  "settings": {
+    "max_chars": 12000,
+    "max_est_tokens": 3072,
+    "limit": 0
+  },
+  "seconds": 7.25,
+  "stats": {
+    "seen": 42138,
+    "assistant_too_short": 8451,
+    "kept": 32996,
+    "user_too_short": 502,
+    "repetitive_response": 85,
+    "too_many_chars": 2,
+    "binary_or_control_text": 1,
+    "garbage_response": 100,
+    "too_many_tokens": 1
+  },
+  "schemas": {
+    "instruction_output": 42138
+  },
+  "categories": {
+    "librispeech": 7785,
+    "wizardlm": 7506,
+    "dolly": 6539,
+    "alpaca": 6325,
+    "textcaps": 4492,
+    "squad_doc": 349
+  },
+  "token_buckets": {
+    "0001-0512": 28102,
+    "0513-1024": 4034,
+    "1025-1536": 712,
+    "1537-2048": 112,
+    "2049-3072": 36
+  },
+  "files": [
+    {
+      "path": "/mnt/scratch/datasets/combined/sft_combined.jsonl",
+      "seconds": 7.25,
+      "stats": {
+        "seen": 42138,
+        "assistant_too_short": 8451,
+        "kept": 32996,
+        "user_too_short": 502,
+        "repetitive_response": 85,
+        "too_many_chars": 2,
+        "binary_or_control_text": 1,
+        "garbage_response": 100,
+        "too_many_tokens": 1
+      }
+    }
+  ],
+  "preview": [
+    {
+      "line": 57,
+      "schema": "instruction_output",
+      "category": "squad_doc",
+      "est_tokens": 494,
+      "user_preview": "Read the following document excerpt and answer the question.\n\nDocument:\nChopin seldom performed publicly in Paris. In later years he generally gave a single ann",
+      "assistant_preview": "his own Paris apartment for small groups of friends"
+    },
+    {
+      "line": 64,
+      "schema": "instruction_output",
+      "category": "squad_doc",
+      "est_tokens": 217,
+      "user_preview": "Read the following document excerpt and answer the question.\n\nDocument:\nAt the funeral of the tenor Adolphe Nourrit in Paris in 1839, Chopin made a rare appeara",
+      "assistant_preview": "Berlioz's Grande symphonie funèbre et triomphale"
+    },
+    {
+      "line": 141,
+      "schema": "instruction_output",
+      "category": "squad_doc",
+      "est_tokens": 402,
+      "user_preview": "Read the following document excerpt and answer the question.\n\nDocument:\nLaird writes that Altan Khan abolished the native Mongol practices of shamanism and bloo",
+      "assistant_preview": "the native Mongol practices of shamanism and blood sacrifice"
+    },
+    {
+      "line": 158,
+      "schema": "instruction_output",
+      "category": "squad_doc",
+      "est_tokens": 232,
+      "user_preview": "Read the following document excerpt and answer the question.\n\nDocument:\nSome independent stereo manufacturers including JVC, Pioneer, Kenwood, Alpine, Sony, and",
+      "assistant_preview": "JVC, Pioneer, Kenwood, Alpine, Sony, and Harman Kardon"
+    },
+    {
+      "line": 165,
+      "schema": "instruction_output",
+      "category": "squad_doc",
+      "est_tokens": 197,
+      "user_preview": "Read the following document excerpt and answer the question.\n\nDocument:\nWith third parties like Namco, Square Enix, Electronic Arts, Sega, and Hudson Soft all m",
+      "assistant_preview": "Namco, Square Enix, Electronic Arts, Sega, and Hudson Soft"
+    }
+  ]
+}

reports/sft_combined_ready_report.md

@@ -0,0 +1,44 @@
+# Sentinel Coder Dataset Preflight
+
+Created: 2026-05-03 10:06:19
+Output: `/mnt/scratch/datasets/combined/sft_combined_ready.jsonl`
+Runtime: 7.25s
+
+## Summary
+
+- Seen rows: 42,138
+- Kept rows: 32,996 (78.3%)
+- Removed rows: 9,142
+- Max estimated tokens: 3,072
+- Max chars: 12,000
+
+## Removal Reasons
+
+- assistant_too_short: 8,451
+- user_too_short: 502
+- garbage_response: 100
+- repetitive_response: 85
+- too_many_chars: 2
+- binary_or_control_text: 1
+- too_many_tokens: 1
+
+## Token Buckets
+
+- 0001-0512: 28,102
+- 0513-1024: 4,034
+- 1025-1536: 712
+- 1537-2048: 112
+- 2049-3072: 36
+
+## Top Categories
+
+- librispeech: 7,785
+- wizardlm: 7,506
+- dolly: 6,539
+- alpaca: 6,325
+- textcaps: 4,492
+- squad_doc: 349
+
+## Source Schemas
+
+- instruction_output: 42,138

reports/train_metrics_final.json

@@ -0,0 +1 @@
+{"step": 5000, "train_loss": 0.0, "val_loss": 5.1359, "lr": 0.0, "tok_per_sec": 0, "eta_hrs": 0.0, "ts": 1777801509.3361254, "gpu": {"used_gb": 118.63, "peak_gb": 159.55, "total_gb": 205.8, "pct": 57.6, "sysfs_used_gb": 205.3, "sysfs_total_gb": 205.8, "sysfs_pct": 99.7}, "phase": "realignment_v2", "status": "done", "total_time_h": 50.28}