--- license: apache-2.0 language: - en library_name: pytorch pipeline_tag: text-generation tags: - cybersecurity - language-model - from-scratch - small-model - cti - ctibench - chat datasets: - trendmicro-ailab/Primus-Seed - trendmicro-ailab/Primus-FineWeb - AI4Sec/cti-bench model-index: - name: GhostLM v0.9 chat (81M) results: - task: type: multiple-choice name: CTIBench MCQ (debiased text-scoring, n=2500) dataset: name: CTIBench MCQ type: AI4Sec/cti-bench split: test metrics: - name: 2-permutation per-perm avg type: accuracy value: 0.289 - task: type: multiple-choice name: SecQA (n=210) dataset: name: SecQA type: zefang-liu/secqa metrics: - name: accuracy type: accuracy value: 0.393 - task: type: multiple-choice name: GhostLM CTF MCQ (in-repo, n=30) dataset: name: GhostLM CTF eval type: in-repo metrics: - name: accuracy type: accuracy value: 0.592 - task: type: text-generation name: GhostLM free-form fact recall (n=50) dataset: name: GhostLM fact-recall bench type: in-repo metrics: - name: substring-match accuracy type: accuracy value: 0.02 --- # GhostLM v0.9 chat (81M, from-scratch cybersecurity LM) GhostLM is a multi-rung scale-ladder cybersecurity language model trained entirely from scratch in PyTorch. **v0.9 chat is the bench winner of the ghost-small (45-81M) line on every multiple-choice benchmark we evaluated.** It is also where the line saturates: at 81M parameters the model has the *register* of cybersec writing but not the *facts* in any retrievable form. The next rung is [ghost-base](https://github.com/joemunene-by/GhostLM/blob/main/docs/ghost_base_spec.md) (~360M, SmolLM2-360M shape), gated on rented GPU compute. This repo holds the slim inference checkpoint (`best_model.pt`, 324 MB, model + config only, optimizer state stripped). ## v0.9.5 update (2026-05-08): nine differentiation bets, 1,505 templated SFT records ready The strategic frame went from "six bets, three measured" (v0.9.4) to "nine bets, all shipped, 1,505 deterministic SFT records ready for the v1.0 GPU run." The new bets answer **"what would make GhostLM exceptional, beyond what general-purpose small LMs offer?"** Strategic frame: [`docs/differentiation.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/differentiation.md). | Bet | Status | Result | |---|---|---| | 1. Tool-grounded SFT | **training data ready** | 424 templated traces, 98.6% acceptance under `trace_quality_ok`; ~10% "not found" injection trains lookup-failure acknowledgement. [tool_use_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/tool_use_synth.md) | | 2. Daily LoRA over fresh threat-intel | scaffolded | `scripts/daily_finetune.py`, ~1-2 GPU hr/day | | 3. Custom 32K BPE | **measured + settled** | +4.0% on cyber, -2.5% on general vs GPT-2 BPE; +25-35% projection falsified. [bpe_corpus_ablation.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/bpe_corpus_ablation.md) | | 4. Long context via RoPE NTK | scaffolded | `scripts/extend_context_ntk.py`, ~3-5 GPU hr | | 5. MoE for ghost-1B+ | **smoke validated** | 100-step training PASS. [moe_training_smoke.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/moe_training_smoke.md); presets `ghost-1b` (2.1B/1.2B-active) and `ghost-3b` (6.0B/3.3B-active) | | 6. Format-aware pretrain (STIX/YARA/Sigma/MISP) | **measured baseline + training data ready** | v0.9 baseline locked at 0/32 = 0% [Wilson 95% CI 0.0-10.7]. 560 templated records ready. [format_baseline_v09.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/format_baseline_v09.md), [format_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/format_synth.md) | | 7. Code-for-security | **NEW**, training data ready | 12-pattern bank covering OWASP-Top-10 CWE classes (Python/JS/C); 48 records, 100% pass. [code_security_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/code_security_synth.md) | | 8. Binary / hex literacy | **NEW**, training data ready, **most novel bet** | 15-pattern bank: PE/ELF/Mach-O/ZIP/PDF/OLE2/PNG file magic, UPX/Themida packers, NOP sleds + x64 syscall, PE Optional Header Magic + Machine, x64 execve('/bin/sh') shellcode; 44 records, 100% pass. **No other small cybersec LM does this.** [binary_literacy_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/binary_literacy_synth.md) | | 9. Provenance / cite tags | **NEW**, training data ready | 429 cite-augmented tool-use traces with `<\|cite\|>{source_type}:{id}#field<\|/cite\|>` inline in the answer; 99.8% acceptance under `trace_with_cites_quality_ok`. Stacks on bet 1 for ~853-record SFT corpus. [provenance_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/provenance_synth.md) | ### Combined templated-synth corpus | Bet | Records | Acceptance | |---|---:|---:| | 1 (tool-use, plain) | 424 | 98.6% | | 6 (STIX / YARA / Sigma / MISP) | 560 | 99.8% | | 7 (code-for-security) | 48 | 100.0% | | 8 (binary / hex literacy) | 44 | 100.0% | | 9 (cite-augmented tool-use) | 429 | 99.8% | | **TOTAL** | **1,505** | **99.4%** | That's the deterministic floor. LLM-distilled records on top (bet 1 production at ~$200, bet 6 production at ~$50-100 on Anthropic) bring the realistic ghost-base SFT mix to ~10K records for a few hundred dollars, with no GPU spend until the actual pretrain run. The v0.9 chat checkpoint in this repo is unchanged; it's the baseline against which all bet measurements are made. ## Bench numbers All benches run with debiased multi-permutation text-scoring on checkpointed CPU/GPU inference. Methodology in [`docs/ctibench_bias_finding.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/ctibench_bias_finding.md). | Benchmark | Records | v0.4 chat-v3 | v0.7 chat | **v0.9 chat** | Random | |---|---:|---:|---:|---:|---:| | CTIBench MCQ (full split) | 2,500 | 27.6% | 27.2% | **28.9%** | 25.0% | | In-repo CTF MCQ eval | 30 | 50.0% | 50.0% | **59.2%** | 25.0% | | SecQA (external, n=210) | 210 | 35.0% | 37.6% | **39.3%** | 25.0% | | Free-form fact recall | 50 | 0/50 | 1/50 | **1/50** | 0/50 | v0.9 wins every multiple-choice benchmark by 0.7-9.2 pp. The MCQ ranking holds across CTIBench, the in-repo CTF eval, and the external SecQA bench. **But free-form fact recall is at floor across the entire 81M ghost-small rung.** A 50-question hand-written fact-recall set (CVE / CWE / MITRE / OWASP / crypto / protocol / misc) graded by substring match scores 0-2% across every chat-tune in the line. The v0.9 model's one "hit" ("256" appearing in a SHA-256 question) is arguably spurious. **MCQ wins measure register matching and topic distinctness, not factual recall.** ## Architecture | Field | Value | |---|---| | Layers | 6 | | d_model | 768 | | Attention heads | 12 (head_dim 64) | | FFN | SwiGLU, hidden = `int(d_ff × 2/3)` rounded to 64 = 2048 | | Normalization | RMSNorm | | Position | RoPE (base 10000) | | Vocab | 50,264 (GPT-2 50K BPE + 7 special tokens) | | Context | 512 train, 1024 inference | | Total params | ~81M | Same architecture as ghost-small-v0.7. The 273M-token v0.9 corpus is what produces the bench delta over v0.7. ## Training data Pretrain corpus: 273M tokens spanning - **PRIMUS-Seed** (Trend Micro AI Lab, Apache 2.0): curated cybersec text - **PRIMUS-FineWeb** (Trend Micro AI Lab, ODC-By): TinyBERT-filtered cybersec subset of CommonCrawl - **NVD CVEs** (NIST, public domain): full v2 description text - **MITRE ATT&CK + CWE + CAPEC** (MITRE, custom permissive): technique / weakness / pattern descriptions - **OWASP** (Top 10, ASVS, Cheat Sheets, WSTG; CC-BY-SA): web-app security guidance - **IETF RFCs** (BCP 78, public): security-relevant RFCs - **CTFtime + Exploit-DB** (open): real CTF write-ups and exploit POCs - **arXiv cs.CR**: full-text academic papers - **fact-QA**: ~11K Q&A pairs distilled by Qwen-14B from the corpus Per-source breakdown in [`CORPUS.md`](https://github.com/joemunene-by/GhostLM/blob/main/CORPUS.md). Chat-tuning: SFT on 1,802 MCQ + small-talk + identity examples using the chat-v3 recipe. Three role tokens (`<|ghost_user|>`, `<|ghost_assistant|>`, `<|ghost_end|>`) added to the tokenizer. ## Intended use - Educational: a transparent, hand-written reference implementation of a from-scratch decoder-only cybersecurity LM, trained on a curated corpus, with all code on GitHub and all recipes documented. - Research: a bench artifact for "what does an 81M from-scratch cyber LM actually score on CTIBench / SecQA?" The honest answer (28.9% / 39.3%) is meaningful evidence about the parameter-count requirement for factual recall on cybersec MCQ. ## What this model is NOT for - **Anything that depends on factual recall.** Free-form fact recall is at floor. CVE numbers, version chains, MITRE technique IDs, CVSS scores produced by this model are unreliable. Verify against authoritative sources. - **General-purpose tasks.** Outside cybersecurity the model politely declines and returns to its domain. Do not expect it to summarize news, write code, or answer arbitrary questions. - **Production cybersec workflows.** Not for incident response, threat hunting, or any decision that affects real systems. ## Loading ```python import torch from huggingface_hub import hf_hub_download from ghostlm.config import GhostLMConfig from ghostlm.model import GhostLM from ghostlm.tokenizer import GhostTokenizer from dataclasses import fields # Pull weights ckpt_path = hf_hub_download( repo_id="Ghostgim/GhostLM-v0.9-experimental", filename="best_model.pt", ) # Load ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False) saved = ckpt["config"] config = GhostLMConfig(**{ f.name: saved[f.name] for f in fields(GhostLMConfig) if f.name in saved }) model = GhostLM(config) model.load_state_dict(ckpt["model_state_dict"]) model.eval() tokenizer = GhostTokenizer() # GPT-2 BPE + 7 special tokens # Multi-turn chat using the role tokens turns = [{"role": "user", "content": "What is XSS?"}] prompt_ids = tokenizer.format_chat_prompt(turns) # ... see scripts/chat.py for full generation loop ``` The full code (architecture, tokenizer, generation, eval, training) is in the [GhostLM GitHub repo](https://github.com/joemunene-by/GhostLM). ## Live demo [`huggingface.co/spaces/Ghostgim/ghostlm`](https://huggingface.co/spaces/Ghostgim/ghostlm) Pulls these weights via `hf_hub_download` on first launch. CPU inference takes ~15-25 s per reply at the default 200-token cap. The demo is intentionally honest about the fact-recall floor; expect register-shaped output rather than reliable answers. ## Caveats - **Hallucination is the norm**, not the exception. This is an 81M from-scratch model, not a fine-tuned 7B foundation model. - **MCQ wins do not imply factual recall.** Test with the free-form fact-recall benchmark, not just CTIBench. - **Pretrain corpus is sub-Chinchilla.** 273M tokens for 81M params is ~3× under Chinchilla-optimal; the chat tune partially compensates, but the model is undertrained relative to its capacity. ## Citation ```bibtex @misc{munene2026ghostlm, title = {GhostLM: a from-scratch cybersecurity language model on a transparent scale ladder}, author = {Munene, Joe}, year = {2026}, howpublished = {\url{https://github.com/joemunene-by/GhostLM}}, note = {v0.9.5 release; 81M-parameter chat checkpoint plus nine differentiation bets, 1505 templated SFT records} } ``` ## Roadmap The next rung is **ghost-base (~360M, SmolLM2-360M shape)**, gated on rented GPU compute. Acceptance gate: - ≥40% on debiased CTIBench (full n=2500), OR - ≥65% on the in-repo CTF MCQ eval, OR - ≥30% on the 50-question free-form fact-recall set. The fact-recall bar is the truth metric. Spec at [`docs/ghost_base_spec.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/ghost_base_spec.md); multi-year pathway through ghost-7B in [`docs/hardware_pathway.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/hardware_pathway.md). After ghost-base lands, the v0.9.4 differentiation bets compose on top of it: tool-use SFT (bet 1) on the fresh ghost-base, format-aware pretrain mix (bet 6) using the 560 templated records plus LLM-distilled traces, RoPE NTK context extension to 16K (bet 4), and eventually ghost-1B with native MoE from step 0 (bet 5). Sequencing detail in [`docs/differentiation.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/differentiation.md). ## License Apache 2.0. Same license as the GhostLM source code. Built by Joe Munene.