README.md

---
license: apache-2.0
language:
  - en
library_name: pytorch
pipeline_tag: text-generation
tags:
  - cybersecurity
  - language-model
  - from-scratch
  - small-model
  - cti
  - ctibench
  - chat
datasets:
  - trendmicro-ailab/Primus-Seed
  - trendmicro-ailab/Primus-FineWeb
  - AI4Sec/cti-bench
model-index:
  - name: GhostLM v0.9 chat (81M)
    results:
      - task:
          type: multiple-choice
          name: CTIBench MCQ (debiased text-scoring, n=2500)
        dataset:
          name: CTIBench MCQ
          type: AI4Sec/cti-bench
          split: test
        metrics:
          - name: 2-permutation per-perm avg
            type: accuracy
            value: 0.289
      - task:
          type: multiple-choice
          name: SecQA (n=210)
        dataset:
          name: SecQA
          type: zefang-liu/secqa
        metrics:
          - name: accuracy
            type: accuracy
            value: 0.393
      - task:
          type: multiple-choice
          name: GhostLM CTF MCQ (in-repo, n=30)
        dataset:
          name: GhostLM CTF eval
          type: in-repo
        metrics:
          - name: accuracy
            type: accuracy
            value: 0.592
      - task:
          type: text-generation
          name: GhostLM free-form fact recall (n=50)
        dataset:
          name: GhostLM fact-recall bench
          type: in-repo
        metrics:
          - name: substring-match accuracy
            type: accuracy
            value: 0.02
---

# GhostLM v0.9 chat (81M, from-scratch cybersecurity LM)

GhostLM is a multi-rung scale-ladder cybersecurity language model
trained entirely from scratch in PyTorch. **v0.9 chat is the bench
winner of the ghost-small (45-81M) line on every multiple-choice
benchmark we evaluated.** It is also where the line saturates: at 81M
parameters the model has the *register* of cybersec writing but not the
*facts* in any retrievable form. The next rung is
[ghost-base](https://github.com/joemunene-by/GhostLM/blob/main/docs/ghost_base_spec.md)
(~360M, SmolLM2-360M shape), gated on rented GPU compute.

This repo holds the slim inference checkpoint
(`best_model.pt`, 324 MB, model + config only, optimizer state stripped).

## v0.9.5 update (2026-05-08): nine differentiation bets, 1,505 templated SFT records ready

The strategic frame went from "six bets, three measured" (v0.9.4) to
"nine bets, all shipped, 1,505 deterministic SFT records ready for
the v1.0 GPU run." The new bets answer **"what would make GhostLM
exceptional, beyond what general-purpose small LMs offer?"**

Strategic frame: [`docs/differentiation.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/differentiation.md).

| Bet | Status | Result |
|---|---|---|
| 1. Tool-grounded SFT | **training data ready** | 424 templated traces, 98.6% acceptance under `trace_quality_ok`; ~10% "not found" injection trains lookup-failure acknowledgement. [tool_use_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/tool_use_synth.md) |
| 2. Daily LoRA over fresh threat-intel | scaffolded | `scripts/daily_finetune.py`, ~1-2 GPU hr/day |
| 3. Custom 32K BPE | **measured + settled** | +4.0% on cyber, -2.5% on general vs GPT-2 BPE; +25-35% projection falsified. [bpe_corpus_ablation.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/bpe_corpus_ablation.md) |
| 4. Long context via RoPE NTK | scaffolded | `scripts/extend_context_ntk.py`, ~3-5 GPU hr |
| 5. MoE for ghost-1B+ | **smoke validated** | 100-step training PASS. [moe_training_smoke.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/moe_training_smoke.md); presets `ghost-1b` (2.1B/1.2B-active) and `ghost-3b` (6.0B/3.3B-active) |
| 6. Format-aware pretrain (STIX/YARA/Sigma/MISP) | **measured baseline + training data ready** | v0.9 baseline locked at 0/32 = 0% [Wilson 95% CI 0.0-10.7]. 560 templated records ready. [format_baseline_v09.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/format_baseline_v09.md), [format_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/format_synth.md) |
| 7. Code-for-security | **NEW**, training data ready | 12-pattern bank covering OWASP-Top-10 CWE classes (Python/JS/C); 48 records, 100% pass. [code_security_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/code_security_synth.md) |
| 8. Binary / hex literacy | **NEW**, training data ready, **most novel bet** | 15-pattern bank: PE/ELF/Mach-O/ZIP/PDF/OLE2/PNG file magic, UPX/Themida packers, NOP sleds + x64 syscall, PE Optional Header Magic + Machine, x64 execve('/bin/sh') shellcode; 44 records, 100% pass. **No other small cybersec LM does this.** [binary_literacy_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/binary_literacy_synth.md) |
| 9. Provenance / cite tags | **NEW**, training data ready | 429 cite-augmented tool-use traces with `<\|cite\|>{source_type}:{id}#field<\|/cite\|>` inline in the answer; 99.8% acceptance under `trace_with_cites_quality_ok`. Stacks on bet 1 for ~853-record SFT corpus. [provenance_synth.md](https://github.com/joemunene-by/GhostLM/blob/main/docs/provenance_synth.md) |

### Combined templated-synth corpus

| Bet | Records | Acceptance |
|---|---:|---:|
| 1 (tool-use, plain) | 424 | 98.6% |
| 6 (STIX / YARA / Sigma / MISP) | 560 | 99.8% |
| 7 (code-for-security) | 48 | 100.0% |
| 8 (binary / hex literacy) | 44 | 100.0% |
| 9 (cite-augmented tool-use) | 429 | 99.8% |
| **TOTAL** | **1,505** | **99.4%** |

That's the deterministic floor. LLM-distilled records on top
(bet 1 production at ~$200, bet 6 production at ~$50-100 on
Anthropic) bring the realistic ghost-base SFT mix to ~10K records
for a few hundred dollars, with no GPU spend until the actual
pretrain run.

The v0.9 chat checkpoint in this repo is unchanged; it's the
baseline against which all bet measurements are made.

## Bench numbers

All benches run with debiased multi-permutation text-scoring on
checkpointed CPU/GPU inference. Methodology in
[`docs/ctibench_bias_finding.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/ctibench_bias_finding.md).

| Benchmark | Records | v0.4 chat-v3 | v0.7 chat | **v0.9 chat** | Random |
|---|---:|---:|---:|---:|---:|
| CTIBench MCQ (full split) | 2,500 | 27.6% | 27.2% | **28.9%** | 25.0% |
| In-repo CTF MCQ eval | 30 | 50.0% | 50.0% | **59.2%** | 25.0% |
| SecQA (external, n=210) | 210 | 35.0% | 37.6% | **39.3%** | 25.0% |
| Free-form fact recall | 50 | 0/50 | 1/50 | **1/50** | 0/50 |

v0.9 wins every multiple-choice benchmark by 0.7-9.2 pp. The MCQ
ranking holds across CTIBench, the in-repo CTF eval, and the
external SecQA bench.

**But free-form fact recall is at floor across the entire 81M ghost-small
rung.** A 50-question hand-written fact-recall set (CVE / CWE / MITRE /
OWASP / crypto / protocol / misc) graded by substring match scores
0-2% across every chat-tune in the line. The v0.9 model's one "hit"
("256" appearing in a SHA-256 question) is arguably spurious. **MCQ wins
measure register matching and topic distinctness, not factual recall.**

## Architecture

| Field | Value |
|---|---|
| Layers | 6 |
| d_model | 768 |
| Attention heads | 12 (head_dim 64) |
| FFN | SwiGLU, hidden = `int(d_ff × 2/3)` rounded to 64 = 2048 |
| Normalization | RMSNorm |
| Position | RoPE (base 10000) |
| Vocab | 50,264 (GPT-2 50K BPE + 7 special tokens) |
| Context | 512 train, 1024 inference |
| Total params | ~81M |

Same architecture as ghost-small-v0.7. The 273M-token v0.9 corpus is
what produces the bench delta over v0.7.

## Training data

Pretrain corpus: 273M tokens spanning

- **PRIMUS-Seed** (Trend Micro AI Lab, Apache 2.0): curated cybersec text
- **PRIMUS-FineWeb** (Trend Micro AI Lab, ODC-By): TinyBERT-filtered cybersec subset of CommonCrawl
- **NVD CVEs** (NIST, public domain): full v2 description text
- **MITRE ATT&CK + CWE + CAPEC** (MITRE, custom permissive): technique / weakness / pattern descriptions
- **OWASP** (Top 10, ASVS, Cheat Sheets, WSTG; CC-BY-SA): web-app security guidance
- **IETF RFCs** (BCP 78, public): security-relevant RFCs
- **CTFtime + Exploit-DB** (open): real CTF write-ups and exploit POCs
- **arXiv cs.CR**: full-text academic papers
- **fact-QA**: ~11K Q&A pairs distilled by Qwen-14B from the corpus

Per-source breakdown in
[`CORPUS.md`](https://github.com/joemunene-by/GhostLM/blob/main/CORPUS.md).

Chat-tuning: SFT on 1,802 MCQ + small-talk + identity examples using
the chat-v3 recipe. Three role tokens (`<|ghost_user|>`,
`<|ghost_assistant|>`, `<|ghost_end|>`) added to the tokenizer.

## Intended use

- Educational: a transparent, hand-written reference implementation of
  a from-scratch decoder-only cybersecurity LM, trained on a curated
  corpus, with all code on GitHub and all recipes documented.
- Research: a bench artifact for "what does an 81M from-scratch cyber
  LM actually score on CTIBench / SecQA?" The honest answer (28.9% /
  39.3%) is meaningful evidence about the parameter-count requirement
  for factual recall on cybersec MCQ.

## What this model is NOT for

- **Anything that depends on factual recall.** Free-form fact recall
  is at floor. CVE numbers, version chains, MITRE technique IDs,
  CVSS scores produced by this model are unreliable. Verify against
  authoritative sources.
- **General-purpose tasks.** Outside cybersecurity the model politely
  declines and returns to its domain. Do not expect it to summarize
  news, write code, or answer arbitrary questions.
- **Production cybersec workflows.** Not for incident response,
  threat hunting, or any decision that affects real systems.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download
from ghostlm.config import GhostLMConfig
from ghostlm.model import GhostLM
from ghostlm.tokenizer import GhostTokenizer
from dataclasses import fields

# Pull weights
ckpt_path = hf_hub_download(
    repo_id="Ghostgim/GhostLM-v0.9-experimental",
    filename="best_model.pt",
)

# Load
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
saved = ckpt["config"]
config = GhostLMConfig(**{
    f.name: saved[f.name] for f in fields(GhostLMConfig) if f.name in saved
})
model = GhostLM(config)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

tokenizer = GhostTokenizer()  # GPT-2 BPE + 7 special tokens

# Multi-turn chat using the role tokens
turns = [{"role": "user", "content": "What is XSS?"}]
prompt_ids = tokenizer.format_chat_prompt(turns)
# ... see scripts/chat.py for full generation loop
```

The full code (architecture, tokenizer, generation, eval, training) is
in the [GhostLM GitHub repo](https://github.com/joemunene-by/GhostLM).

## Live demo

[`huggingface.co/spaces/Ghostgim/ghostlm`](https://huggingface.co/spaces/Ghostgim/ghostlm)

Pulls these weights via `hf_hub_download` on first launch. CPU inference
takes ~15-25 s per reply at the default 200-token cap. The demo is
intentionally honest about the fact-recall floor; expect register-shaped
output rather than reliable answers.

## Caveats

- **Hallucination is the norm**, not the exception. This is an 81M
  from-scratch model, not a fine-tuned 7B foundation model.
- **MCQ wins do not imply factual recall.** Test with the free-form
  fact-recall benchmark, not just CTIBench.
- **Pretrain corpus is sub-Chinchilla.** 273M tokens for 81M params is
  ~3× under Chinchilla-optimal; the chat tune partially compensates,
  but the model is undertrained relative to its capacity.

## Citation

```bibtex
@misc{munene2026ghostlm,
  title         = {GhostLM: a from-scratch cybersecurity language model on a transparent scale ladder},
  author        = {Munene, Joe},
  year          = {2026},
  howpublished  = {\url{https://github.com/joemunene-by/GhostLM}},
  note          = {v0.9.5 release; 81M-parameter chat checkpoint plus nine differentiation bets, 1505 templated SFT records}
}
```

## Roadmap

The next rung is **ghost-base (~360M, SmolLM2-360M shape)**, gated on
rented GPU compute. Acceptance gate:

- ≥40% on debiased CTIBench (full n=2500), OR
- ≥65% on the in-repo CTF MCQ eval, OR
- ≥30% on the 50-question free-form fact-recall set.

The fact-recall bar is the truth metric. Spec at
[`docs/ghost_base_spec.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/ghost_base_spec.md);
multi-year pathway through ghost-7B in
[`docs/hardware_pathway.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/hardware_pathway.md).

After ghost-base lands, the v0.9.4 differentiation bets compose on
top of it: tool-use SFT (bet 1) on the fresh ghost-base, format-aware
pretrain mix (bet 6) using the 560 templated records plus
LLM-distilled traces, RoPE NTK context extension to 16K (bet 4), and
eventually ghost-1B with native MoE from step 0 (bet 5). Sequencing
detail in
[`docs/differentiation.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/differentiation.md).

## License

Apache 2.0. Same license as the GhostLM source code.

Built by Joe Munene.