| --- |
| title: GhostLM |
| emoji: 🔐 |
| colorFrom: purple |
| colorTo: gray |
| sdk: gradio |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| short_description: From-scratch 81M cybersecurity LM, v0.9 chat demo |
| --- |
| |
| # GhostLM Chat (v0.9) |
|
|
| Interactive Gradio chat for **GhostLM v0.9 chat**, an 81M-parameter |
| cybersecurity language model trained from scratch in PyTorch. |
|
|
| The Space ships a multi-turn chat interface backed by the v0.9 chat |
| weights. Generation uses the model's three role tokens |
| (`<|ghost_user|>`, `<|ghost_assistant|>`, `<|ghost_end|>`) and stops the |
| moment the assistant's `<|ghost_end|>` is sampled. |
|
|
| ## Bench numbers (v0.9 chat) |
|
|
| The v0.9 chat checkpoint is the bench winner of the ghost-small line on |
| every multiple-choice benchmark we ran: |
|
|
| | Benchmark | n | Score | |
| |---|---:|---:| |
| | [CTIBench MCQ](https://huggingface.co/datasets/AI4Sec/cti-bench), 2-permutation debiased | 2,500 | **28.9%** | |
| | in-repo CTF MCQ eval | 30 | **59.2%** | |
| | SecQA (external) | 210 | **39.3%** | |
| | free-form fact recall, hand-written | 50 | 1/50 (at floor) | |
|
|
| Free-form fact recall is at floor across the entire 81M ghost-small |
| rung by design. At this parameter count the model has the *register* |
| of cybersec writing but not the *facts* in any retrievable form. The |
| next rung (ghost-base ~360M, SmolLM2-360M shape) is gated on rented |
| GPU compute. Spec: [`docs/ghost_base_spec.md`](https://github.com/joemunene-by/GhostLM/blob/main/docs/ghost_base_spec.md). |
|
|
| ## Architecture |
|
|
| 6 layers, d_model 768, 12 heads, with RoPE + SwiGLU + RMSNorm. Pretrain |
| corpus: 273M tokens spanning PRIMUS-Seed, PRIMUS-FineWeb, NVD CVEs, |
| MITRE ATT&CK, CWE, CAPEC, OWASP, IETF RFCs, Exploit-DB, CTFtime, arXiv |
| cs.CR, plus a fact-dense Q&A set. Chat-tuned with the chat-v3 SFT recipe. |
| |
| ## Where the weights live |
| |
| The 324 MB slim weights are stored in the Models repo |
| [`Ghostgim/GhostLM-v0.9-experimental`](https://huggingface.co/Ghostgim/GhostLM-v0.9-experimental). |
| The Space's `app.py` calls `huggingface_hub.hf_hub_download` on first |
| launch and caches them locally. This keeps the Space comfortably under |
| HF's 1 GB free-tier LFS cap; the source code stays small and the |
| weights are versioned separately. |
|
|
| ## Source |
|
|
| GitHub: [`joemunene-by/GhostLM`](https://github.com/joemunene-by/GhostLM) |
|
|
| Run locally: |
|
|
| ```bash |
| git clone https://github.com/joemunene-by/GhostLM |
| cd GhostLM |
| pip install -r demo/requirements.txt |
| PYTHONPATH=. python3 demo/app.py |
| ``` |
|
|
| The model is small enough to run on a laptop CPU; expect ~10-25 s per |
| chat reply at the default 200-token cap. |
|
|
| ## Caveats |
|
|
| - **Hallucinates facts.** CVE IDs, CVSS scores, technique IDs, version |
| ranges are all unreliable. Outputs are register-shaped fiction, not |
| reference material. Verify against authoritative sources. |
| - **No general-knowledge tuning.** Outside cybersecurity the model |
| politely declines and returns to its domain. Don't expect it to |
| summarize a news article or write Python. |
| - **The MCQ wins do not mean factual recall.** The 28.9% on debiased |
| CTIBench measures the register-matching component of the |
| benchmark; the free-form fact recall floor (1/50) is the truth metric. |
|
|
| ## License |
|
|
| Apache 2.0. Built by Joe Munene. |
|
|
