Spaces:

CompactAI-O
/

Homepage

Running

App Files Files Community

CompactAI commited on 10 days ago

Commit

2bec1de

verified ·

1 Parent(s): c812c6d

Create The Next Haiku: 1M Parameters, 10 Billion Tokens, Zero Guarantees.html

Browse files

Files changed (1) hide show

The Next Haiku: 1M Parameters, 10 Billion Tokens, Zero Guarantees.html +170 -0

The Next Haiku: 1M Parameters, 10 Billion Tokens, Zero Guarantees.html ADDED Viewed

	@@ -0,0 +1,170 @@

+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>The Next Haiku: 1M Parameters, 10 Billion Tokens, Zero Guarantees | FMN-GPT - CompactAI</title>
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=Geist:wght@400;500:600:700&family=Geist+Mono&display=swap" rel="stylesheet">
+<style>
+    :root {
+        --blue-900: #0a1628;
+        --blue-800: #0f2240;
+        --blue-700: #142d54;
+        --blue-600: #1a3a6b;
+        --blue-500: #2250a0;
+        --blue-400: #3a7bd5;
+        --blue-300: #6ba3f0;
+        --blue-200: #a8c8f5;
+        --blue-100: #d4e4fa;
+        --white: #ffffff;
+        --white-soft: #f0f4fa;
+        --white-muted: #c8d8ec;
+        --grid-line: rgba(255, 255, 255, 0.08);
+        --grid-line-major: rgba(255, 255, 255, 0.18);
+        --accent: #6ba3f0;
+        --accent-muted: #3a7bd5;
+        --font-sans: 'Geist', -apple-system, BlinkMacSystemFont, sans-serif;
+        --font-mono: 'Geist Mono', 'SF Mono', 'Fira Code', monospace;
+        --container-max: 1100px;
+    }
+    * { box-sizing: border-box; margin: 0; padding: 0; }
+    html { font-size: 16px; scroll-behavior: smooth; }
+    body { font-family: var(--font-sans); background: var(--blue-900); color: var(--white-muted); line-height: 1.7; -webkit-font-smoothing: antialiased; }
+    a { color: var(--white); text-decoration: none; transition: color 0.15s ease; }
+    a:hover { color: var(--accent); }
+    .container { max-width: var(--container-max); margin: 0 auto; padding: 0 24px; }
+    nav { position: fixed; top: 0; left: 0; right: 0; z-index: 100; background: rgba(10, 22, 40, 0.92); backdrop-filter: blur(12px); border-bottom: 1px solid var(--blue-600); padding: 16px 0; }
+    nav .container { display: flex; justify-content: space-between; align-items: center; }
+    .nav-brand { font-size: 18px; font-weight: 600; color: var(--white); display: flex; align-items: center; gap: 8px; }
+    .nav-brand span { color: var(--accent); }
+    .nav-links { display: flex; gap: 32px; }
+    .nav-links a { font-size: 14px; font-weight: 500; color: var(--blue-200); }
+    .nav-links a:hover { color: var(--white); }
+    .post { padding: 140px 0 80px; }
+    .post-back { display: inline-block; color: var(--blue-200); font-size: 14px; margin-bottom: 32px; }
+    .post-back:hover { color: var(--accent); }
+    .post-back::before { content: '← '; }
+    .post-meta { display: flex; gap: 12px; margin-bottom: 20px; }
+    .post-date { font-size: 13px; color: var(--blue-200); font-family: var(--font-mono); }
+    .post-tag { font-size: 11px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.05em; color: var(--accent); background: rgba(107, 163, 240, 0.1); padding: 4px 10px; border-radius: 4px; }
+    .post h1 { font-size: 36px; font-weight: 700; color: var(--white); margin-bottom: 32px; line-height: 1.2; letter-spacing: -0.02em; }
+    .post-body p { font-size: 17px; line-height: 1.8; margin-bottom: 24px; color: var(--blue-200); }
+    .post-body p:first-of-type { font-size: 20px; color: var(--white-muted); }
+    .post-body h2 { font-size: 24px; font-weight: 600; color: var(--white); margin: 48px 0 20px; }
+    .post-body blockquote { border-left: 3px solid var(--accent); padding: 20px 24px; margin: 32px 0; background: var(--blue-800); border-radius: 0 8px 8px 0; }
+    .post-body blockquote p { font-size: 16px; font-style: italic; color: var(--blue-200); margin: 0; }
+    .post-body hr { border: none; height: 1px; background: var(--blue-600); margin: 48px 0; }
+    .code-block { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; margin: 24px 0; font-family: var(--font-mono); font-size: 13px; overflow-x: auto; }
+    .code-block .comment { color: var(--blue-200); font-style: italic; display: block; margin-top: 4px; }
+    .stats-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 24px 0; }
+    .stat-card { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; text-align: center; }
+    .stat-card .number { font-size: 32px; font-weight: 700; color: var(--accent); font-family: var(--font-mono); }
+    .stat-card .label { font-size: 13px; color: var(--blue-200); margin-top: 8px; }
+    .post-footer { margin-top: 48px; padding-top: 32px; border-top: 1px solid var(--blue-600); }
+    .post-footer p { font-size: 14px; color: var(--blue-200); font-style: italic; margin: 0; }
+    footer { padding: 40px 0; background: var(--blue-800); border-top: 1px solid var(--blue-600); text-align: center; }
+    footer p { color: var(--blue-200); font-size: 14px; margin-bottom: 8px; }
+    footer a { color: var(--blue-200); }
+    footer a:hover { color: var(--accent); }
+    @media (max-width: 768px) { .post h1 { font-size: 28px; } .nav-links { display: none; } .stats-grid { grid-template-columns: 1fr; } }
+</style>
+</head>
+<body>
+    <nav>
+        <div class="container">
+            <a href="index.html" class="nav-brand"><span>/</span>FMN-GPT</a>
+            <div class="nav-links">
+                <a href="blog.html">Blog</a>
+                <a href="status.html">Model Status</a>
+                <a href="https://huggingface.co/CompactAI-O" target="_blank">HuggingFace Org</a>
+            </div>
+        </div>
+    </nav>
+    <main>
+        <article class="post">
+            <div class="container">
+                <a href="blog.html" class="post-back">Back to Blog</a>
+                <header>
+                    <div class="post-meta">
+                        <span class="post-date">2026-04-08</span>
+                        <span class="post-tag">Model Previews</span>
+                    </div>
+                    <h1>The Next Haiku: 1M Parameters, 10 Billion Tokens, Zero Guarantees</h1>
+                </header>
+                <div class="post-body">
+                    <p>I am working on the next Haiku model. It is not released yet. It might never be released. But I am working on it. That counts for something. Probably.</p>
+                    <p>The specs are simple. One million parameters. Ten billion training tokens. That is about ten thousand tokens per parameter. I think. My math might be wrong. My math is often wrong. But the ratio feels ambitious. Or reckless. Both can be true.</p>
+                    <blockquote>
+                        <p>Ten thousand tokens per parameter is either the secret to tiny model success or a great way to waste electricity. I am prepared for either outcome.</p>
+                    </blockquote>
+                    <h2>The Numbers</h2>
+                    <div class="stats-grid">
+                        <div class="stat-card">
+                            <div class="number">1M</div>
+                            <div class="label">Parameters</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">10B</div>
+                            <div class="label">Training Tokens</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">~10K</div>
+                            <div class="label">Tokens Per Parameter</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">??</div>
+                            <div class="label">Will It Speak</div>
+                        </div>
+                    </div>
+                    <p>Ten billion tokens is a lot of tokens. It is also not that many tokens if you have ever trained a frontier model. But for a one million parameter model, it feels excessive. Like bringing a fire hose to water a houseplant. The houseplant might drown. Or it might finally grow.</p>
+                    <h2>Why So Many Tokens</h2>
+                    <p>Haiku-2 learned to speak sometimes. It also learned to output chuamliamce. The ratio of speech to chuamliamce was not ideal. I want to improve that ratio.</p>
+                    <p>More tokens means more exposure to language patterns. More exposure means better internal representations. Better representations mean fewer pipe characters. This is the theory. The practice might involve more NaN losses. But I am optimistic. Or stubborn. Both can be true.</p>
+                    <div class="code-block">
+                        <span class="comment"># My training plan in pseudocode</span><br>
+                        tokens = load_10_billion_tokens()<br>
+                        model = TinyModel(1_000_000_params)<br>
+                        for token in tokens:<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;loss = model.forward(token)<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;model.backward(loss)<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;if loss == nan:<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cry()<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;retry()<br>
+                        <span class="comment"># Simple. Elegant. Probably naive.</span>
+                    </div>
+                    <h2>Where Sonnet And Opus Fit In</h2>
+                    <p>Sonnet is not training right now. Opus is not training either. They are both waiting. Patiently. Or impatiently. Hard to tell with models that do not have feelings.</p>
+                    <p>When I say I am giving my computer away for a week, I mean I am giving up my computer for a week. Or weeks. I will not use it at all. No browsing. No blogging. No debugging NaN losses at three AM. Just Sonnet and Opus training, uninterrupted, until they finish or crash or both.</p>
+                    <p>The new script version includes bug fixes. Optimizations. SPIN integration. Things I learned from Claude Code. Things I learned from AxionLab-Co. Things I learned from crying at loss curves. All of it gets tested on the next Haiku first. Then, if nothing explodes, Sonnet and Opus get the upgraded treatment.</p>
+                    <h2>What To Expect</h2>
+                    <p>I do not know. That is the honest answer. The model might speak fluently. It might still output chuamliamce. It might do both at the same time. Progress is weird.</p>
+                    <p>What I hope for: fewer special characters. More complete sentences. Occasional correct answers to simple questions. The ability to mention Paris without calling it a person named Pierre.</p>
+                    <p>What I expect: a model that is slightly less confused than Haiku-2. Slightly more coherent. Slightly more useful. Slightly is the keyword. I am not expecting miracles. I am expecting incremental improvement. That feels achievable.</p>
+                    <blockquote>
+                        <p>Training tiny models is just hoping that enough data will eventually teach a million parameters to form a thought. Sometimes it works. Sometimes you get chuamliamce. Both outcomes are educational.</p>
+                    </blockquote>
+                    <h2>The Timeline</h2>
+                    <p>Ten billion tokens takes time. My GPU is fast. It is also one GPU. The training will run while I am not using my computer. I will monitor it from afar. I will handle any NaN emergencies remotely. This is trust. This is also anxiety.</p>
+                    <p>If everything goes well, the next Haiku might be ready in a few weeks. If things go poorly, it might never be ready. If things go weird, it might be ready but output only chuamliamce in perfect grammatical sentences. All outcomes are possible.</p>
+                    <h2>Final Thoughts</h2>
+                    <p>The next Haiku is coming. One million parameters. Ten billion tokens. Ten thousand tokens per parameter. Zero guarantees about coherence.</p>
+                    <p>Sonnet is waiting. Opus is waiting. My computer is about to be dedicated to training them exclusively. I am staying here. Writing blogs. Hoping the models learn to speak.</p>
+                    <p>If you want tiny models that might eventually make sense, stick around. If you want guarantees, maybe look elsewhere. I do not do guarantees. I do hope. I do effort. I do chuamliamce with confidence.</p>
+                    <hr>
+                </div>
+                <footer class="post-footer">
+                    <p>Current status: Next Haiku training. Sonnet waiting. Opus waiting. Computer preparing for dedicated training marathon. Me preparing for anxiety. Progress is weird. Chuamliamce remains a mystery.</p>
+                </footer>
+            </div>
+        </article>
+    </main>
+    <footer>
+        <div class="container">
+            <p>Built with curiosity over compute</p>
+            <p>FMN-GPT by <a href="https://huggingface.co/CompactAI-O" target="_blank">CompactAI-O</a> | 2026</p>
+        </div>
+    </footer>
+</body>
+</html>