Spaces:

SupraLabs
/

Blog

Running

App Files Files Community

AxionLab-official commited on 10 days ago

Commit

af833ed

verified ·

1 Parent(s): a8476f9

Create 1bit-quantization.html

Browse files

Files changed (1) hide show

1bit-quantization.html +298 -0

1bit-quantization.html ADDED Viewed

	@@ -0,0 +1,298 @@

+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>1Bit Quantization: a 50/50 chance for small models | SupraLabs Blog</title>
+    <style>
+        :root {
+            --bg: #0f0f0f;
+            --surface: #1a1a1a;
+            --border: #333;
+            --text: #e0e0e0;
+            --accent: #536bfe;
+            --muted: #888;
+            --font-mono: 'JetBrains Mono', 'Fira Code', monospace;
+        }
+        * { margin: 0; padding: 0; box-sizing: border-box; }
+        body {
+            background-color: var(--bg);
+            color: var(--text);
+            font-family: 'Inter', -apple-system, sans-serif;
+            line-height: 1.6;
+            padding: 2rem;
+        }
+        code, pre, .mono { font-family: var(--font-mono); }
+        .container { max-width: 900px; margin: 0 auto; }
+        /* --- Header --- */
+        header {
+            border-bottom: 2px solid var(--border);
+            padding-bottom: 2rem;
+            margin-bottom: 3rem;
+            display: flex;
+            justify-content: space-between;
+            align-items: flex-end;
+        }
+        .logo-area h1 {
+            font-size: 1.2rem;
+            text-transform: uppercase;
+            letter-spacing: 2px;
+            color: var(--accent);
+            line-height: 1;
+            display: flex;
+            align-items: center;
+            gap: 10px;
+        }
+        .logo-area a { text-decoration: none; color: inherit; }
+        nav a {
+            color: var(--text);
+            text-decoration: none;
+            margin-left: 1.5rem;
+            font-size: 0.9rem;
+            border-bottom: 1px solid transparent;
+        }
+        nav a:hover { border-bottom: 1px solid var(--accent); }
+        /* --- Blog Post Layout --- */
+        .post-header { margin-bottom: 3rem; }
+        .post-header h2 {
+            font-size: 3rem;
+            line-height: 1.1;
+            margin-bottom: 1rem;
+            font-weight: 800;
+        }
+        .post-meta {
+            font-family: var(--font-mono);
+            color: var(--accent);
+            font-size: 0.9rem;
+            margin-bottom: 2rem;
+        }
+        .post-content {
+            background: var(--surface);
+            border: 1px solid var(--border);
+            padding: 3rem;
+            margin-bottom: 4rem;
+        }
+        .post-content h2 {
+            font-size: 1.8rem;
+            margin: 2.5rem 0 1rem 0;
+            color: var(--accent);
+        }
+        .post-content h2:first-child { margin-top: 0; }
+        .post-content p {
+            margin-bottom: 1.5rem;
+            font-size: 1.1rem;
+            color: var(--text);
+        }
+        .post-content ul {
+            margin-bottom: 1.5rem;
+            padding-left: 1.5rem;
+        }
+        .post-content li { margin-bottom: 0.5rem; font-size: 1.1rem; }
+        .post-content strong { color: #fff; }
+        /* --- Inline code --- */
+        .post-content code {
+            background: #111;
+            border: 1px solid var(--border);
+            padding: 2px 6px;
+            border-radius: 3px;
+            font-size: 0.95em;
+            color: var(--accent);
+        }
+        /* --- Math-style callout box --- */
+        .callout {
+            border-left: 3px solid var(--accent);
+            background: #111;
+            padding: 1rem 1.5rem;
+            margin: 2rem 0;
+            font-family: var(--font-mono);
+            font-size: 0.95rem;
+            color: #ccc;
+        }
+        .callout span {
+            display: block;
+            color: var(--muted);
+            font-size: 0.8rem;
+            margin-bottom: 0.4rem;
+        }
+        /* --- Comparison table --- */
+        .table-wrap { overflow-x: auto; margin: 2rem 0; }
+        table {
+            width: 100%;
+            border-collapse: collapse;
+            font-family: var(--font-mono);
+            font-size: 0.9rem;
+        }
+        th {
+            background: #111;
+            color: var(--accent);
+            padding: 0.75rem 1rem;
+            text-align: left;
+            border: 1px solid var(--border);
+        }
+        td {
+            padding: 0.7rem 1rem;
+            border: 1px solid var(--border);
+            color: var(--text);
+        }
+        tr:nth-child(even) td { background: #111; }
+        .highlight td { color: #fff; font-weight: 600; }
+        /* --- Tags --- */
+        .tags { display: flex; gap: 0.5rem; margin-top: 2rem; flex-wrap: wrap; }
+        .tag {
+            font-family: var(--font-mono);
+            font-size: 0.7rem;
+            padding: 2px 8px;
+            border: 1px solid var(--border);
+            border-radius: 4px;
+            color: var(--muted);
+        }
+        footer {
+            margin-top: 6rem;
+            padding-bottom: 2rem;
+            font-size: 0.8rem;
+            color: var(--muted);
+            text-align: center;
+        }
+        .logo-area {
+            display: flex;
+            align-items: center;
+            gap: 10px;
+            font-weight: bold;
+            font-size: 1.2rem;
+        }
+        @media (max-width: 600px) {
+            .post-header h2 { font-size: 2rem; }
+            .post-content { padding: 1.5rem; }
+            header { flex-direction: column; align-items: flex-start; gap: 1rem; }
+            nav a { margin-left: 0; margin-right: 1rem; }
+        }
+    </style>
+</head>
+<body>
+    <div class="container">
+        <header>
+            <div class="logo-area" style="font-size: 1.5em;">
+                <a href="./index.html"><h1><img src="./image.png" style="height: 2em"> SupraLabs_</h1></a>
+            </div>
+            <nav>
+                <a href="./index.html#news">News</a>
+                <a href="https://huggingface.co/SupraLabs" target="blank">HuggingFace</a>
+                <a href="./index.html#hardware">Hardware</a>
+            </nav>
+        </header>
+        <article>
+            <div class="post-header">
+                <div class="post-meta">// 2026-05-13 | Research</div>
+                <h2>1-Bit Quantization:<br>Shrinking Models to the Bone</h2>
+            </div>
+            <div class="post-content">
+                <p>What if each weight in a neural network could only be <strong>−1, 0, or +1</strong>? That is the premise of 1-bit quantization, and it is more powerful than it sounds. This post breaks down how it works, why it matters, and where it falls short.</p>
+                <h2>What is Quantization?</h2>
+                <p>A standard neural network stores weights as 32-bit or 16-bit floating point numbers. Those floats carry a lot of information, but also a lot of memory cost. <strong>Quantization</strong> is the process of reducing the precision of those numbers to save space and speed up computation.</p>
+                <p>Most production models today use <strong>8-bit (INT8)</strong> or <strong>4-bit (INT4)</strong> quantization. These methods compress weights into integers while still preserving enough numeric range to keep quality high. 1-bit takes this to the extreme: <strong>every single weight is represented by just one bit.</strong></p>
+                <div class="callout">
+                    <span>// memory comparison for a 7B model</span>
+                    FP16 &nbsp;&nbsp;&nbsp;→ ~14 GB<br>
+                    INT8 &nbsp;&nbsp;&nbsp;→ ~7 GB<br>
+                    INT4 &nbsp;&nbsp;&nbsp;→ ~3.5 GB<br>
+                    1-bit &nbsp;&nbsp;→ ~0.9 GB
+                </div>
+                <h2>How Does 1-Bit Actually Work?</h2>
+                <p>Pure binary quantization maps every weight to either <code>+1</code> or <code>−1</code>. The model learns <em>which sign</em> each weight should carry, not its magnitude. During inference, all multiplications become cheap additions and subtractions, no floating point needed.</p>
+                <p>The most important recent work in this space is <strong>BitNet</strong> (Microsoft Research, 2023) and its successor <strong>BitNet b1.58</strong> (2024). BitNet b1.58 uses a ternary scheme: weights are constrained to <code>{−1, 0, +1}</code>. The extra zero value turns many operations into a complete no-op, making inference even faster.</p>
+                <div class="callout">
+                    <span>// bitnet b1.58 weight constraint</span>
+                    W ∈ {−1, 0, +1} &nbsp;— ternary, not strictly binary<br>
+                    activations are still quantized to INT8
+                </div>
+                <p>It's like a dream for weak hardware users.</p>
+                <h2>Training vs Post-Training Quantization</h2>
+                <p>There are two fundamentally different approaches here, and the distinction matters a lot.</p>
+                <ul>
+                    <li><strong>Post-Training Quantization (PTQ)</strong>: take a pre-trained FP16 model and quantize it after the fact. Fast and convenient, but quality degrades — especially below 4 bits.</li>
+                    <li><strong>Quantization-Aware Training (QAT)</strong>: train the model from scratch with quantized weights. The model adapts to its constraints during training. This is how BitNet works — and it is what makes 1-bit viable at all.</li>
+                </ul>
+                <p>Trying to PTQ a standard model down to 1-bit produces catastrophic quality loss. <strong>1-bit only works if the model is trained to be 1-bit from day one.</strong></p>
+                <h2>The Numbers: How Much Do You Lose?</h2>
+                <p>The honest answer: <strong>it depends heavily on model size.</strong> Small models suffer more than large ones. A 125M parameter BitNet model loses noticeably more quality than a 7B BitNet model when compared to their FP16 equivalents.</p>
+                <div class="table-wrap">
+                    <table>
+                        <thead>
+                            <tr>
+                                <th>Format</th>
+                                <th>Bits/weight</th>
+                                <th>Memory (7B)</th>
+                                <th>Speed</th>
+                                <th>Quality loss</th>
+                            </tr>
+                        </thead>
+                        <tbody>
+                            <tr><td>FP16</td><td>16</td><td>~14 GB</td><td>baseline</td><td>none</td></tr>
+                            <tr><td>INT8</td><td>8</td><td>~7 GB</td><td>1.5–2×</td><td>minimal</td></tr>
+                            <tr><td>INT4</td><td>4</td><td>~3.5 GB</td><td>2–4×</td><td>low</td></tr>
+                            <tr class="highlight"><td>1.58-bit</td><td>~1.58</td><td>~0.9 GB</td><td>up to 8×</td><td>moderate*</td></tr>
+                        </tbody>
+                    </table>
+                </div>
+                <p style="font-size:0.85rem; color: var(--muted); margin-top: -1rem;">* at large scale (7B+), quality loss becomes very competitive with INT4.</p>
+                <h2>Why This Matters for Edge and Tiny Models</h2>
+                <p>For us at SupraLabs, 1-bit quantization is an interesting reference point. At sub-1M parameters, the scale of Supra Mini, the quality penalty of 1-bit QAT is severe. The model simply does not have enough capacity to absorb the constraint. <strong>At our scale, every bit of precision counts.</strong></p>
+                <p>Where 1-bit shines is on large models deployed at the edge: think 7B+ models running on phones, embedded devices, or microcontrollers without a GPU. The memory savings are dramatic and the inference speedup from replacing multiplications with additions is real and measurable.</p>
+                <h2>The Catch</h2>
+                <p>1-bit is not a free lunch. The main trade-offs are:</p>
+                <ul>
+                    <li><strong>Requires purpose-built training</strong> = no PTQ shortcut.</li>
+                    <li><strong>It's a 50/50 chance for small models</strong> = it can help our model or kill it lol</li>
+                    <li><strong>Small models suffer</strong> = below ~1B parameters, the quality loss is hard to justify.</li>
+                    <li><strong>Activations still need INT8</strong> = it's not fully binary end-to-end yet.</li>
+                </ul>
+                <h2>How this helps us?(and YOU!)</h2>
+                <p>We, SupraLabs, are going to try every type of experiment, quantization, pruning, distillation, all to create the best models for you!</p>
+                <h2>Final Thought</h2>
+                <p>1Bit quantization is a little bit sensitive area for small models, but we are going to try everything to do it works!</p>
+                <div class="tags">
+                    <span class="tag">#quantization</span>
+                    <span class="tag">#1bit</span>
+                    <span class="tag">#bitnet</span>
+                    <span class="tag">#tinyml</span>
+                    <span class="tag">#research</span>
+                    <span class="tag">#edge-ai</span>
+                    <span class="tag">#open-source</span>
+                </div>
+            </div>
+        </article>
+        <footer>
+            <p class="mono">&copy; 2026 SupraLabs // Built for the community.</p>
+        </footer>
+    </div>
+</body>
+</html>