Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>I Dreamed Of NaN And Woke Up To NaN | TinyMemoryLM</title> | |
| <link rel="stylesheet" href="bluesheet.css"> | |
| <link rel="preconnect" href="https://fonts.googleapis.com"> | |
| <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin> | |
| <link href="https://fonts.googleapis.com/css2?family=Geist:wght@400;500;600;700&family=Geist+Mono&display=swap" rel="stylesheet"> | |
| <style> | |
| :root { | |
| --blue-900: #000000; | |
| --blue-800: #0a0a0a; | |
| --blue-700: #111111; | |
| --blue-600: #1a1a1a; | |
| --blue-500: #333333; | |
| --blue-400: #555555; | |
| --blue-300: #777777; | |
| --blue-200: #888888; | |
| --blue-100: #aaaaaa; | |
| --white: #ffffff; | |
| --white-soft: #f5f5f5; | |
| --white-muted: #e0e0e0; | |
| --grid-line: rgba(255, 255, 255, 0.03); | |
| --grid-line-major: rgba(255, 255, 255, 0.06); | |
| --accent: #ededed; | |
| --accent-muted: #888888; | |
| --font-sans: 'Geist', -apple-system, BlinkMacSystemFont, sans-serif; | |
| --font-mono: 'Geist Mono', 'SF Mono', 'Fira Code', monospace; | |
| --container-max: 1100px; | |
| } | |
| * { box-sizing: border-box; margin: 0; padding: 0; } | |
| html { font-size: 16px; scroll-behavior: smooth; } | |
| body { font-family: var(--font-sans); background: var(--blue-900); color: var(--white-muted); line-height: 1.7; -webkit-font-smoothing: antialiased; } | |
| a { color: var(--white); text-decoration: none; transition: color 0.15s ease; } | |
| a:hover { color: var(--accent); } | |
| .container { max-width: var(--container-max); margin: 0 auto; padding: 0 24px; } | |
| nav { position: fixed; top: 0; left: 0; right: 0; z-index: 100; background: rgba(0, 0, 0, 0.85); backdrop-filter: blur(12px); border-bottom: 1px solid var(--blue-600); padding: 16px 0; } | |
| nav .container { display: flex; justify-content: space-between; align-items: center; } | |
| .nav-brand { font-size: 18px; font-weight: 600; color: var(--white); display: flex; align-items: center; gap: 8px; } | |
| .nav-brand span { color: var(--accent); } | |
| .nav-links { display: flex; gap: 32px; } | |
| .nav-links a { font-size: 14px; font-weight: 500; color: var(--blue-200); } | |
| .nav-links a:hover { color: var(--white); } | |
| .post { padding: 140px 0 80px; } | |
| .post-back { display: inline-block; color: var(--blue-200); font-size: 14px; margin-bottom: 32px; } | |
| .post-back:hover { color: var(--accent); } | |
| .post-back::before { content: '← '; } | |
| .post-meta { display: flex; gap: 12px; margin-bottom: 20px; } | |
| .post-date { font-size: 13px; color: var(--blue-200); font-family: var(--font-mono); } | |
| .post-tag { font-size: 11px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.05em; color: var(--white); background: rgba(255, 255, 255, 0.08); padding: 4px 10px; border-radius: 4px; } | |
| .post h1 { font-size: 36px; font-weight: 700; color: var(--white); margin-bottom: 32px; line-height: 1.2; letter-spacing: -0.02em; } | |
| .post-body p { font-size: 17px; line-height: 1.8; margin-bottom: 24px; color: var(--blue-200); } | |
| .post-body p:first-of-type { font-size: 20px; color: var(--white-muted); } | |
| .post-body h2 { font-size: 24px; font-weight: 600; color: var(--white); margin: 48px 0 20px; } | |
| .post-body blockquote { border-left: 3px solid var(--accent); padding: 20px 24px; margin: 32px 0; background: var(--blue-800); border-radius: 0 8px 8px 0; } | |
| .post-body blockquote p { font-size: 16px; font-style: italic; color: var(--blue-200); margin: 0; } | |
| .post-body hr { border: none; height: 1px; background: var(--blue-600); margin: 48px 0; } | |
| .stats-grid { display: grid; grid-template-columns: 1fr 1fr 1fr; gap: 16px; margin: 24px 0; } | |
| .stat-card { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; text-align: center; } | |
| .stat-card .number { font-size: 32px; font-weight: 700; color: var(--accent); font-family: var(--font-mono); } | |
| .stat-card .label { font-size: 13px; color: var(--blue-200); margin-top: 8px; } | |
| .code-block { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; margin: 24px 0; font-family: var(--font-mono); font-size: 13px; overflow-x: auto; } | |
| .code-block .comment { color: var(--blue-200); font-style: italic; display: block; margin-top: 4px; } | |
| .post-footer { margin-top: 48px; padding-top: 32px; border-top: 1px solid var(--blue-600); } | |
| .post-footer p { font-size: 14px; color: var(--blue-200); font-style: italic; margin: 0; } | |
| footer { padding: 40px 0; background: var(--blue-800); border-top: 1px solid var(--blue-600); text-align: center; } | |
| footer p { color: var(--blue-200); font-size: 14px; margin-bottom: 8px; } | |
| footer a { color: var(--blue-200); } | |
| footer a:hover { color: var(--accent); } | |
| @media (max-width: 768px) { .post h1 { font-size: 28px; } .nav-links { display: none; } .stats-grid { grid-template-columns: 1fr; } } | |
| </style> | |
| </head> | |
| <body> | |
| <svg class="scribbles" viewBox="0 0 1440 900" preserveAspectRatio="xMidYMid slice"> | |
| <path d="M100,50 Q150,30 200,60 T300,40 T400,70" fill="none" stroke="white" stroke-width="1"/> | |
| <path d="M800,200 Q850,180 900,210 T1000,190 T1100,220" fill="none" stroke="white" stroke-width="0.8"/> | |
| <path d="M200,700 Q250,680 300,710 T400,690 T500,720" fill="none" stroke="white" stroke-width="0.6"/> | |
| <path d="M1200,400 Q1250,380 1300,410 T1400,390" fill="none" stroke="white" stroke-width="0.7"/> | |
| <path d="M50,400 Q100,380 150,420 T250,400" fill="none" stroke="white" stroke-width="0.5"/> | |
| <circle cx="350" cy="150" r="30" fill="none" stroke="white" stroke-width="0.6"/> | |
| <circle cx="1100" cy="600" r="25" fill="none" stroke="white" stroke-width="0.5"/> | |
| <path d="M600,100 L620,80 L640,100 L660,80" fill="none" stroke="white" stroke-width="0.7"/> | |
| <path d="M1300,750 Q1320,730 1340,760 T1380,740" fill="none" stroke="white" stroke-width="0.5"/> | |
| <path d="M100,800 Q120,780 140,810 T180,790 T220,820" fill="none" stroke="white" stroke-width="0.6"/> | |
| <path d="M700,500 Q720,480 740,510 T780,490 T820,520" fill="none" stroke="white" stroke-width="0.4"/> | |
| <path d="M400,300 C420,280 440,320 460,300 C480,280 500,320 520,300" fill="none" stroke="white" stroke-width="0.5"/> | |
| <path d="M900,700 C920,680 940,720 960,700 C980,680 1000,720 1020,700" fill="none" stroke="white" stroke-width="0.6"/> | |
| <path d="M150,250 Q170,230 190,260 Q210,240 230,270" fill="none" stroke="white" stroke-width="0.4"/> | |
| <path d="M1050,100 Q1070,80 1090,110 Q1110,90 1130,120" fill="none" stroke="white" stroke-width="0.5"/> | |
| <path d="M500,850 C520,830 540,860 560,840 C580,820 600,860 620,840" fill="none" stroke="white" stroke-width="0.4"/> | |
| <path d="M1350,50 Q1370,30 1390,60 T1430,40" fill="none" stroke="white" stroke-width="0.5"/> | |
| <path d="M30,600 Q50,580 70,610 T110,590" fill="none" stroke="white" stroke-width="0.4"/> | |
| </svg> | |
| <nav> | |
| <div class="container"> | |
| <a href="index.html" class="nav-brand"><span>/</span>TinyMemoryLM</a> | |
| <div class="nav-links"> | |
| <a href="index.html">Home</a> | |
| <a href="blog.html">Blog</a> | |
| <a href="status.html">Status</a> | |
| </div> | |
| </div> | |
| </nav> | |
| <main> | |
| <article class="post"> | |
| <div class="container"> | |
| <a href="blog.html" class="post-back">Back to Blog</a> | |
| <header> | |
| <div class="post-meta"> | |
| <span class="post-date">2026-03-30</span> | |
| <span class="post-tag">Training Nightmares</span> | |
| </div> | |
| <h1>I Dreamed Of NaN And Woke Up To NaN</h1> | |
| </header> | |
| <div class="post-body"> | |
| <p>I dreamed about NaN last night. Not a metaphorical NaN. A literal loss: nan in bright red terminal text. I was running through a field of gradients. They were all exploding. I woke up in a panic. I checked my phone. I checked the logs. I needed to know.</p> | |
| <p>The logs said loss: nan. My subconscious was monitoring my training run better than I was. My brain gave up on sleep to tell me the gradients exploded. I am now one with the loss curve. This is not healthy.</p> | |
| <blockquote> | |
| <p>When your dreams start monitoring your terminal output you know you have spent too much time watching progress bars. I have crossed the threshold. I am part machine now.</p> | |
| </blockquote> | |
| <h2>The Patch That Wasnt</h2> | |
| <p>I thought I fixed this. I added gradient clipping. I lowered the learning rate. I added warmup steps. I prayed to the GPU gods. I went to sleep confident. Confidence is the enemy of completed training runs.</p> | |
| <div class="code-block"> | |
| <span class="comment"># My patch notes vs. Reality</span><br> | |
| Patch: Added gradient clipping at 1.0<br> | |
| Patch: Lowered LR to 1e-5<br> | |
| Patch: Added epsilon to denominator<br> | |
| Reality: loss: nan<br> | |
| <span class="comment"># The void always wins.</span> | |
| </div> | |
| <p>Clearly I missed something. Maybe the clipping norm was too high. Maybe the learning rate was still too aggressive. Maybe the data had a bad batch. Maybe the universe hates me. I will never know. I just know it crashed.</p> | |
| <h2>The Status Update</h2> | |
| <div class="stats-grid"> | |
| <div class="stat-card"> | |
| <div class="number">100%</div> | |
| <div class="label">Haiku (Published)</div> | |
| </div> | |
| <div class="stat-card"> | |
| <div class="number">0%</div> | |
| <div class="label">Sonnet (Crashed)</div> | |
| </div> | |
| <div class="stat-card"> | |
| <div class="number">0%</div> | |
| <div class="label">Opus (Dream)</div> | |
| </div> | |
| </div> | |
| <p>Haiku is out. It says "|fdish|||||!@|". It is published. It exists. It is complete. Sonnet is back to zero percent. All that training time. All that electricity. All that heat generated in my room. Gone. Opus remains a concept. A beautiful dream. A dream that requires Sonnet to finish first.</p> | |
| <h2>Restarting Again</h2> | |
| <p>I am restarting Sonnet. Again. This is the third time. Maybe the fourth. I have lost count. Each restart teaches me something. Mostly it teaches me patience. Also it teaches me that NaN is inevitable.</p> | |
| <p>I will lower the learning rate again. I will clip gradients harder. I will check the data for infinities. I will do everything differently. It will probably crash again. I will probably restart again. This is the cycle.</p> | |
| <blockquote> | |
| <p>Training models locally is just debugging with extra steps and more existential dread. I am excellent at debugging. I am excellent at dread.</p> | |
| </blockquote> | |
| <h2>Why I Keep Going</h2> | |
| <p>Haiku exists. That proves it is possible. My tiny models can learn. They can speak. They can output pipe characters with confidence. If Haiku can do it Sonnet can do it. Opus can do it. Eventually.</p> | |
| <p>I have the hardware. I have the code. I have the data. I lack the stability. Stability comes with time. Time I am spending. Lots of time. Too much time. My family misses me. My GPU does not.</p> | |
| <h2>The Dream Continues</h2> | |
| <p>I will dream about NaN tonight too. I know this. My brain has decided this is important. My brain has decided loss curves are more important than sleep. My brain might be right. Loss curves are tangible. Sleep is temporary. NaN is eternal.</p> | |
| <p>Tomorrow I will check the logs again. I will fear the NaN. I will see the NaN. I will restart. I will try again. This is my life now. This is what I chose. This is fine.</p> | |
| <h2>Final Thoughts</h2> | |
| <p>Haiku is at 100 percent. Sonnet is at 0 percent. Opus is at 0 percent. I am at 100 percent exhaustion. We are all at different stages of completion. We are all trying our best.</p> | |
| <p>If you train models locally you understand this pain. If you do not train models locally you should sleep well tonight. Enjoy your rest. I will be here watching gradients explode.</p> | |
| <hr> | |
| </div> | |
| <footer class="post-footer"> | |
| <p>Current status: Sonnet restarted at 0%. Haiku published at 100%. Opus waiting. Me dreaming of NaN. Will check logs in 5 minutes.</p> | |
| </footer> | |
| </div> | |
| </article> | |
| </main> | |
| <footer> | |
| <div class="container"> | |
| <p>Built with curiosity over compute</p> | |
| <p>TinyMemoryLM by AILAY | 2026</p> | |
| </div> | |
| </footer> | |
| </body> | |
| </html> |