sato2ru/dream-transformer

DreamTransformer

A language model that generates mid-scene with no start token and no narrative entry point.

Architecture

• 4 Transformer blocks, d_model=128, 4 heads
• RoPE positional encoding — local order awareness, no global anchor
• Random latent injection replaces BOS token
• Local window attention (64 tokens) — simulates suppressed prefrontal coherence
• Trained on DreamBank (~21K dream reports)
• Parameters: 7.2M | Val loss: 4.2456 | Val PPL: 69.8

Quick start

import torch
ckpt = torch.load('dream_transformer.pt')
model.load_state_dict(ckpt['model_state'])
text = dream(model, temperature=1.3)