Fine-Tuning xLSTM-7B on a budget: An Experimental Journey with Chat Templates
•
1
AI & ML interests
Building practical, ethical and sustainable AI/ML projects.
Recent Activity
View all activity
Articles
mrs83
updated a
collection about 4 hours ago
Post
38
While the 
flwrlabs community gathered in London for their Summit, we released ethicalabs/FlowerTune-Echo-DSRN-114M-Finance-PEFT onto the Flower Hub: a federated PEFT adapter for financial sentiment, built on a novel architecture called Echo-DSRN, a project I started working on 2 years ago.
The core problem we set out to solve: financial data on ledgers, earnings calls, tick streams, blows up the memory footprint of standard Transformers.
KV-Cache scaling makes federated training on the edge increasingly difficult. You cannot preserve data privacy if your decentralized nodes keep running out of memory.
Echo-DSRN addresses this at the architectural level. It uses a dual recurrent state design: a GRU fast path for short-range dynamics, and a surprise-gated slow memory whose write intensity is modulated by prediction error.
The result is O(1) memory regardless of context length. Runs on CPU, AMD ROCm, Apple MPS, NVIDIA GPUs.
Combined with the Flower federated framework, financial institutions can now run local fine-tuning on proprietary data without it ever leaving their infrastructure.
Results on standard financial sentiment benchmarks:
→ FPB: 70.2%
→ TFNS: 70.2%
→ FIQA: 63.8%
This is a 114M baseline. The next step is scaling.
The surprise gating mechanism independently converged on what
google described in their Titans paper. No working open implementation existed. This one does.
Flower Hub: https://flower.ai/apps/mrs83/echo-dsrn-114m-finance
Hugging Face: ethicalabs/FlowerTune-Echo-DSRN-114M-Finance-PEFT
The core problem we set out to solve: financial data on ledgers, earnings calls, tick streams, blows up the memory footprint of standard Transformers.
KV-Cache scaling makes federated training on the edge increasingly difficult. You cannot preserve data privacy if your decentralized nodes keep running out of memory.
Echo-DSRN addresses this at the architectural level. It uses a dual recurrent state design: a GRU fast path for short-range dynamics, and a surprise-gated slow memory whose write intensity is modulated by prediction error.
The result is O(1) memory regardless of context length. Runs on CPU, AMD ROCm, Apple MPS, NVIDIA GPUs.
Combined with the Flower federated framework, financial institutions can now run local fine-tuning on proprietary data without it ever leaving their infrastructure.
Results on standard financial sentiment benchmarks:
→ FPB: 70.2%
→ TFNS: 70.2%
→ FIQA: 63.8%
This is a 114M baseline. The next step is scaling.
The surprise gating mechanism independently converged on what
Flower Hub: https://flower.ai/apps/mrs83/echo-dsrn-114m-finance
Hugging Face: ethicalabs/FlowerTune-Echo-DSRN-114M-Finance-PEFT
mrs83
updated a
model about 7 hours ago
mrs83
published a
model about 7 hours ago
mrs83
updated 3
models about 19 hours ago
mrs83
published a
model about 19 hours ago
Apply for a GPU community grant: Personal project
#1 opened 1 day ago
by 
mrs83
mrs83
updated 3
models 1 day ago
mrs83
published a
model 1 day ago
Echo-DSRN-114M-Base v0.1.1
1
#4 opened 1 day ago
by mrs83
mrs83
published a
model 4 days ago
🚨 BREAKING: Echo-DSRN-114M downloads Pizza via HTTP
#1 opened 5 days ago
by mrs83
Training stability on AMD ROCm (distributed backend, gloo vs nccl)
#3 opened 7 days ago
by mrs83
mrs83
updated a
Space 7 days ago
Post
134
Echo-DSRN-114M: A Constant-Memory O(1) Semantic Compressor
While traditional models target general conversational reasoning, Echo-DSRN(N) is a specialized structural prototype.
It is a dual-state recurrent neural network engineered strictly for low-latency semantic compression and continuous text streaming with a permanent O(1) memory footprint.
⚙️ Echo-DSRN (114M Parameters) manages context via continuous structural compression:
- 8 Layers | 512 Hidden Dim
- Transformer Fast State + DSRN/GRU Recurrent Slow State + Surprise Gating
Initial pre-training on a single AMD Instinct MI300X, followed by localized refinement across AMD Radeon PRO GPUs and an AMD Ryzen AI Max+ 395 (Strix Halo).
🖥️ A Hugging Face Space showcasing the architecture is currently running on the free shared CPU tier.
- The Compressor: Ingest a long document and crush it into a fixed 2048-dimensional .npy state vector.
- Vector Similarity: Upload two compressed .npy states to instantly calculate cosine similarity for ultra-lightweight RAG pre-filtering.
- The CPU Streamer: Continuous, fluent text generation running on raw CPU compute.
⚠️ Disclaimer: This is a structural prototype. It has internalized formatting and conversational syntax, but it possesses zero world knowledge. It will confidently hallucinate. Use it for streaming transcription, style mimicry, and local semantic hashing, not for factual reasoning.
Try the CPU Demo: ethicalabs/Echo-DSRN-114M
Try the Model: ethicalabs/Echo-DSRN-114M
While traditional models target general conversational reasoning, Echo-DSRN(N) is a specialized structural prototype.
It is a dual-state recurrent neural network engineered strictly for low-latency semantic compression and continuous text streaming with a permanent O(1) memory footprint.
⚙️ Echo-DSRN (114M Parameters) manages context via continuous structural compression:
- 8 Layers | 512 Hidden Dim
- Transformer Fast State + DSRN/GRU Recurrent Slow State + Surprise Gating
Initial pre-training on a single AMD Instinct MI300X, followed by localized refinement across AMD Radeon PRO GPUs and an AMD Ryzen AI Max+ 395 (Strix Halo).
🖥️ A Hugging Face Space showcasing the architecture is currently running on the free shared CPU tier.
- The Compressor: Ingest a long document and crush it into a fixed 2048-dimensional .npy state vector.
- Vector Similarity: Upload two compressed .npy states to instantly calculate cosine similarity for ultra-lightweight RAG pre-filtering.
- The CPU Streamer: Continuous, fluent text generation running on raw CPU compute.
⚠️ Disclaimer: This is a structural prototype. It has internalized formatting and conversational syntax, but it possesses zero world knowledge. It will confidently hallucinate. Use it for streaming transcription, style mimicry, and local semantic hashing, not for factual reasoning.
Try the CPU Demo: ethicalabs/Echo-DSRN-114M
Try the Model: ethicalabs/Echo-DSRN-114M
mrs83
published a
Space 8 days ago