audreyt/nemotron-cascade-2-attn-repeat-L5-GGUF

Nemotron-Cascade-2-30B-A3B with Attention Repeat (Layer 5) - GGUF Q4_K_M

A modified GGUF of NVIDIA Nemotron-Cascade-2-30B-A3B that applies the RYS (Repeat Your Steps) layer duplication technique from shi3z/nemotron-cascade-2-attn-repeat-L5, but in GGUF format for use with ollama and llama.cpp.

What changed

Layer 5 (the first GQA Attention layer) is physically duplicated in the GGUF. The model goes from 52 to 53 layers, with layers 5 and 6 being identical attention blocks. All subsequent layers are shifted by 1.

This is the GGUF equivalent of shi3z's BlockRepeatWrapper approach. No weights were modified -- only duplicated.

Per-layer metadata

The GGUF includes correct 53-element per-layer arrays for attention.head_count_kv and feed_forward_length, so ollama correctly classifies each layer (Mamba2 / GQA Attention / MoE).

Architecture

53 layers total (was 52):

• Mamba-2: 23 layers (SSM-based, sequential)
• MoE: 23 layers (128 routed experts + 1 shared, top-6)
• GQA Attention: 7 layers (was 6) at positions 5, 6, 13, 20, 27, 34, 43

Parameters: 31.6B total, ~3B active per token. Quantization: Q4_K_M.

Performance

Benchmarked on NVIDIA DGX Spark (GB10 Blackwell, 120GB unified LPDDR5X):

Metric	Base cascade-2	attn-repeat-L5
Generation speed	74.5 tok/s	62.5 tok/s
Prompt eval	81.7 tok/s	299.0 tok/s
Model size	23 GB	22.6 GB
VRAM loaded	26 GB	27 GB

~16% slower generation, ~3.7x faster prompt processing. Per shi3z's benchmarks, +6.7 percentage points on BBH-style reasoning.

Usage with ollama

# Download and create model
ollama create nemotron-cascade-2-attn-repeat -f Modelfile

# Run
ollama run nemotron-cascade-2-attn-repeat "Hello!"

Modelfile:

FROM nemotron-cascade-2-attn-repeat-L5-Q4_K_M.gguf
TEMPLATE {{ .Prompt }}
RENDERER nemotron-3-nano
PARSER nemotron-3-nano
PARAMETER temperature 1
PARAMETER top_p 0.95

How this GGUF was built

The GGUF was produced by a Python script that:

1. Reads the original nemotron-cascade-2 Q4_K_M GGUF from ollama
2. Duplicates all 5 tensors of block 5 (attn_q, attn_k, attn_v, attn_output, attn_norm) as block 6
3. Shifts all blocks >= 6 up by 1
4. Updates block_count from 52 to 53
5. Rebuilds the per-layer attention.head_count_kv and feed_forward_length arrays with the inserted entry

Source: built on DGX Spark running ollama 0.20.0, using gguf-py 0.18.0.

Credits

• NVIDIA for Nemotron-Cascade-2-30B-A3B
• shi3z for the attn-repeat-L5 discovery and RYS technique
• Built with assistance from Claude Code (Anthropic)

License

NVIDIA Open Model License (inherited from base model).