docs/true-ternary/TRUE-TERNARY-REFACTOR11.md

# TRUE-TERNARY-REFACTOR11

Date: 2026-05-19

## Scope

Final readiness pass for the platform-style ARB tree:

- Hit the exact 1.5B logical ternary weight target.
- Keep imported DINOv2/Moonshine sidecars frozen and int8-quantized.
- Make graph/VQ/MoE wiring consistent with the current config.
- Repair strict ternary training, standalone modality training, and LoRA finetuning entrypoints.
- Verify CUDA kernel paths and smoke-test training without restoring hidden FP master weights.

## Architecture Changes

- `CODEBOOK_SIZE` is now `34108`.
- `MOE_SHARED_INTER` is now `21216`.
- Full multimodal ARB audit now lands at exactly `1,500,000,000` logical ternary weights.
- The exact count includes MoE shared-width RMS ternary state; the final VQ codebook size compensates for that extra per-width ternary state.
- `ARBModel` now wires graph vocab size from `MultimodalVQBridge.total_codebook_size` instead of the old hardcoded `16384`.
- MemGram and ConvVQ now use config constants instead of duplicated literals.
- Supervised calls with `targets` always use `ByteHead`, even under `model.eval()`, so eval loss cannot accidentally route into video/audio heads.

## Kernel And Runtime Changes

- Large `TernaryGraph` instances now use an active-code path when `total_vocab_size > 4096`.
- This avoids projecting every VQ codebook node on each forward. The full graph still keeps the existing Triton aggregate/gather kernels for small/full graph tests.
- The 32-expert/top-4 MoE now disables dense all-expert dispatch (`dense_dispatch_max_tokens=0`) because the dense path is too expensive at 1.5B scale.
- `TernaryScaleTensor.forward()` now forces a detached CUDA input to require grad while grad mode is active. This lets ternary weights after frozen sidecars train correctly.
- `VideoHead` initial latent now requires grad in training so `cross_attn_q` can receive ternary gradient signals.
- `TalkerHead.token_logits()` was added so audio training can use cross entropy on logits instead of non-differentiable argmax tokens.

## Training Fixes

- Replaced stale `arbitor/train.py` with a current ARB trainer:
  - strict ternary updates by default,
  - no bitsandbytes dependency,
  - correct trigram target alignment (`x[:, 3:]`),
  - optional `--no-save` for smoke tests,
  - explicit sidecar/modal flags,
  - no optimizer when there are no trainable float params.
- `training/text.py` now always runs ternary state updates after backward.
- `training/audio.py` now trains `AudioSequencer -> TalkerHead.token_logits()` against `AudioVQEncoder` targets.
- `training/vision.py` now uses VQ commitment to train image-side ternary projection state and avoids building the full MoE when it is not used.
- `training/diffusion.py` now feeds relational tokens directly into `VideoHead` and avoids the unused full MoE.
- LoRA finetuners now create checkpoint directories, use trigram-aligned targets, expose `--max-moe-iters`, and default to `1` iteration for local 8GB runs.
- Audio input normalization now accepts `[T]`, `[B, T]`, and `[B, 1, T]`.

## Verification

Passed:

- `python -m compileall -q arbitor training testing/model/test_arb.py testing/test_tscale.py`
- `python -m pytest -q testing/test_tscale.py -k "cuda_triton_correctness_update_E or cuda_triton_tscale_path"`
- Custom CUDA kernel smoke for graph aggregate, graph gather/add, MoE dense combine, and video denoise.
- Full multimodal audit:
  - logical ternary weights: `1,500,000,000`
  - ternary training state: `1956.05 MB`
  - trainable float params: `0`
  - frozen sidecar params: `318.80 MB`
  - graph vocab: `58684`
  - DINOv2 int8: `True`
  - Moonshine int8: `True`
- Active graph CUDA train smoke passed with VQ+graph and no MoE.
- `arbitor.train` strict ternary CUDA smoke passed with 1.5B MoE path.
- Standalone modality smokes passed:
  - `training/audio.py --steps 1 --batch 1`
  - `training/vision.py --steps 1 --batch 1`
  - `training/diffusion.py --steps 1 --batch 1`
- LoRA text finetune smoke passed:
  - `training/finetuning/text.py --steps 1 --batch 1 --accum 1 --ctx 4 --lora-rank 1 --max-moe-iters 1`
- pig-vae load now passes after installing `diffusers` into the user site.
- The local `.safetensors` checkpoint must load through `AutoencoderKLWan.from_single_file`; direct `load_state_dict(strict=False)` had 194 missing and 194 unexpected keys and was silently leaving random VAE weights.
- pig-vae int8 load smoke:
  - inner module: `AutoencoderKLWan`
  - quantized int8: `True`
  - trainable float params: `0`
- pig-vae encode/decode smoke:
  - input video: `[1, 3, 4, 64, 64]`
  - latents: `[1, 16, 1, 8, 8]`
  - reconstruction: `[1, 3, 1, 64, 64]`

## Remaining Constraints

- Full 1.5B MoE training is functionally correct but still slow on the RTX 4060 class GPU. One strict smoke step with MoE took about 77 seconds after the sparse-dispatch change.
- The active graph path is the practical path for the 58k VQ vocabulary. A future native CUDA graph kernel should fuse active-node projection, neighbor selection, hop update, and pooling.
- LoRA finetuning uses float adapter parameters by design; strict base ternary state remains frozen in that path.
- Diffusers emits a non-fatal warning about missing `torchao` tensor support in this environment. The local safetensors pig-vae checkpoint loaded and ran without torchao.
- `pyproject.toml` now requires `diffusers>=0.38.0` for the `diffusers` and `video` extras because older versions may not expose `AutoencoderKLWan`.