docs/ADR/006-bf16-native-inference.md

ADR 006: BF16 Native Inference for Local Triage

Status

Approved

Context

The initial implementation of the local specialist (Tier 1) used Unsloth with 4-bit quantization (bitsandbytes) to minimize memory footprint. However, during validation on AMD MI300X hardware, we observed "semantic collapse" and repetitive token generation (e.g., repeating punctuation or phrases indefinitely).

Initial investigations suggested that 4-bit quantization artifacts, combined with the CDNA3 architecture's handling of certain sub-8bit kernels, were degrading inference quality for clinical tasks requiring high precision.

Decision

We will migrate the LocalModelManager from Unsloth/4-bit to native BFloat16 (BF16) precision using the standard transformers and peft libraries.

Given the AMD MI300X's massive 192GB of VRAM, the increased memory requirement of BF16 (~14GB for a 7B model) is well within the hardware's capabilities, even with multiple concurrent agents.

Consequences

• Positive: Improved semantic stability and elimination of repetitive output artifacts.
• Positive: Full compatibility with ROCm native bfloat16 kernels.
• Positive: Higher fidelity clinical reasoning compared to 4-bit quantized versions.
• Neutral: Higher VRAM consumption (approx. 2.5x compared to 4-bit), which is negligible on MI300X.
• Neutral: Slightly slower initialization time as larger weights are loaded into HBM3.