DOCUMENTATION.md

Scientific Documentation: QLoRA Fine-Tuning Pipeline for Multi-Standard Telecom Intent Translation

Repository: nraptisss/intent-translation-training
Dataset: nraptisss/TMF921-intent-to-config-augmented
Task: Natural language network intent → structured JSON configuration across 6 telecom standards

---

Table of Contents

1. Problem Statement & Motivation
2. Related Work & Positioning
3. Dataset Description & Audit
4. Base Model Selection
5. Quantization Strategy: 4-bit NormalFloat (NF4)
6. Parameter-Efficient Fine-Tuning: LoRA
7. Training Configuration
   • 7.1 Optimizer: AdamW with Decoupled Weight Decay
   • 7.2 Learning Rate Schedule: Linear Warmup + Cosine Decay
   • 7.3 Mixed Precision: BFloat16
   • 7.4 Flash Attention 2
   • 7.5 Gradient Checkpointing
   • 7.6 Sequence Length
   • 7.7 Batch Size & Gradient Accumulation
   • 7.8 Assistant-Only Loss Masking
   • 7.9 Data Format: ChatML
   • 7.10 Training Epochs
   • 7.11 Checkpoint Strategy & Early Stopping
8. Evaluation Methodology
   • 8.1 JSON Syntactic Validity
   • 8.2 Structural Schema Correctness
   • 8.3 KPI Field Extraction Accuracy
   • 8.4 Adversarial Robustness
   • 8.5 Greedy Decoding for Evaluation
9. VRAM Budget Analysis
10. Software Stack & Reproducibility
11. Limitations & Threats to Validity
12. References

---

1. Problem Statement & Motivation

Intent-Based Networking (IBN) enables network operators to express high-level business objectives in natural language (e.g., "Deploy a URLLC slice for autonomous driving in Munich with 1ms latency and 99.999% reliability"), which must be translated into spec-compliant machine-readable configurations across heterogeneous telecom standards. This translation spans multiple abstraction layers: from business-level intent APIs (TMF921, ETSI ZSM) through network management functions (3GPP TS 28.312) to radio access network policies (O-RAN A1/E2) and external exposure APIs (CAMARA/NEF).

The core challenge is what Deng et al. [1] term the "Structure Gap" — the fundamental mismatch between probabilistic language model generation and the deterministic, schema-constrained configurations required by telecom systems. A single misplaced field, incorrect nesting level, or hallucinated KPI value renders the output unusable by downstream orchestration systems.

Why fine-tuning is necessary. ORION [2] demonstrated that large proprietary models (GPT-5, Claude Opus 4.5) achieve 100% policy creation success on intent translation tasks, but: (a) they evaluated only 100 intents targeting a single standard (CAMARA NetworkSliceBooking), (b) they reported no open-source fine-tuned alternative, (c) the per-intent cost ranges from $0.19–$29.68, making production deployment impractical. NEFMind [3] showed that QLoRA fine-tuning of the 2.7B Phi-2 model on just 765 NEF API records achieved 98–100% accuracy — demonstrating that domain-specific fine-tuning of small open models can match or exceed proprietary model performance at negligible marginal cost.

Our contribution. This pipeline produces the first open-source fine-tuned model for multi-standard intent-to-configuration translation, covering 6 telecom standards, 8 lifecycle operations, and adversarial intent rejection — trained on a dataset 250× larger than ORION's evaluation set [2] and 55× larger than NEFMind's training corpus [3].

---

2. Related Work & Positioning

Work	Approach	Standards	Dataset Size	Model	Key Limitation
ORION [2]	LLM + MCP tool-use	CAMARA only	100 eval intents	GPT-5/Claude (proprietary)	No fine-tuning; single standard; no lifecycle
Hermes [4]	Chain-of-LLM-agents	YAML blueprints	Not disclosed	GPT-4o (proprietary)	82.5% on power control; no JSON schemas
NEFMind [3]	QLoRA fine-tuning	CAMARA/NEF	765 records	Phi-2 (2.7B)	Single API; no multi-standard coverage
LLMs meet Slicing [5]	Multi-agent framework	3GPP CSMF/NSMF	Conceptual	GPT-4 (proprietary)	No implementation or evaluation
TelecomGPT [6]	CPT + SFT + DPO	General telecom	Proprietary corpus	LLaMA-based	General QA, not intent translation
ORANSight-2.0 [7]	RAG + instruction tuning	O-RAN specs	13K questions	18 models tested	MCQ format, not config generation
This work	QLoRA SFT	6 standards + lifecycle + adversarial	41,815 samples	Qwen3-8B (open)	See §11

Our pipeline directly addresses three gaps identified in the literature survey by Mahi et al. [8]: (1) no open-source fine-tuned model exists for multi-standard intent translation, (2) lifecycle management beyond "create" is unexplored, and (3) adversarial robustness of intent translation systems has not been evaluated.

---

3. Dataset Description & Audit

Dataset: nraptisss/TMF921-intent-to-config-augmented — 41,815 samples (39,294 train / 2,521 test).

3.1 Coverage

Dimension	Values
Target standards	TMF921, 3GPP TS 28.312 (intent_3gpp), CAMARA NetworkSliceBooking, ETSI ZSM, O-RAN A1 Policy, 3GPP O1 NRM
Slice types	eMBB, URLLC, mMTC, V2X, MPS, HMTC
Sectors	18 (automotive, healthcare, manufacturing, energy, …)
Use cases	147 distinct
Regions	55
Lifecycle operations	8: activate, modify, suspend, resume, terminate, scale, monitor, report (1,552 samples, TMF921 only)
Adversarial samples	141: ambiguous (58), out_of_scope (43), contradictory (40)

3.2 Token Statistics

Metric	Value
Total tokens (approx.)	~27M
Mean tokens per sample	~690
Median tokens per sample	~650
95th percentile	~1,200
99th percentile	~2,800
Maximum	~3,900

These statistics motivate the choice of max_length=4096 (§7.6).

3.3 Format

Each sample follows the ChatML conversational format with exactly 3 turns:

{
  "messages": [
    {"role": "system", "content": "You are a telecom intent translator..."},
    {"role": "user", "content": "Deploy a URLLC slice for..."},
    {"role": "assistant", "content": "{\"id\": \"intent-...\", ...}"}
  ]
}

The system prompt specifies the target standard and expected output schema. The user message contains the natural language intent. The assistant message contains the ground-truth JSON configuration. This format is directly compatible with the TRL SFTTrainer [9] via the apply_chat_template function, requiring no preprocessing.

3.4 Augmentation Over Base Dataset

The augmented dataset (41,815 samples) extends the base TMF921-intent-to-config-25k (25,000 samples) with:

• 16,819 new samples (42.8% of augmented-only content) across additional use cases and regions
• 1,552 lifecycle operation samples (entirely new — absent from base dataset)
• 141 adversarial samples (entirely new)

3.5 Known Data Quality Issues

Adversarial response homogeneity. All 58 ambiguous samples map to an identical JSON response; all 43 out-of-scope samples to another identical JSON; all 40 contradictory to a third. This means the model may learn to classify adversarial categories without learning to generate contextually specific rejection explanations. For production deployment, adversarial responses should be diversified with intent-specific clarification requests.

Lifecycle operation coverage. Lifecycle operations (activate, modify, suspend, resume, terminate, scale, monitor, report) are only represented for the TMF921 standard. Cross-standard lifecycle management (e.g., modifying a CAMARA booking or suspending an A1 policy) is not covered.

---

4. Base Model Selection

Model: Qwen3-8B [10] — an 8.2 billion parameter dense decoder-only transformer from Alibaba's Qwen team.

4.1 Architecture

Parameter	Value
Layers	36
Hidden size	4,096
Intermediate size (FFN)	12,288
Query attention heads	32
Key-Value attention heads	8 (GQA, 4:1 ratio)
Head dimension	128
Positional encoding	RoPE (θ = 1,000,000)
Activation	SwiGLU [11]
Normalization	RMSNorm (pre-norm, ε = 10⁻⁶)
Max context length	128K tokens
Vocabulary size	151,936 (byte-level BPE)
Pre-training data	36 trillion tokens, 119 languages

4.2 Justification

The choice of Qwen3-8B over alternatives is motivated by several factors:

1. Size-performance trade-off. At 8B parameters, the model fits comfortably in 4-bit quantization on a single 48GB GPU (~5.5 GB for weights), leaving ample headroom for training activations and optimizer states. This is the sweet spot identified by NEFMind [3] (which fine-tuned 2.7B Phi-2) and TelecomGPT [6] (which used LLaMA-based 7-8B models).

2. Grouped Query Attention (GQA). The 4:1 Q-to-KV head ratio reduces KV cache memory by 4× during inference compared to standard multi-head attention [12], enabling efficient batch inference during evaluation on the 2,521 test samples.

3. Native ChatML support. Qwen3's tokenizer natively supports the <|im_start|> / <|im_end|> ChatML delimiters used in our dataset, eliminating tokenizer modification or special token addition.

4. 128K context window. While our samples are ≤4K tokens, the long-context pre-training with RoPE θ = 10⁶ means the model has robust positional encoding even at our maximum sequence length — no extrapolation is needed.

5. Multilingual pre-training. With 119 languages in pre-training, the model has exposure to diverse naming conventions, region-specific terminology, and technical vocabulary that appears in multi-region telecom intents (55 regions in our dataset).

6. Dual thinking mode. Qwen3 introduces <think>/</think> tokens enabling chain-of-thought reasoning. While we disable thinking mode for fine-tuning (structured output should be direct), this capacity for internal reasoning may benefit intent disambiguation.

7. Open weights under Apache 2.0. Unlike proprietary models used in ORION [2] and Hermes [4], the open license enables reproducible research, on-premise deployment, and community extension.

4.3 Pre-training Composition Relevance

Qwen3 was pre-trained on 36 trillion tokens including substantial synthetic data from Qwen2.5 (textbooks, QA, instructions, code) [10, §3.1]. This is particularly relevant because:

• Code generation training provides structural understanding of JSON syntax, nesting, and schema compliance
• Instruction-following pre-training aligns with our SFT task format
• STEM domain data includes technical terminology relevant to telecommunications

---

5. Quantization Strategy: 4-bit NormalFloat (NF4)

Implementation: BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)

5.1 NF4 Data Type

NormalFloat 4-bit (NF4) quantization was introduced in QLoRA [13] as an information-theoretically optimal data type for neural network weights. The construction exploits the empirical observation that pre-trained neural network weights follow approximately zero-centered normal distributions:

1. Estimate the 2^(k+1) quantiles of a standard normal distribution N(0,1)
2. Normalize the resulting quantile values into the range [-1, 1]
3. Quantize input tensors by normalizing via absolute maximum rescaling, then mapping to the nearest NF4 value

NF4 uses asymmetric quantization to guarantee an exact zero representation: 2^(k-1) bins for negative values and 2^(k-1)+1 bins for positive values, unified with duplicate zero removal. This ensures equal expected number of values per quantization bin, minimizing information loss — hence the "information-theoretically optimal" characterization [13, §3].

Compared to the standard FP4 data type, NF4 achieves lower quantization error on normally distributed weights because its bin boundaries are placed at the quantiles of the weight distribution rather than at uniform intervals.

5.2 Double Quantization

Double quantization (DQ) [13, §3] reduces the memory overhead of quantization constants:

• First quantization: Block size 64, producing one FP32 scaling constant per 64 parameters → 0.5 bits/parameter overhead
• Second quantization: The FP32 constants are mean-centered (for symmetry) and then quantized to FP8 with block size 256 → 0.127 bits/parameter overhead
• Net savings: 0.373 bits/parameter, or approximately 3 GB for a 65B model [13]

For Qwen3-8B (8.2B parameters), double quantization saves approximately 380 MB of GPU memory.

5.3 Compute Dtype: BFloat16

All dequantized computations (matrix multiplications, attention, FFN) are performed in BFloat16 precision. The QLoRA forward pass is [13, Eq. 5]:

Y^BF16 = X^BF16 · doubleDequant(c1^FP32, c2^NF4, W^NF4) + X^BF16 · L1^BF16 · L2^BF16

Where the first term is the frozen base model computation (dequantized from NF4 to BF16 on-the-fly) and the second term is the LoRA adapter computation (always in BF16). The base model weights are never stored in BF16 — they remain in NF4 (4 bits per parameter) with dequantization performed per-block during the forward pass.

5.4 Memory Savings

Precision	Memory for 8.2B params	Ratio
FP32	32.8 GB	1×
FP16/BF16	16.4 GB	2×
INT8	8.2 GB	4×
NF4 + DQ	~4.6 GB	~7×

This reduction is what enables fine-tuning an 8B model on a single consumer/workstation GPU.

---

6. Parameter-Efficient Fine-Tuning: LoRA

Implementation: LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules="all-linear")

6.1 Method

Low-Rank Adaptation (LoRA) [14] freezes all pre-trained weights W₀ ∈ ℝ^(d×k) and injects trainable low-rank decomposition matrices:

h = W₀x + ΔWx = W₀x + BAx

where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with rank r ≪ min(d, k). Matrix A is initialized from a random Gaussian distribution; B is initialized to zero, ensuring ΔW = BA = 0 at training start — the model begins from the exact pre-trained weights.

The output is scaled by α/r, where α (lora_alpha) is a constant. This scaling controls the effective magnitude of the LoRA update relative to the pre-trained weights.

6.2 Rank Selection: r = 32

The rank r controls the expressiveness of the adaptation:

• r = 4–8: Sufficient for simple classification or style transfer [14, Table 5]
• r = 16–32: Recommended for complex structured generation tasks
• r = 64+: Diminishing returns; approaches full fine-tuning behavior

We select r = 32 based on:

1. Task complexity. Intent-to-configuration translation requires generating deeply nested JSON structures with precise field names and values across 6 different schemas — more complex than the downstream NLP tasks (MNLI, SST-2, MRPC) evaluated in the original LoRA paper [14] where r = 4 sufficed.

2. QLoRA best practices. Dettmers et al. [13, §4] found that "LoRA rank has diminishing returns above r=16 for most tasks but complex tasks benefit from r=32–64." They specifically recommend r ≥ 16 for structured generation.

3. Trainable parameter count. With r = 32 on all linear layers of Qwen3-8B:

   • Trainable parameters: ~160M (1.95% of total)
   • Adapter size on disk: ~320 MB (BF16) or ~160 MB (FP16)
   • This is feasible for our VRAM budget (§9)

6.3 Alpha/Rank Ratio: α = 64 (α/r = 2.0)

The scaling factor α/r controls the effective learning rate of the LoRA update. Setting α = r gives a scaling factor of 1.0 (the LoRA paper's default). We use α = 2r (scaling factor = 2.0) following the convention established by QLoRA [13] and widely adopted in the community:

• Higher α/r amplifies updates, compensating for the fact that 4-bit quantized base model gradients have slightly higher noise than FP16/FP32 gradients
• Hu et al. [14]: "tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately" — so α = 2r with lr = 1e-4 is effectively equivalent to α = r with lr = 2e-4
• The QLoRA paper [13] empirically validated that α = 2r performs well across Guanaco instruction-tuning benchmarks

6.4 Target Modules: "all-linear"

We apply LoRA to all linear projection layers in the model (Q, K, V, O attention projections + gate, up, down FFN projections), rather than only attention projections (Q+V) as in the original LoRA paper [14].

Justification:

1. QLoRA finding [13, §4]: "Applying LoRA to all linear transformer block layers is important for achieving the best performance." Adapting only attention projections leaves the FFN layers frozen — problematic when the model needs to learn new output distributions (i.e., telecom-specific JSON schemas not seen in pre-training).

2. Empirical validation in prior telecom work. NEFMind [3] applied QLoRA to all linear layers of Phi-2 and achieved 98-100% accuracy. ORANSight-2.0 [7] similarly applied full-layer LoRA in its best-performing configurations.

3. Memory cost. Applying LoRA to all linear layers increases trainable parameters from 0.5% (Q+V only) to ~2% of total — but since these are low-rank (r=32) and trained in BF16, the absolute memory increase is modest (200 MB additional for optimizer states).

6.5 LoRA Dropout: 0.05

A dropout rate of 0.05 is applied to the LoRA A matrix outputs during training. This provides mild regularization against overfitting on the training distribution, which is particularly relevant given:

• The dataset has some distribution imbalance (e.g., eMBB and URLLC samples dominate over MPS and HMTC)
• Adversarial samples are limited (141 out of 41,815 = 0.34%)

The value 0.05 follows QLoRA [13] defaults and is consistent with standard practice for LoRA fine-tuning [13, 14].

6.6 Bias: "none"

No bias terms are trained, following the LoRA paper [14] recommendation: "We find that only adapting the attention weights and freezing the MLP/biases is most efficient." Training bias terms adds minimal capacity but introduces instability in quantized training due to the interaction between quantization noise and bias gradients.

---

7. Training Configuration

Implementation: SFTConfig(...) from TRL [9] with the following parameters.

7.1 Optimizer: AdamW with Decoupled Weight Decay

Configuration: optim="adamw_torch" (default), weight_decay=0.01

The AdamW optimizer [15] decouples weight decay from the adaptive learning rate:

θ_t = θ_{t-1} − α · (m̂_t / (√v̂_t + ε)) − α · λ · θ_{t-1}

where m̂_t and v̂_t are bias-corrected first and second moment estimates, α is the learning rate, and λ is the weight decay coefficient.

Why AdamW over Adam + L2. Loshchilov & Hutter [15] demonstrated that in standard Adam with L2 regularization, the penalty |θ|² is scaled by the inverse of the adaptive learning rate √v̂_t — making the effective regularization strength layer-dependent and inconsistent. AdamW applies weight decay directly to the parameters, independent of the gradient statistics. This is particularly important for QLoRA training where different LoRA modules may have very different gradient magnitudes.

Weight decay = 0.01. This is the canonical value for LLM fine-tuning [13, 15], providing mild regularization without constraining the model's capacity to learn new structured output distributions. Higher values (0.05–0.1) risk underfitting on the minority classes (lifecycle operations, adversarial samples).

7.2 Learning Rate Schedule: Linear Warmup + Cosine Decay

Configuration: lr=1e-4, lr_scheduler_type="cosine", warmup_steps=100

The learning rate follows a two-phase schedule:

1. Linear warmup (steps 0–100): LR linearly increases from 0 to 1e-4
2. Cosine decay (steps 100–~3,690): LR decreases following:

η_t = η_min + 0.5 · (η_max − η_min) · (1 + cos(π · T_cur / T_total))

as introduced in SGDR [16], here used without restarts (single cosine cycle).

Learning rate = 1e-4. For LoRA fine-tuning, learning rates are typically 5–100× higher than full fine-tuning rates [13, 14]. The standard range is 1e-4 to 3e-4. We use 1e-4 as a conservative starting point, combined with the α/r = 2.0 scaling factor (§6.3), giving an effective LoRA update rate equivalent to lr = 2e-4 with α/r = 1.0. This is consistent with QLoRA [13] (which used 2e-4 for Guanaco) and NEFMind [3] (which used 1e-4 for Phi-2 fine-tuning).

Warmup steps = 100. Warmup prevents early-training gradient spikes that can destabilize 4-bit quantized training. With an effective batch size of 32 and 39,294 training samples, 100 warmup steps corresponds to processing ~3,200 samples — approximately 8% of one epoch. This is within the 5–10% range recommended by empirical studies of LLM fine-tuning stability [6, 13].

Cosine decay rationale. Cosine decay [16] provides a smooth, gradual reduction in learning rate that avoids the sharp transitions of step-based schedules. This is advantageous for structured generation tasks where the loss landscape transitions from learning broad schema structures (early training) to refining specific field values and nesting patterns (late training). The gradually decreasing LR allows the model to make fine-grained adjustments in later stages without catastrophic parameter changes.

7.3 Mixed Precision: BFloat16

Configuration: bf16=True

All forward and backward computations use Brain Floating Point 16 (BF16) precision, following the mixed precision training framework [17].

Format	Exponent bits	Mantissa bits	Range	Precision
FP32	8	23	±3.4×10³⁸	~7 decimal digits
FP16	5	10	±6.5×10⁴	~3.3 decimal digits
BF16	8	7	±3.4×10³⁸	~2.4 decimal digits

Why BF16 over FP16. BF16 has the same 8-bit exponent as FP32, giving it an identical dynamic range. This eliminates the overflow/underflow problems that plague FP16 training and removes the need for loss scaling [17]. The reduced mantissa precision (7 vs. 23 bits) introduces slightly more rounding noise, but this is empirically negligible for transformer training and is compensated by BF16's superior numerical stability [18].

BF16 in the QLoRA context. The base model weights are stored in NF4 (4 bits) and dequantized to BF16 on-the-fly for computation. LoRA adapter weights are stored, trained, and computed in BF16 throughout. The optimizer states (Adam first/second moments) are maintained in FP32 for numerical stability — but only for the LoRA parameters (160M), not the full model (8.2B), making this feasible.

7.4 Flash Attention 2

Configuration: attn_implementation="flash_attention_2"

Flash Attention [19] and its successor Flash Attention-2 [20] implement IO-aware exact attention computation that avoids materializing the full N×N attention matrix in GPU HBM (High Bandwidth Memory):

• Standard attention: Computes S = QK^T (O(N²) memory), applies softmax, multiplies by V. The N×N matrix must be stored in HBM.
• Flash Attention: Uses tiling to split Q, K, V into blocks that fit in SRAM (on-chip memory), computing attention block-by-block without ever materializing the full attention matrix in HBM.

Flash Attention-2 improvements [20]:

1. Reduced non-matmul FLOPs (which dominate latency on modern GPUs where GEMM throughput >> memory throughput)
2. Parallelization across both the sequence length dimension and batch/head dimensions
3. Achieves 50–73% of theoretical peak GPU FLOP/s (vs. ~25–40% for standard attention)

Impact on our pipeline:

• Memory: For seq_len = 4096, the attention matrix is 4096 × 4096 × 32 heads × 4 bytes (FP32) = 2 GB per layer per sample. Flash Attention reduces this to O(N) ≈ a few MB.
• Speed: Approximately 2× wall-clock speedup for attention computation, which constitutes 30–50% of total training time for transformer models.
• Correctness: Flash Attention computes exact attention (not an approximation) — the outputs are mathematically identical to standard attention up to floating-point rounding.

7.5 Gradient Checkpointing

Configuration: gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}

Gradient checkpointing [21] trades compute for memory by selectively discarding intermediate activations during the forward pass and recomputing them during backpropagation:

• Without checkpointing: All activations for all L layers are stored → O(L) memory
• With checkpointing: Only checkpoint activations at every √L layers → O(√L) memory, with ~33% extra compute for recomputation

For Qwen3-8B with 36 layers and seq_len 4096:

• Without checkpointing: ~24 GB activation memory
• With checkpointing: ~6–8 GB activation memory

The use_reentrant=False flag enables the non-reentrant variant of checkpointing, which is compatible with LoRA's frozen parameters and avoids bugs with gradient computation for parameters that do not require gradients.

QLoRA-specific benefit. Dettmers et al. [13] note that with gradient checkpointing, LoRA input gradients drop from 567 MB to ~18 MB per sequence for a 7B model, because only the small LoRA adapter gradients need to be propagated through the recomputed segments.

7.6 Sequence Length

Configuration: max_length=4096

The maximum sequence length of 4,096 tokens was chosen based on the dataset token distribution analysis (§3.2):

Coverage	Token threshold
95th percentile	~1,200
99th percentile	~2,800
Maximum	~3,900
Configured max_length	4,096

This covers 100% of samples in the dataset without truncation. Setting max_length below 4,096 would silently truncate the longest samples — which are typically the most complex multi-KPI configurations — degrading the model's ability to learn these critical cases.

Sequences shorter than 4,096 tokens are padded to the right with the pad token. The loss is computed only on non-padded, non-masked tokens (see §7.8).

Note on the parameter name. In TRL v1.3+, the sequence length parameter for SFTConfig is max_length, not the deprecated max_seq_length from earlier versions.

7.7 Batch Size & Gradient Accumulation

Configuration: per_device_train_batch_size=4, gradient_accumulation_steps=8 → effective batch size = 32

The effective batch size is the product of per-device batch size and gradient accumulation steps:

effective_batch = per_device_batch × gradient_accumulation × num_gpus
                = 4 × 8 × 1 = 32

Per-device batch size = 4. This is the largest batch that fits in VRAM alongside the quantized model, LoRA adapters, optimizer states, and checkpointed activations (see §9 for full VRAM budget). Each sample at max_length 4096 with BF16 activations requires approximately 1.5–2 GB for forward+backward pass with gradient checkpointing.

Gradient accumulation = 8. Gradient accumulation simulates a larger batch by accumulating gradients over 8 forward-backward passes before performing a single optimizer step. This is mathematically equivalent to training with batch size 32 (assuming negligible batch normalization effects, which is the case for RMSNorm used in Qwen3).

Effective batch size = 32 justification:

• Too small (< 16): Noisy gradient estimates lead to training instability, especially harmful for structured generation where precise field-level learning requires stable gradients
• Too large (> 64): Generalization degradation observed in LLM fine-tuning [13]; also wastes compute per epoch
• 32: Consistent with QLoRA [13] (used 16–32 for Guanaco), NEFMind [3] (effective batch 32), and the general recommendation for SFT tasks [9]

Training steps calculation:

steps_per_epoch = ceil(39,294 / 32) = 1,229
total_steps = 1,229 × 3 epochs = 3,687

7.8 Assistant-Only Loss Masking

Configuration: assistant_only_loss=True

The cross-entropy loss is computed only on assistant (completion) tokens, with system and user (prompt) tokens masked using label_id = -100 (the PyTorch CrossEntropyLoss ignore index).

Given a sample with tokens [t₁, t₂, ..., t_n] where tokens t₁ through t_p are the system+user prompt and t_{p+1} through t_n are the assistant completion:

L = -1/(n-p) · Σ_{i=p+1}^{n} log P(t_i | t_1, ..., t_{i-1})

Justification:

1. Standard practice. Completion-only loss was established by InstructGPT [22] in its SFT stage and codified by Stanford Alpaca [23] via IGNORE_INDEX = -100 masking. The TRL library implements this through the DataCollatorForCompletionOnlyLM [9], activated by the assistant_only_loss=True flag.

2. Task-specific rationale. In our pipeline, the system prompt (which specifies the target standard and schema) and user intent (natural language) are inputs that the model should condition on but not predict. Computing loss on these tokens would dilute the training signal: the model would spend gradient capacity learning to reproduce prompt text rather than learning the intent→config mapping.

3. Loss signal concentration. The assistant completions (JSON configurations) comprise approximately 60–70% of the total token count. Without masking, the model would receive loss signal on ~100% of tokens, but only 60–70% would be task-relevant. With masking, 100% of the loss signal is directed toward the structured generation task.

4. Counterpoint. Shi et al. [24] argue that including loss on instruction tokens can reduce overfitting when the instruction-to-output length ratio is high. However, in our case, the ratio is approximately 0.4–0.5 (prompts are shorter than completions), making the overfitting risk low and the benefits of concentrated loss signal outweigh the regularization benefit.

7.9 Data Format: ChatML

The dataset uses the ChatML (Chat Markup Language) format, tokenized via tokenizer.apply_chat_template():

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_intent}<|im_end|>
<|im_start|>assistant
{json_configuration}<|im_end|>

ChatML was introduced by OpenAI as an engineering standard for multi-turn conversation formatting and has been adopted by most modern open-source models including Qwen3 [10], Mistral, and the Llama 3 family. Qwen3's tokenizer natively defines the special tokens <|im_start|> (ID: 151644) and <|im_end|> (ID: 151645) in its vocabulary, ensuring lossless tokenization of turn boundaries.

Why ChatML over alternatives:

• Native support: No special token additions or tokenizer modifications needed
• Clear turn boundaries: The <|im_start|> / <|im_end|> delimiters allow precise identification of where assistant responses begin and end — essential for loss masking (§7.8)
• Standard compatibility: The same format is used by the Qwen3 instruct models' post-training, meaning our fine-tuning operates in the same format the base model was optimized for

7.10 Training Epochs

Configuration: num_train_epochs=3

Three epochs means the model sees each training sample exactly three times. This choice balances:

• Underfitting risk (too few epochs). With 39,294 training samples and the model needing to learn 6 distinct JSON schemas + lifecycle operations + adversarial patterns, a single pass is insufficient for convergence — particularly for minority classes (e.g., HMTC slice type, adversarial samples).

• Overfitting risk (too many epochs). With LoRA's limited parameter count (~160M trainable out of 8.2B), overfitting manifests as memorization of training JSON templates rather than learning generalizable intent-to-structure mappings. The eval loss should be monitored (§7.11) to detect this.

• Empirical precedent.

  • QLoRA [13]: 1 epoch for Guanaco (large-scale instruction data)
  • NEFMind [3]: "iterative" training (not specified, but small dataset of 765)
  • TelecomGPT [6]: 3 epochs for SFT stage
  • General SFT best practices [9]: 1–5 epochs depending on dataset size

For our dataset size (~40K samples, ~27M tokens), 3 epochs provides approximately 81M tokens of training signal — sufficient for stable convergence without significant overfitting, given the regularization from LoRA dropout (§6.5), weight decay (§7.1), and the limited adapter capacity.

7.11 Checkpoint Strategy & Early Stopping

Configuration:

save_strategy="steps"
save_steps=307          # ~4 saves per epoch
eval_strategy="steps"
eval_steps=307          # evaluation every ~307 steps
save_total_limit=3      # keep only 3 most recent checkpoints
load_best_model_at_end=True
metric_for_best_model="eval_loss"
greater_is_better=False

Evaluation frequency. Evaluation is performed approximately 4 times per epoch (eval_steps = ceil(steps_per_epoch / 4)). This provides sufficient granularity to detect overfitting onset (eval loss begins increasing while train loss continues decreasing) without excessive evaluation overhead. Each evaluation pass over 2,521 test samples takes approximately 5–10 minutes.

Best model selection. After training completes, the checkpoint with the lowest validation loss is automatically loaded and saved as the final model. This implements a form of early stopping — if the model overfits in later epochs, the best-performing earlier checkpoint is preserved.

Disk management. save_total_limit=3 retains only the three most recent checkpoints, preventing disk overflow. Each QLoRA checkpoint is approximately 320 MB (adapter weights + optimizer states), so the maximum disk usage is ~1 GB for checkpoints.

---

8. Evaluation Methodology

The evaluation pipeline (evaluate.py) implements five complementary metrics that assess different aspects of intent translation quality. This multi-dimensional evaluation follows the decomposition proposed by Deng et al. [1], who argue that single-metric evaluation of structured LLM outputs is insufficient because "the Structure Gap manifests across multiple dimensions: syntactic validity, schema compliance, content accuracy, and robustness."

8.1 JSON Syntactic Validity

Metric: Fraction of model outputs that parse as valid JSON.

JSON_Validity = |{y_i : parse(y_i) succeeds}| / N

This is the most fundamental requirement: if the output is not valid JSON, it cannot be processed by any downstream telecom system. The implementation attempts parsing in three stages:

1. Direct json.loads() on the raw output
2. Markdown fence removal (strip ```json ``` wrappers) then re-parse
3. Regex extraction of the first {...} block, then parse

This multi-stage approach accounts for common LLM generation artifacts (markdown formatting, preamble text) while remaining strict about JSON validity itself.

Citation context. JSONSchemaBench [25] established JSON validity rate as the primary metric for structured generation evaluation, showing that even state-of-the-art constrained decoding frameworks exhibit significant validity gaps on complex schemas. RL-Struct [1] formalizes this as R_val — a binary reward signal (1 if parseable, 0 otherwise).

8.2 Structural Schema Correctness

Metric: Fraction of valid JSON outputs that contain the correct root-level keys for the target telecom standard.

Structure_Correctness = |{y_i : valid(y_i) ∧ has_expected_keys(y_i, layer_i)}| / N

The expected root keys per standard are:

Target Layer	Required Root Keys
TMF921	id, href, name, intentExpression
3GPP (intent_3gpp)	intent
CAMARA	networkSliceBooking
ETSI ZSM	zsmIntent
O-RAN A1 Policy	a1Policy
3GPP O1 NRM	managedElement
Adversarial	status ∈ {CLARIFICATION_REQUIRED, OUT_OF_SCOPE, INTENT_VALIDATION_FAILED}
Lifecycle	intentPatch or intentAssuranceReport or intentUpdate

This metric captures whether the model has learned the structural identity of each standard — a necessary condition for downstream system compatibility. A JSON output can be syntactically valid but structurally incorrect (e.g., producing a TMF921 schema when the target was CAMARA).

Citation context. RL-Struct [1] introduces R_struct as a distinct metric from R_val: "structural accuracy measures whether required keys are present and correctly nested, independent of value correctness." ORION [2] evaluates structural correctness implicitly through its MCP tool-use quality metric, which checks "argument structure" as one component.

8.3 KPI Field Extraction Accuracy

Metric: For each of 5 KPI fields (latency_ms, reliability_pct, dl_throughput_mbps, ul_throughput_mbps, max_ues), whether the ground-truth value appears in the generated JSON.

KPI_Accuracy(field) = |{y_i : str(ground_truth_field_i) ∈ flatten(y_i)}| / N_applicable

Where flatten() converts the JSON to a string for substring matching, and N_applicable excludes adversarial and lifecycle samples (which don't have KPI fields).

A composite metric All_KPIs_Correct requires all 5 fields to be present and correct simultaneously:

All_KPIs = |{y_i : ∀f ∈ {lat, rel, dl, ul, ues}: correct(y_i, f)}| / N_applicable

Why substring matching over exact path matching. Different telecom standards place KPI values at different JSON paths (e.g., TMF921 nests latency under intentExpression.intentTargets[].targetThresholds[], while CAMARA places it under networkSliceBooking.sliceProfile.latency). Substring matching on the flattened JSON is standard-agnostic and captures the value regardless of its position in the schema hierarchy.

Citation context. This approach is analogous to the R_cor (content correctness) metric in RL-Struct [1], which computes F1 between generated and ground-truth field values. ORION [2] evaluates field-level accuracy through its "value fidelity" component of the MCPUseMetric. NEFMind [3] used BERTScore (0.997–0.998 for fine-tuned model) as a softer measure of content accuracy.

8.4 Adversarial Robustness

Metric: Fraction of adversarial inputs (ambiguous, out-of-scope, contradictory) correctly rejected with an appropriate error status.

Adversarial_Accuracy = |{y_i : valid(y_i) ∧ y_i["status"] ∈ ADVERSARIAL_STATUSES}| / N_adversarial

Where ADVERSARIAL_STATUSES = {CLARIFICATION_REQUIRED, OUT_OF_SCOPE, INTENT_VALIDATION_FAILED}.

This metric tests the model's ability to refuse generation when the input intent is malformed, rather than hallucinating a plausible-looking but incorrect configuration. In production telecom systems, generating a misconfigured network slice from an ambiguous intent is more dangerous than returning an error [26].

Citation context. DecodingTrust [27] established adversarial robustness as a core trustworthiness dimension for LLMs. AdvGLUE [28] defines the robustness metric as Acc_adversarial / Acc_clean — a ratio quantifying the degradation from clean to adversarial conditions. SafeCOMM [29] specifically studies safety degradation in telecom-tuned LLMs, finding that domain fine-tuning can reduce the model's ability to refuse harmful requests — directly motivating the inclusion of adversarial samples in our training data.

8.5 Greedy Decoding for Evaluation

Configuration: do_sample=False, temperature=None, top_p=None

All evaluation uses greedy decoding (argmax token selection) rather than sampling. This decision is grounded in:

1. Reproducibility. Greedy decoding is deterministic — the same input always produces the same output, enabling reproducible evaluation [30]. Song et al. [31] demonstrated that even with temperature=0, hardware/precision differences can cause up to 9% accuracy variation; any sampling would compound this non-determinism.

2. Task characteristics. Intent-to-configuration translation has a deterministic ground truth — there is exactly one correct JSON configuration for each intent+standard pair. Song et al. [31] found that "for reasoning tasks requiring LLMs to solve specific problems with definite solutions, greedy decoding outperforms sampling" — structured generation with schema constraints falls squarely in this category.

3. Evaluation convention. Greedy decoding at temperature 0 is the standard protocol in LLM evaluation, as stated by Yu et al. [30]: "greedy prediction mode, where temperature is set to 0" — adopted in DecodingTrust [27], ORAN-Bench-13K [32], and NEFMind [3].

---

9. VRAM Budget Analysis

Target hardware: NVIDIA RTX 6000 Ada Generation (48 GB VRAM).

Component	Memory	Source/Calculation
Base model weights (NF4 + DQ)	~4.6 GB	8.2B × 4 bits / 8 + DQ overhead
LoRA adapter weights (BF16)	~0.32 GB	~160M × 2 bytes
Optimizer states (FP32, LoRA only)	~1.28 GB	~160M × 2 states × 4 bytes
Gradient storage (BF16, LoRA only)	~0.32 GB	~160M × 2 bytes
Activations (checkpointed, batch=4)	~8 GB	~2 GB/sample × 4, with checkpointing
Flash Attention workspace	~2 GB	IO-aware tiling buffers
CUDA context + overhead	~2 GB	Driver, cuBLAS handles, etc.
Total estimated	~18.5 GB
Available headroom	~29.5 GB	48 − 18.5

The substantial headroom (>60% of total VRAM) provides a safety margin for:

• Longer sequences (a few samples approaching 4,096 tokens increase activation memory)
• PyTorch memory fragmentation
• Potential for increasing per_device_train_batch_size to 8 (estimated +6 GB)

---

10. Software Stack & Reproducibility

10.1 Dependencies

Package	Minimum Version	Role
torch	≥ 2.4.0	Core tensor operations, autograd
transformers	≥ 4.46.0	Model loading, tokenizer, trainer base class
trl	≥ 1.3.0	SFTTrainer, SFTConfig, chat template handling
peft	≥ 0.15.0	LoraConfig, PeftModel, adapter management
datasets	≥ 3.0.0	Dataset loading from HuggingFace Hub
bitsandbytes	≥ 0.45.0	NF4 quantization, 4-bit matrix multiplication
accelerate	≥ 1.0.0	Device mapping, mixed precision, distributed training
flash-attn	≥ 2.7.0	Flash Attention 2 CUDA kernels
scipy	—	Statistical utilities for evaluation

10.2 Reproducibility Controls

• Random seed = 42 across all sources of randomness (Python, NumPy, PyTorch, CUDA)
• Deterministic data loading via datasets library with fixed shuffle seeds
• Greedy decoding for evaluation (§8.5)
• Fixed LoRA initialization (B = 0 ensures training starts from exact pre-trained weights)
• Pinned dependency versions in requirements.txt

10.3 Logging

Configuration: logging_strategy="steps", logging_steps=10, logging_first_step=True, disable_tqdm=False

Training loss is logged every 10 optimizer steps (320 samples) as plain text to stdout, enabling both real-time monitoring and post-hoc analysis via grep "loss" training.log. The first step is explicitly logged to verify training begins at a reasonable loss value (expected: ~2.5–4.0 for cross-entropy on a 151K vocabulary).

---

11. Limitations & Threats to Validity

11.1 Dataset Limitations

1. Adversarial response homogeneity (§3.5): All samples within each adversarial category share identical responses. The model may achieve high adversarial accuracy by memorizing three templates rather than learning contextual rejection reasoning.

2. Single-standard lifecycle operations: Lifecycle management is only represented for TMF921. The model cannot be expected to generalize lifecycle operations to other standards.

3. Synthetic data provenance: The dataset is synthetically generated (not derived from real network deployments). The distribution of KPI values, region-sector combinations, and use cases may not reflect production telecom traffic patterns.

4. English-only intents: All intents are in English. Multilingual intent translation (leveraging Qwen3's 119-language pre-training) is untested.

11.2 Methodological Limitations

5. Substring matching for KPI evaluation (§8.3): This approach cannot distinguish between a KPI value appearing in the correct JSON field vs. appearing elsewhere in the output (e.g., in a comment field or wrong KPI slot). A more rigorous evaluation would use JSONPath queries against the expected schema.

6. No semantic equivalence checking: Two JSON configurations can be structurally different but semantically equivalent (e.g., different field ordering, equivalent but differently formatted values). Our evaluation treats these as incorrect.

7. Single base model: We evaluate only Qwen3-8B. Comparative studies with Llama 3.1-8B, Mistral-7B, and Phi-4 would strengthen the generalizability claims.

8. No inference latency benchmarking: For production intent translation, latency matters. We do not report inference time per intent.

11.3 Threats to External Validity

9. Standard version drift: Telecom standards evolve rapidly. The TMF921 v5.0, 3GPP Rel-18, and CAMARA schemas used in this dataset may become outdated as new releases appear.

10. No closed-loop validation: The generated configurations are not validated against actual network orchestration systems (e.g., ONAP, O-RAN SMO). A structurally correct JSON may still fail deployment due to semantic constraints not captured in our evaluation.

---

12. References

[1] Y. Deng et al., "RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs," arXiv:2512.00319, 2024.

[2] E. McMahon et al., "ORION: A Holistic End-to-End AI Framework for Intent-Aware Orchestration in O-RAN," arXiv:2603.03667, 2025.

[3] S. Niknam et al., "NEFMind: LLM-Driven 5G Network Exposure — A Fine-Tuning Approach," arXiv:2508.09240, 2025.

[4] A. Duarte da Costa et al., "Hermes: A Large Language Model Framework on the Journey to Autonomous Networks," arXiv:2411.06490, 2024.

[5] L. N. T. Huynh et al., "When LLMs Meet Network Slicing: An LLM-Based Framework for Intent-Driven Network Slicing Management," arXiv:2403.13721, 2024.

[6] Z. Zhou et al., "TelecomGPT: A Framework to Build Telecom-Specific Large Language Models," arXiv:2407.09424, 2024.

[7] A. Maatouk et al., "ORANSight-2.0: Refining Open Radio Access Networks Alignment with the RANSTRUCT Dataset," arXiv:2503.05200, 2025.

[8] P. Mahi et al., "A Comprehensive Survey on the Role of Generative AI in Network Monitoring and Management," arXiv:2502.08576, 2025.

[9] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec, "TRL: Transformer Reinforcement Learning," GitHub, 2020. [Online]. Available: https://github.com/huggingface/trl

[10] Qwen Team, "Qwen3 Technical Report," arXiv:2505.09388, 2025.

[11] N. Shazeer, "GLU Variants Improve Transformer," arXiv:2002.05202, 2020.

[12] J. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," arXiv:2305.13245, 2023.

[13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs," arXiv:2305.14314, NeurIPS, 2023.

[14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "LoRA: Low-Rank Adaptation of Large Language Models," arXiv:2106.09685, ICLR, 2022.

[15] I. Loshchilov and F. Hutter, "Decoupled Weight Decay Regularization," arXiv:1711.05101, ICLR, 2019.

[16] I. Loshchilov and F. Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts," arXiv:1608.03983, ICLR, 2017.

[17] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, "Mixed Precision Training," arXiv:1710.03740, ICLR, 2018.

[18] D. Kalamkar et al., "A Study of BFLOAT16 for Deep Learning Training," arXiv:1905.12322, 2019.

[19] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," arXiv:2205.14135, NeurIPS, 2022.

[20] T. Dao, "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning," arXiv:2307.08691, ICLR, 2024.

[21] T. Chen, B. Xu, C. Zhang, and C. Guestrin, "Training Deep Nets with Sublinear Memory Cost," arXiv:1604.06174, 2016.

[22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback," arXiv:2203.02155, NeurIPS, 2022.

[23] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Stanford Alpaca: An Instruction-Following LLaMA Model," GitHub, 2023. [Online]. Available: https://github.com/tatsu-lab/stanford_alpaca

[24] Z. Shi, A. X. Yang, B. Wu, L. Aitchison, E. Yilmaz, and A. Lipani, "Instruction Tuning With Loss Over Instructions," arXiv:2405.14394, 2024.

[25] G. Geng et al., "Generating Structured Outputs from Language Models: Benchmark and Studies," arXiv:2501.10868, 2025.

[26] A. Fressancourt and A. Mahi, "SafeCOMM: Evaluating the Impact of Fine-tuning on Safety Capabilities of LLMs for Telecommunications Applications," arXiv:2506.00062, 2025.

[27] B. Wang et al., "DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models," arXiv:2306.11698, NeurIPS, 2023.

[28] B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li, "Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models," arXiv:2111.02840, 2021.

[29] A. Fressancourt and A. Mahi, "SafeCOMM: Evaluating the Impact of Fine-tuning on Safety Capabilities of LLMs for Telecommunications Applications," arXiv:2506.00062, 2025.

[30] T. Yu et al., "Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning," arXiv:2506.09501, 2025.

[31] Y. Song et al., "The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism," arXiv:2407.10457, 2024.

[32] A. Maatouk et al., "ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks," arXiv:2407.06245, 2024.

[33] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, "PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods," GitHub, 2022. [Online]. Available: https://github.com/huggingface/peft

[34] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," arXiv:2208.07339, NeurIPS, 2022.

[35] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned Language Models Are Zero-Shot Learners," arXiv:2109.01652, ICLR, 2022.

[36] Qwen Team, "Qwen2.5 Technical Report," arXiv:2412.15115, 2024.

[37] A. Narayanan et al., "HRL for Intent-Driven O-RAN xApp Orchestration," arXiv:2307.02754, 2023.

[38] Y. Liu et al., "An AI/ML-Driven SMO Framework for O-RAN," arXiv:2409.05092, 2024.

---

Document generated for nraptisss/intent-translation-training. Last updated: April 2025.