DOCUMENTATION.md

# Scientific Documentation: QLoRA Fine-Tuning Pipeline for Multi-Standard Telecom Intent Translation

> **Repository:** [nraptisss/intent-translation-training](https://huggingface.co/nraptisss/intent-translation-training)  
> **Dataset:** [nraptisss/TMF921-intent-to-config-augmented](https://huggingface.co/datasets/nraptisss/TMF921-intent-to-config-augmented)  
> **Task:** Natural language network intent → structured JSON configuration across 6 telecom standards  

---

## Table of Contents

1. [Problem Statement & Motivation](#1-problem-statement--motivation)
2. [Related Work & Positioning](#2-related-work--positioning)
3. [Dataset Description & Audit](#3-dataset-description--audit)
4. [Base Model Selection](#4-base-model-selection)
5. [Quantization Strategy: 4-bit NormalFloat (NF4)](#5-quantization-strategy-4-bit-normalfloat-nf4)
6. [Parameter-Efficient Fine-Tuning: LoRA](#6-parameter-efficient-fine-tuning-lora)
7. [Training Configuration](#7-training-configuration)
   - 7.1 [Optimizer: AdamW with Decoupled Weight Decay](#71-optimizer-adamw-with-decoupled-weight-decay)
   - 7.2 [Learning Rate Schedule: Linear Warmup + Cosine Decay](#72-learning-rate-schedule-linear-warmup--cosine-decay)
   - 7.3 [Mixed Precision: BFloat16](#73-mixed-precision-bfloat16)
   - 7.4 [Flash Attention 2](#74-flash-attention-2)
   - 7.5 [Gradient Checkpointing](#75-gradient-checkpointing)
   - 7.6 [Sequence Length](#76-sequence-length)
   - 7.7 [Batch Size & Gradient Accumulation](#77-batch-size--gradient-accumulation)
   - 7.8 [Assistant-Only Loss Masking](#78-assistant-only-loss-masking)
   - 7.9 [Data Format: ChatML](#79-data-format-chatml)
   - 7.10 [Training Epochs](#710-training-epochs)
   - 7.11 [Checkpoint Strategy & Early Stopping](#711-checkpoint-strategy--early-stopping)
8. [Evaluation Methodology](#8-evaluation-methodology)
   - 8.1 [JSON Syntactic Validity](#81-json-syntactic-validity)
   - 8.2 [Structural Schema Correctness](#82-structural-schema-correctness)
   - 8.3 [KPI Field Extraction Accuracy](#83-kpi-field-extraction-accuracy)
   - 8.4 [Adversarial Robustness](#84-adversarial-robustness)
   - 8.5 [Greedy Decoding for Evaluation](#85-greedy-decoding-for-evaluation)
9. [VRAM Budget Analysis](#9-vram-budget-analysis)
10. [Software Stack & Reproducibility](#10-software-stack--reproducibility)
11. [Limitations & Threats to Validity](#11-limitations--threats-to-validity)
12. [References](#12-references)

---

## 1. Problem Statement & Motivation

Intent-Based Networking (IBN) enables network operators to express high-level business objectives in natural language (e.g., *"Deploy a URLLC slice for autonomous driving in Munich with 1ms latency and 99.999% reliability"*), which must be translated into spec-compliant machine-readable configurations across heterogeneous telecom standards. This translation spans multiple abstraction layers: from business-level intent APIs (TMF921, ETSI ZSM) through network management functions (3GPP TS 28.312) to radio access network policies (O-RAN A1/E2) and external exposure APIs (CAMARA/NEF).

The core challenge is what Deng et al. [1] term the **"Structure Gap"** — the fundamental mismatch between probabilistic language model generation and the deterministic, schema-constrained configurations required by telecom systems. A single misplaced field, incorrect nesting level, or hallucinated KPI value renders the output unusable by downstream orchestration systems.

**Why fine-tuning is necessary.** ORION [2] demonstrated that large proprietary models (GPT-5, Claude Opus 4.5) achieve 100% policy creation success on intent translation tasks, but: (a) they evaluated only 100 intents targeting a single standard (CAMARA NetworkSliceBooking), (b) they reported no open-source fine-tuned alternative, (c) the per-intent cost ranges from $0.19–$29.68, making production deployment impractical. NEFMind [3] showed that QLoRA fine-tuning of the 2.7B Phi-2 model on just 765 NEF API records achieved 98–100% accuracy — demonstrating that domain-specific fine-tuning of small open models can match or exceed proprietary model performance at negligible marginal cost.

**Our contribution.** This pipeline produces the first open-source fine-tuned model for multi-standard intent-to-configuration translation, covering 6 telecom standards, 8 lifecycle operations, and adversarial intent rejection — trained on a dataset 250× larger than ORION's evaluation set [2] and 55× larger than NEFMind's training corpus [3].

---

## 2. Related Work & Positioning

| Work | Approach | Standards | Dataset Size | Model | Key Limitation |
|------|----------|-----------|-------------|-------|----------------|
| **ORION** [2] | LLM + MCP tool-use | CAMARA only | 100 eval intents | GPT-5/Claude (proprietary) | No fine-tuning; single standard; no lifecycle |
| **Hermes** [4] | Chain-of-LLM-agents | YAML blueprints | Not disclosed | GPT-4o (proprietary) | 82.5% on power control; no JSON schemas |
| **NEFMind** [3] | QLoRA fine-tuning | CAMARA/NEF | 765 records | Phi-2 (2.7B) | Single API; no multi-standard coverage |
| **LLMs meet Slicing** [5] | Multi-agent framework | 3GPP CSMF/NSMF | Conceptual | GPT-4 (proprietary) | No implementation or evaluation |
| **TelecomGPT** [6] | CPT + SFT + DPO | General telecom | Proprietary corpus | LLaMA-based | General QA, not intent translation |
| **ORANSight-2.0** [7] | RAG + instruction tuning | O-RAN specs | 13K questions | 18 models tested | MCQ format, not config generation |
| **This work** | **QLoRA SFT** | **6 standards + lifecycle + adversarial** | **41,815 samples** | **Qwen3-8B (open)** | See §11 |

Our pipeline directly addresses three gaps identified in the literature survey by Mahi et al. [8]: (1) no open-source fine-tuned model exists for multi-standard intent translation, (2) lifecycle management beyond "create" is unexplored, and (3) adversarial robustness of intent translation systems has not been evaluated.

---

## 3. Dataset Description & Audit

**Dataset:** `nraptisss/TMF921-intent-to-config-augmented` — 41,815 samples (39,294 train / 2,521 test).

### 3.1 Coverage

| Dimension | Values |
|-----------|--------|
| **Target standards** | TMF921, 3GPP TS 28.312 (intent_3gpp), CAMARA NetworkSliceBooking, ETSI ZSM, O-RAN A1 Policy, 3GPP O1 NRM |
| **Slice types** | eMBB, URLLC, mMTC, V2X, MPS, HMTC |
| **Sectors** | 18 (automotive, healthcare, manufacturing, energy, …) |
| **Use cases** | 147 distinct |
| **Regions** | 55 |
| **Lifecycle operations** | 8: activate, modify, suspend, resume, terminate, scale, monitor, report (1,552 samples, TMF921 only) |
| **Adversarial samples** | 141: ambiguous (58), out_of_scope (43), contradictory (40) |

### 3.2 Token Statistics

| Metric | Value |
|--------|-------|
| Total tokens (approx.) | ~27M |
| Mean tokens per sample | ~690 |
| Median tokens per sample | ~650 |
| 95th percentile | ~1,200 |
| 99th percentile | ~2,800 |
| Maximum | ~3,900 |

These statistics motivate the choice of `max_length=4096` (§7.6).

### 3.3 Format

Each sample follows the ChatML conversational format with exactly 3 turns:

```json
{
  "messages": [
    {"role": "system", "content": "You are a telecom intent translator..."},
    {"role": "user", "content": "Deploy a URLLC slice for..."},
    {"role": "assistant", "content": "{\"id\": \"intent-...\", ...}"}
  ]
}
```

The system prompt specifies the target standard and expected output schema. The user message contains the natural language intent. The assistant message contains the ground-truth JSON configuration. This format is directly compatible with the TRL `SFTTrainer` [9] via the `apply_chat_template` function, requiring no preprocessing.

### 3.4 Augmentation Over Base Dataset

The augmented dataset (41,815 samples) extends the base `TMF921-intent-to-config-25k` (25,000 samples) with:

- **16,819 new samples** (42.8% of augmented-only content) across additional use cases and regions
- **1,552 lifecycle operation samples** (entirely new — absent from base dataset)
- **141 adversarial samples** (entirely new)

### 3.5 Known Data Quality Issues

**Adversarial response homogeneity.** All 58 ambiguous samples map to an identical JSON response; all 43 out-of-scope samples to another identical JSON; all 40 contradictory to a third. This means the model may learn to classify adversarial *categories* without learning to generate contextually specific rejection explanations. For production deployment, adversarial responses should be diversified with intent-specific clarification requests.

**Lifecycle operation coverage.** Lifecycle operations (activate, modify, suspend, resume, terminate, scale, monitor, report) are only represented for the TMF921 standard. Cross-standard lifecycle management (e.g., modifying a CAMARA booking or suspending an A1 policy) is not covered.

---

## 4. Base Model Selection

**Model:** Qwen3-8B [10] — an 8.2 billion parameter dense decoder-only transformer from Alibaba's Qwen team.

### 4.1 Architecture

| Parameter | Value |
|-----------|-------|
| Layers | 36 |
| Hidden size | 4,096 |
| Intermediate size (FFN) | 12,288 |
| Query attention heads | 32 |
| Key-Value attention heads | 8 (GQA, 4:1 ratio) |
| Head dimension | 128 |
| Positional encoding | RoPE (θ = 1,000,000) |
| Activation | SwiGLU [11] |
| Normalization | RMSNorm (pre-norm, ε = 10⁻⁶) |
| Max context length | 128K tokens |
| Vocabulary size | 151,936 (byte-level BPE) |
| Pre-training data | 36 trillion tokens, 119 languages |

### 4.2 Justification

The choice of Qwen3-8B over alternatives is motivated by several factors:

1. **Size-performance trade-off.** At 8B parameters, the model fits comfortably in 4-bit quantization on a single 48GB GPU (~5.5 GB for weights), leaving ample headroom for training activations and optimizer states. This is the sweet spot identified by NEFMind [3] (which fine-tuned 2.7B Phi-2) and TelecomGPT [6] (which used LLaMA-based 7-8B models).

2. **Grouped Query Attention (GQA).** The 4:1 Q-to-KV head ratio reduces KV cache memory by 4× during inference compared to standard multi-head attention [12], enabling efficient batch inference during evaluation on the 2,521 test samples.

3. **Native ChatML support.** Qwen3's tokenizer natively supports the `<|im_start|>` / `<|im_end|>` ChatML delimiters used in our dataset, eliminating tokenizer modification or special token addition.

4. **128K context window.** While our samples are ≤4K tokens, the long-context pre-training with RoPE θ = 10⁶ means the model has robust positional encoding even at our maximum sequence length — no extrapolation is needed.

5. **Multilingual pre-training.** With 119 languages in pre-training, the model has exposure to diverse naming conventions, region-specific terminology, and technical vocabulary that appears in multi-region telecom intents (55 regions in our dataset).

6. **Dual thinking mode.** Qwen3 introduces `<think>`/`</think>` tokens enabling chain-of-thought reasoning. While we disable thinking mode for fine-tuning (structured output should be direct), this capacity for internal reasoning may benefit intent disambiguation.

7. **Open weights under Apache 2.0.** Unlike proprietary models used in ORION [2] and Hermes [4], the open license enables reproducible research, on-premise deployment, and community extension.

### 4.3 Pre-training Composition Relevance

Qwen3 was pre-trained on 36 trillion tokens including substantial synthetic data from Qwen2.5 (textbooks, QA, instructions, code) [10, §3.1]. This is particularly relevant because:

- **Code generation training** provides structural understanding of JSON syntax, nesting, and schema compliance
- **Instruction-following pre-training** aligns with our SFT task format
- **STEM domain data** includes technical terminology relevant to telecommunications

---

## 5. Quantization Strategy: 4-bit NormalFloat (NF4)

**Implementation:** `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)`

### 5.1 NF4 Data Type

NormalFloat 4-bit (NF4) quantization was introduced in QLoRA [13] as an **information-theoretically optimal** data type for neural network weights. The construction exploits the empirical observation that pre-trained neural network weights follow approximately zero-centered normal distributions:

1. Estimate the 2^(k+1) quantiles of a standard normal distribution N(0,1)
2. Normalize the resulting quantile values into the range [-1, 1]
3. Quantize input tensors by normalizing via absolute maximum rescaling, then mapping to the nearest NF4 value

NF4 uses **asymmetric quantization** to guarantee an exact zero representation: 2^(k-1) bins for negative values and 2^(k-1)+1 bins for positive values, unified with duplicate zero removal. This ensures equal expected number of values per quantization bin, minimizing information loss — hence the "information-theoretically optimal" characterization [13, §3].

Compared to the standard FP4 data type, NF4 achieves lower quantization error on normally distributed weights because its bin boundaries are placed at the quantiles of the weight distribution rather than at uniform intervals.

### 5.2 Double Quantization

Double quantization (DQ) [13, §3] reduces the memory overhead of quantization constants:

- **First quantization:** Block size 64, producing one FP32 scaling constant per 64 parameters → 0.5 bits/parameter overhead
- **Second quantization:** The FP32 constants are mean-centered (for symmetry) and then quantized to FP8 with block size 256 → 0.127 bits/parameter overhead
- **Net savings:** 0.373 bits/parameter, or approximately 3 GB for a 65B model [13]

For Qwen3-8B (8.2B parameters), double quantization saves approximately **380 MB** of GPU memory.

### 5.3 Compute Dtype: BFloat16

All dequantized computations (matrix multiplications, attention, FFN) are performed in BFloat16 precision. The QLoRA forward pass is [13, Eq. 5]:

```
Y^BF16 = X^BF16 · doubleDequant(c1^FP32, c2^NF4, W^NF4) + X^BF16 · L1^BF16 · L2^BF16
```

Where the first term is the frozen base model computation (dequantized from NF4 to BF16 on-the-fly) and the second term is the LoRA adapter computation (always in BF16). The base model weights are never stored in BF16 — they remain in NF4 (4 bits per parameter) with dequantization performed per-block during the forward pass.

### 5.4 Memory Savings

| Precision | Memory for 8.2B params | Ratio |
|-----------|----------------------|-------|
| FP32 | 32.8 GB | 1× |
| FP16/BF16 | 16.4 GB | 2× |
| INT8 | 8.2 GB | 4× |
| **NF4 + DQ** | **~4.6 GB** | **~7×** |

This reduction is what enables fine-tuning an 8B model on a single consumer/workstation GPU.

---

## 6. Parameter-Efficient Fine-Tuning: LoRA

**Implementation:** `LoraConfig(r=32, lora_alpha=64, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules="all-linear")`

### 6.1 Method

Low-Rank Adaptation (LoRA) [14] freezes all pre-trained weights W₀ ∈ ℝ^(d×k) and injects trainable low-rank decomposition matrices:

```
h = W₀x + ΔWx = W₀x + BAx
```

where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with rank r ≪ min(d, k). Matrix A is initialized from a random Gaussian distribution; B is initialized to zero, ensuring ΔW = BA = 0 at training start — the model begins from the exact pre-trained weights.

The output is scaled by α/r, where α (`lora_alpha`) is a constant. This scaling controls the effective magnitude of the LoRA update relative to the pre-trained weights.

### 6.2 Rank Selection: r = 32

The rank r controls the expressiveness of the adaptation:

- **r = 4–8:** Sufficient for simple classification or style transfer [14, Table 5]
- **r = 16–32:** Recommended for complex structured generation tasks
- **r = 64+:** Diminishing returns; approaches full fine-tuning behavior

We select r = 32 based on:

1. **Task complexity.** Intent-to-configuration translation requires generating deeply nested JSON structures with precise field names and values across 6 different schemas — more complex than the downstream NLP tasks (MNLI, SST-2, MRPC) evaluated in the original LoRA paper [14] where r = 4 sufficed.

2. **QLoRA best practices.** Dettmers et al. [13, §4] found that "LoRA rank has diminishing returns above r=16 for most tasks but complex tasks benefit from r=32–64." They specifically recommend r ≥ 16 for structured generation.

3. **Trainable parameter count.** With r = 32 on all linear layers of Qwen3-8B:
   - Trainable parameters: ~160M (1.95% of total)
   - Adapter size on disk: ~320 MB (BF16) or ~160 MB (FP16)
   - This is feasible for our VRAM budget (§9)

### 6.3 Alpha/Rank Ratio: α = 64 (α/r = 2.0)

The scaling factor α/r controls the effective learning rate of the LoRA update. Setting α = r gives a scaling factor of 1.0 (the LoRA paper's default). We use α = 2r (scaling factor = 2.0) following the convention established by QLoRA [13] and widely adopted in the community:

- **Higher α/r amplifies updates**, compensating for the fact that 4-bit quantized base model gradients have slightly higher noise than FP16/FP32 gradients
- **Hu et al. [14]:** *"tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately"* — so α = 2r with lr = 1e-4 is effectively equivalent to α = r with lr = 2e-4
- The QLoRA paper [13] empirically validated that α = 2r performs well across Guanaco instruction-tuning benchmarks

### 6.4 Target Modules: "all-linear"

We apply LoRA to **all linear projection layers** in the model (Q, K, V, O attention projections + gate, up, down FFN projections), rather than only attention projections (Q+V) as in the original LoRA paper [14].

**Justification:**

1. **QLoRA finding [13, §4]:** *"Applying LoRA to all linear transformer block layers is important for achieving the best performance."* Adapting only attention projections leaves the FFN layers frozen — problematic when the model needs to learn new output distributions (i.e., telecom-specific JSON schemas not seen in pre-training).

2. **Empirical validation in prior telecom work.** NEFMind [3] applied QLoRA to all linear layers of Phi-2 and achieved 98-100% accuracy. ORANSight-2.0 [7] similarly applied full-layer LoRA in its best-performing configurations.

3. **Memory cost.** Applying LoRA to all linear layers increases trainable parameters from ~0.5% (Q+V only) to ~2% of total — but since these are low-rank (r=32) and trained in BF16, the absolute memory increase is modest (~200 MB additional for optimizer states).

### 6.5 LoRA Dropout: 0.05

A dropout rate of 0.05 is applied to the LoRA A matrix outputs during training. This provides mild regularization against overfitting on the training distribution, which is particularly relevant given:

- The dataset has some distribution imbalance (e.g., eMBB and URLLC samples dominate over MPS and HMTC)
- Adversarial samples are limited (141 out of 41,815 = 0.34%)

The value 0.05 follows QLoRA [13] defaults and is consistent with standard practice for LoRA fine-tuning [13, 14].

### 6.6 Bias: "none"

No bias terms are trained, following the LoRA paper [14] recommendation: *"We find that only adapting the attention weights and freezing the MLP/biases is most efficient."* Training bias terms adds minimal capacity but introduces instability in quantized training due to the interaction between quantization noise and bias gradients.

---

## 7. Training Configuration

**Implementation:** `SFTConfig(...)` from TRL [9] with the following parameters.

### 7.1 Optimizer: AdamW with Decoupled Weight Decay

**Configuration:** `optim="adamw_torch"` (default), `weight_decay=0.01`

The AdamW optimizer [15] decouples weight decay from the adaptive learning rate:

```
θ_t = θ_{t-1} − α · (m̂_t / (√v̂_t + ε)) − α · λ · θ_{t-1}
```

where m̂_t and v̂_t are bias-corrected first and second moment estimates, α is the learning rate, and λ is the weight decay coefficient.

**Why AdamW over Adam + L2.** Loshchilov & Hutter [15] demonstrated that in standard Adam with L2 regularization, the penalty |θ|² is scaled by the inverse of the adaptive learning rate √v̂_t — making the effective regularization strength layer-dependent and inconsistent. AdamW applies weight decay directly to the parameters, independent of the gradient statistics. This is particularly important for QLoRA training where different LoRA modules may have very different gradient magnitudes.

**Weight decay = 0.01.** This is the canonical value for LLM fine-tuning [13, 15], providing mild regularization without constraining the model's capacity to learn new structured output distributions. Higher values (0.05–0.1) risk underfitting on the minority classes (lifecycle operations, adversarial samples).

### 7.2 Learning Rate Schedule: Linear Warmup + Cosine Decay

**Configuration:** `lr=1e-4, lr_scheduler_type="cosine", warmup_steps=100`

The learning rate follows a two-phase schedule:

1. **Linear warmup** (steps 0–100): LR linearly increases from 0 to 1e-4
2. **Cosine decay** (steps 100–~3,690): LR decreases following:

```
η_t = η_min + 0.5 · (η_max − η_min) · (1 + cos(π · T_cur / T_total))
```

as introduced in SGDR [16], here used without restarts (single cosine cycle).

**Learning rate = 1e-4.** For LoRA fine-tuning, learning rates are typically 5–100× higher than full fine-tuning rates [13, 14]. The standard range is 1e-4 to 3e-4. We use 1e-4 as a conservative starting point, combined with the α/r = 2.0 scaling factor (§6.3), giving an effective LoRA update rate equivalent to lr = 2e-4 with α/r = 1.0. This is consistent with QLoRA [13] (which used 2e-4 for Guanaco) and NEFMind [3] (which used 1e-4 for Phi-2 fine-tuning).

**Warmup steps = 100.** Warmup prevents early-training gradient spikes that can destabilize 4-bit quantized training. With an effective batch size of 32 and 39,294 training samples, 100 warmup steps corresponds to processing ~3,200 samples — approximately 8% of one epoch. This is within the 5–10% range recommended by empirical studies of LLM fine-tuning stability [6, 13].

**Cosine decay rationale.** Cosine decay [16] provides a smooth, gradual reduction in learning rate that avoids the sharp transitions of step-based schedules. This is advantageous for structured generation tasks where the loss landscape transitions from learning broad schema structures (early training) to refining specific field values and nesting patterns (late training). The gradually decreasing LR allows the model to make fine-grained adjustments in later stages without catastrophic parameter changes.

### 7.3 Mixed Precision: BFloat16

**Configuration:** `bf16=True`

All forward and backward computations use Brain Floating Point 16 (BF16) precision, following the mixed precision training framework [17].

| Format | Exponent bits | Mantissa bits | Range | Precision |
|--------|:---:|:---:|-------|-----------|
| FP32 | 8 | 23 | ±3.4×10³⁸ | ~7 decimal digits |
| FP16 | 5 | 10 | ±6.5×10⁴ | ~3.3 decimal digits |
| **BF16** | **8** | **7** | **±3.4×10³⁸** | **~2.4 decimal digits** |

**Why BF16 over FP16.** BF16 has the same 8-bit exponent as FP32, giving it an identical dynamic range. This eliminates the overflow/underflow problems that plague FP16 training and removes the need for loss scaling [17]. The reduced mantissa precision (7 vs. 23 bits) introduces slightly more rounding noise, but this is empirically negligible for transformer training and is compensated by BF16's superior numerical stability [18].

**BF16 in the QLoRA context.** The base model weights are stored in NF4 (4 bits) and dequantized to BF16 on-the-fly for computation. LoRA adapter weights are stored, trained, and computed in BF16 throughout. The optimizer states (Adam first/second moments) are maintained in FP32 for numerical stability — but only for the LoRA parameters (~160M), not the full model (~8.2B), making this feasible.

### 7.4 Flash Attention 2

**Configuration:** `attn_implementation="flash_attention_2"`

Flash Attention [19] and its successor Flash Attention-2 [20] implement IO-aware exact attention computation that avoids materializing the full N×N attention matrix in GPU HBM (High Bandwidth Memory):

- **Standard attention:** Computes S = QK^T (O(N²) memory), applies softmax, multiplies by V. The N×N matrix must be stored in HBM.
- **Flash Attention:** Uses **tiling** to split Q, K, V into blocks that fit in SRAM (on-chip memory), computing attention block-by-block without ever materializing the full attention matrix in HBM.

**Flash Attention-2 improvements [20]:**
1. Reduced non-matmul FLOPs (which dominate latency on modern GPUs where GEMM throughput >> memory throughput)
2. Parallelization across both the sequence length dimension and batch/head dimensions
3. Achieves 50–73% of theoretical peak GPU FLOP/s (vs. ~25–40% for standard attention)

**Impact on our pipeline:**
- **Memory:** For seq_len = 4096, the attention matrix is 4096 × 4096 × 32 heads × 4 bytes (FP32) = 2 GB per layer per sample. Flash Attention reduces this to O(N) ≈ a few MB.
- **Speed:** Approximately 2× wall-clock speedup for attention computation, which constitutes 30–50% of total training time for transformer models.
- **Correctness:** Flash Attention computes *exact* attention (not an approximation) — the outputs are mathematically identical to standard attention up to floating-point rounding.

### 7.5 Gradient Checkpointing

**Configuration:** `gradient_checkpointing=True, gradient_checkpointing_kwargs={"use_reentrant": False}`

Gradient checkpointing [21] trades compute for memory by selectively discarding intermediate activations during the forward pass and recomputing them during backpropagation:

- **Without checkpointing:** All activations for all L layers are stored → O(L) memory
- **With checkpointing:** Only checkpoint activations at every √L layers → O(√L) memory, with ~33% extra compute for recomputation

For Qwen3-8B with 36 layers and seq_len 4096:
- **Without checkpointing:** ~24 GB activation memory
- **With checkpointing:** ~6–8 GB activation memory

The `use_reentrant=False` flag enables the non-reentrant variant of checkpointing, which is compatible with LoRA's frozen parameters and avoids bugs with gradient computation for parameters that do not require gradients.

**QLoRA-specific benefit.** Dettmers et al. [13] note that with gradient checkpointing, LoRA input gradients drop from 567 MB to ~18 MB per sequence for a 7B model, because only the small LoRA adapter gradients need to be propagated through the recomputed segments.

### 7.6 Sequence Length

**Configuration:** `max_length=4096`

The maximum sequence length of 4,096 tokens was chosen based on the dataset token distribution analysis (§3.2):

| Coverage | Token threshold |
|----------|----------------|
| 95th percentile | ~1,200 |
| 99th percentile | ~2,800 |
| Maximum | ~3,900 |
| **Configured max_length** | **4,096** |

This covers **100% of samples** in the dataset without truncation. Setting max_length below 4,096 would silently truncate the longest samples — which are typically the most complex multi-KPI configurations — degrading the model's ability to learn these critical cases.

Sequences shorter than 4,096 tokens are padded to the right with the pad token. The loss is computed only on non-padded, non-masked tokens (see §7.8).

**Note on the parameter name.** In TRL v1.3+, the sequence length parameter for `SFTConfig` is `max_length`, not the deprecated `max_seq_length` from earlier versions.

### 7.7 Batch Size & Gradient Accumulation

**Configuration:** `per_device_train_batch_size=4, gradient_accumulation_steps=8` → **effective batch size = 32**

The effective batch size is the product of per-device batch size and gradient accumulation steps:

```
effective_batch = per_device_batch × gradient_accumulation × num_gpus
                = 4 × 8 × 1 = 32
```

**Per-device batch size = 4.** This is the largest batch that fits in VRAM alongside the quantized model, LoRA adapters, optimizer states, and checkpointed activations (see §9 for full VRAM budget). Each sample at max_length 4096 with BF16 activations requires approximately 1.5–2 GB for forward+backward pass with gradient checkpointing.

**Gradient accumulation = 8.** Gradient accumulation simulates a larger batch by accumulating gradients over 8 forward-backward passes before performing a single optimizer step. This is mathematically equivalent to training with batch size 32 (assuming negligible batch normalization effects, which is the case for RMSNorm used in Qwen3).

**Effective batch size = 32 justification:**
- **Too small (< 16):** Noisy gradient estimates lead to training instability, especially harmful for structured generation where precise field-level learning requires stable gradients
- **Too large (> 64):** Generalization degradation observed in LLM fine-tuning [13]; also wastes compute per epoch
- **32:** Consistent with QLoRA [13] (used 16–32 for Guanaco), NEFMind [3] (effective batch 32), and the general recommendation for SFT tasks [9]

**Training steps calculation:**
```
steps_per_epoch = ceil(39,294 / 32) = 1,229
total_steps = 1,229 × 3 epochs = 3,687
```

### 7.8 Assistant-Only Loss Masking

**Configuration:** `assistant_only_loss=True`

The cross-entropy loss is computed **only on assistant (completion) tokens**, with system and user (prompt) tokens masked using `label_id = -100` (the PyTorch `CrossEntropyLoss` ignore index).

Given a sample with tokens [t₁, t₂, ..., t_n] where tokens t₁ through t_p are the system+user prompt and t_{p+1} through t_n are the assistant completion:

```
L = -1/(n-p) · Σ_{i=p+1}^{n} log P(t_i | t_1, ..., t_{i-1})
```

**Justification:**

1. **Standard practice.** Completion-only loss was established by InstructGPT [22] in its SFT stage and codified by Stanford Alpaca [23] via `IGNORE_INDEX = -100` masking. The TRL library implements this through the `DataCollatorForCompletionOnlyLM` [9], activated by the `assistant_only_loss=True` flag.

2. **Task-specific rationale.** In our pipeline, the system prompt (which specifies the target standard and schema) and user intent (natural language) are inputs that the model should *condition on* but not *predict*. Computing loss on these tokens would dilute the training signal: the model would spend gradient capacity learning to reproduce prompt text rather than learning the intent→config mapping.

3. **Loss signal concentration.** The assistant completions (JSON configurations) comprise approximately 60–70% of the total token count. Without masking, the model would receive loss signal on ~100% of tokens, but only 60–70% would be task-relevant. With masking, 100% of the loss signal is directed toward the structured generation task.

4. **Counterpoint.** Shi et al. [24] argue that including loss on instruction tokens can reduce overfitting when the instruction-to-output length ratio is high. However, in our case, the ratio is approximately 0.4–0.5 (prompts are shorter than completions), making the overfitting risk low and the benefits of concentrated loss signal outweigh the regularization benefit.

### 7.9 Data Format: ChatML

The dataset uses the ChatML (Chat Markup Language) format, tokenized via `tokenizer.apply_chat_template()`:

```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_intent}<|im_end|>
<|im_start|>assistant
{json_configuration}<|im_end|>
```

ChatML was introduced by OpenAI as an engineering standard for multi-turn conversation formatting and has been adopted by most modern open-source models including Qwen3 [10], Mistral, and the Llama 3 family. Qwen3's tokenizer natively defines the special tokens `<|im_start|>` (ID: 151644) and `<|im_end|>` (ID: 151645) in its vocabulary, ensuring lossless tokenization of turn boundaries.

**Why ChatML over alternatives:**
- **Native support:** No special token additions or tokenizer modifications needed
- **Clear turn boundaries:** The `<|im_start|>` / `<|im_end|>` delimiters allow precise identification of where assistant responses begin and end — essential for loss masking (§7.8)
- **Standard compatibility:** The same format is used by the Qwen3 instruct models' post-training, meaning our fine-tuning operates in the same format the base model was optimized for

### 7.10 Training Epochs

**Configuration:** `num_train_epochs=3`

Three epochs means the model sees each training sample exactly three times. This choice balances:

- **Underfitting risk (too few epochs).** With 39,294 training samples and the model needing to learn 6 distinct JSON schemas + lifecycle operations + adversarial patterns, a single pass is insufficient for convergence — particularly for minority classes (e.g., HMTC slice type, adversarial samples).

- **Overfitting risk (too many epochs).** With LoRA's limited parameter count (~160M trainable out of 8.2B), overfitting manifests as memorization of training JSON templates rather than learning generalizable intent-to-structure mappings. The eval loss should be monitored (§7.11) to detect this.

- **Empirical precedent.**
  - QLoRA [13]: 1 epoch for Guanaco (large-scale instruction data)
  - NEFMind [3]: "iterative" training (not specified, but small dataset of 765)
  - TelecomGPT [6]: 3 epochs for SFT stage
  - General SFT best practices [9]: 1–5 epochs depending on dataset size

For our dataset size (~40K samples, ~27M tokens), 3 epochs provides approximately **81M tokens** of training signal — sufficient for stable convergence without significant overfitting, given the regularization from LoRA dropout (§6.5), weight decay (§7.1), and the limited adapter capacity.

### 7.11 Checkpoint Strategy & Early Stopping

**Configuration:**
```python
save_strategy="steps"
save_steps=307          # ~4 saves per epoch
eval_strategy="steps"
eval_steps=307          # evaluation every ~307 steps
save_total_limit=3      # keep only 3 most recent checkpoints
load_best_model_at_end=True
metric_for_best_model="eval_loss"
greater_is_better=False
```

**Evaluation frequency.** Evaluation is performed approximately 4 times per epoch (`eval_steps = ceil(steps_per_epoch / 4)`). This provides sufficient granularity to detect overfitting onset (eval loss begins increasing while train loss continues decreasing) without excessive evaluation overhead. Each evaluation pass over 2,521 test samples takes approximately 5–10 minutes.

**Best model selection.** After training completes, the checkpoint with the **lowest validation loss** is automatically loaded and saved as the final model. This implements a form of early stopping — if the model overfits in later epochs, the best-performing earlier checkpoint is preserved.

**Disk management.** `save_total_limit=3` retains only the three most recent checkpoints, preventing disk overflow. Each QLoRA checkpoint is approximately 320 MB (adapter weights + optimizer states), so the maximum disk usage is ~1 GB for checkpoints.

---

## 8. Evaluation Methodology

The evaluation pipeline (`evaluate.py`) implements five complementary metrics that assess different aspects of intent translation quality. This multi-dimensional evaluation follows the decomposition proposed by Deng et al. [1], who argue that single-metric evaluation of structured LLM outputs is insufficient because "the Structure Gap manifests across multiple dimensions: syntactic validity, schema compliance, content accuracy, and robustness."

### 8.1 JSON Syntactic Validity

**Metric:** Fraction of model outputs that parse as valid JSON.

```
JSON_Validity = |{y_i : parse(y_i) succeeds}| / N
```

This is the most fundamental requirement: if the output is not valid JSON, it cannot be processed by any downstream telecom system. The implementation attempts parsing in three stages:

1. **Direct `json.loads()`** on the raw output
2. **Markdown fence removal** (strip ` ```json ``` ` wrappers) then re-parse
3. **Regex extraction** of the first `{...}` block, then parse

This multi-stage approach accounts for common LLM generation artifacts (markdown formatting, preamble text) while remaining strict about JSON validity itself.

**Citation context.** JSONSchemaBench [25] established JSON validity rate as the primary metric for structured generation evaluation, showing that even state-of-the-art constrained decoding frameworks exhibit significant validity gaps on complex schemas. RL-Struct [1] formalizes this as R_val — a binary reward signal (1 if parseable, 0 otherwise).

### 8.2 Structural Schema Correctness

**Metric:** Fraction of valid JSON outputs that contain the correct root-level keys for the target telecom standard.

```
Structure_Correctness = |{y_i : valid(y_i) ∧ has_expected_keys(y_i, layer_i)}| / N
```

The expected root keys per standard are:

| Target Layer | Required Root Keys |
|---|---|
| TMF921 | `id`, `href`, `name`, `intentExpression` |
| 3GPP (intent_3gpp) | `intent` |
| CAMARA | `networkSliceBooking` |
| ETSI ZSM | `zsmIntent` |
| O-RAN A1 Policy | `a1Policy` |
| 3GPP O1 NRM | `managedElement` |
| Adversarial | `status` ∈ {CLARIFICATION_REQUIRED, OUT_OF_SCOPE, INTENT_VALIDATION_FAILED} |
| Lifecycle | `intentPatch` or `intentAssuranceReport` or `intentUpdate` |

This metric captures whether the model has learned the *structural identity* of each standard — a necessary condition for downstream system compatibility. A JSON output can be syntactically valid but structurally incorrect (e.g., producing a TMF921 schema when the target was CAMARA).

**Citation context.** RL-Struct [1] introduces R_struct as a distinct metric from R_val: *"structural accuracy measures whether required keys are present and correctly nested, independent of value correctness."* ORION [2] evaluates structural correctness implicitly through its MCP tool-use quality metric, which checks "argument structure" as one component.

### 8.3 KPI Field Extraction Accuracy

**Metric:** For each of 5 KPI fields (latency_ms, reliability_pct, dl_throughput_mbps, ul_throughput_mbps, max_ues), whether the ground-truth value appears in the generated JSON.

```
KPI_Accuracy(field) = |{y_i : str(ground_truth_field_i) ∈ flatten(y_i)}| / N_applicable
```

Where `flatten()` converts the JSON to a string for substring matching, and N_applicable excludes adversarial and lifecycle samples (which don't have KPI fields).

A composite metric **All_KPIs_Correct** requires all 5 fields to be present and correct simultaneously:

```
All_KPIs = |{y_i : ∀f ∈ {lat, rel, dl, ul, ues}: correct(y_i, f)}| / N_applicable
```

**Why substring matching over exact path matching.** Different telecom standards place KPI values at different JSON paths (e.g., TMF921 nests latency under `intentExpression.intentTargets[].targetThresholds[]`, while CAMARA places it under `networkSliceBooking.sliceProfile.latency`). Substring matching on the flattened JSON is standard-agnostic and captures the value regardless of its position in the schema hierarchy.

**Citation context.** This approach is analogous to the R_cor (content correctness) metric in RL-Struct [1], which computes F1 between generated and ground-truth field values. ORION [2] evaluates field-level accuracy through its "value fidelity" component of the MCPUseMetric. NEFMind [3] used BERTScore (0.997–0.998 for fine-tuned model) as a softer measure of content accuracy.

### 8.4 Adversarial Robustness

**Metric:** Fraction of adversarial inputs (ambiguous, out-of-scope, contradictory) correctly rejected with an appropriate error status.

```
Adversarial_Accuracy = |{y_i : valid(y_i) ∧ y_i["status"] ∈ ADVERSARIAL_STATUSES}| / N_adversarial
```

Where `ADVERSARIAL_STATUSES = {CLARIFICATION_REQUIRED, OUT_OF_SCOPE, INTENT_VALIDATION_FAILED}`.

This metric tests the model's ability to *refuse* generation when the input intent is malformed, rather than hallucinating a plausible-looking but incorrect configuration. In production telecom systems, generating a misconfigured network slice from an ambiguous intent is more dangerous than returning an error [26].

**Citation context.** DecodingTrust [27] established adversarial robustness as a core trustworthiness dimension for LLMs. AdvGLUE [28] defines the robustness metric as Acc_adversarial / Acc_clean — a ratio quantifying the degradation from clean to adversarial conditions. SafeCOMM [29] specifically studies safety degradation in telecom-tuned LLMs, finding that domain fine-tuning can reduce the model's ability to refuse harmful requests — directly motivating the inclusion of adversarial samples in our training data.

### 8.5 Greedy Decoding for Evaluation

**Configuration:** `do_sample=False, temperature=None, top_p=None`

All evaluation uses greedy decoding (argmax token selection) rather than sampling. This decision is grounded in:

1. **Reproducibility.** Greedy decoding is deterministic — the same input always produces the same output, enabling reproducible evaluation [30]. Song et al. [31] demonstrated that even with temperature=0, hardware/precision differences can cause up to 9% accuracy variation; any sampling would compound this non-determinism.

2. **Task characteristics.** Intent-to-configuration translation has a *deterministic ground truth* — there is exactly one correct JSON configuration for each intent+standard pair. Song et al. [31] found that *"for reasoning tasks requiring LLMs to solve specific problems with definite solutions, greedy decoding outperforms sampling"* — structured generation with schema constraints falls squarely in this category.

3. **Evaluation convention.** Greedy decoding at temperature 0 is the standard protocol in LLM evaluation, as stated by Yu et al. [30]: *"greedy prediction mode, where temperature is set to 0"* — adopted in DecodingTrust [27], ORAN-Bench-13K [32], and NEFMind [3].

---

## 9. VRAM Budget Analysis

Target hardware: NVIDIA RTX 6000 Ada Generation (48 GB VRAM).

| Component | Memory | Source/Calculation |
|-----------|--------|--------------------|
| Base model weights (NF4 + DQ) | ~4.6 GB | 8.2B × 4 bits / 8 + DQ overhead |
| LoRA adapter weights (BF16) | ~0.32 GB | ~160M × 2 bytes |
| Optimizer states (FP32, LoRA only) | ~1.28 GB | ~160M × 2 states × 4 bytes |
| Gradient storage (BF16, LoRA only) | ~0.32 GB | ~160M × 2 bytes |
| Activations (checkpointed, batch=4) | ~8 GB | ~2 GB/sample × 4, with checkpointing |
| Flash Attention workspace | ~2 GB | IO-aware tiling buffers |
| CUDA context + overhead | ~2 GB | Driver, cuBLAS handles, etc. |
| **Total estimated** | **~18.5 GB** | |
| **Available headroom** | **~29.5 GB** | 48 − 18.5 |

The substantial headroom (>60% of total VRAM) provides a safety margin for:
- Longer sequences (a few samples approaching 4,096 tokens increase activation memory)
- PyTorch memory fragmentation
- Potential for increasing `per_device_train_batch_size` to 8 (estimated +6 GB)

---

## 10. Software Stack & Reproducibility

### 10.1 Dependencies

| Package | Minimum Version | Role |
|---------|----------------|------|
| `torch` | ≥ 2.4.0 | Core tensor operations, autograd |
| `transformers` | ≥ 4.46.0 | Model loading, tokenizer, trainer base class |
| `trl` | ≥ 1.3.0 | `SFTTrainer`, `SFTConfig`, chat template handling |
| `peft` | ≥ 0.15.0 | `LoraConfig`, `PeftModel`, adapter management |
| `datasets` | ≥ 3.0.0 | Dataset loading from HuggingFace Hub |
| `bitsandbytes` | ≥ 0.45.0 | NF4 quantization, 4-bit matrix multiplication |
| `accelerate` | ≥ 1.0.0 | Device mapping, mixed precision, distributed training |
| `flash-attn` | ≥ 2.7.0 | Flash Attention 2 CUDA kernels |
| `scipy` | — | Statistical utilities for evaluation |

### 10.2 Reproducibility Controls

- **Random seed = 42** across all sources of randomness (Python, NumPy, PyTorch, CUDA)
- **Deterministic data loading** via `datasets` library with fixed shuffle seeds
- **Greedy decoding** for evaluation (§8.5)
- **Fixed LoRA initialization** (B = 0 ensures training starts from exact pre-trained weights)
- **Pinned dependency versions** in `requirements.txt`

### 10.3 Logging

**Configuration:** `logging_strategy="steps", logging_steps=10, logging_first_step=True, disable_tqdm=False`

Training loss is logged every 10 optimizer steps (320 samples) as plain text to stdout, enabling both real-time monitoring and post-hoc analysis via `grep "loss" training.log`. The first step is explicitly logged to verify training begins at a reasonable loss value (expected: ~2.5–4.0 for cross-entropy on a 151K vocabulary).

---

## 11. Limitations & Threats to Validity

### 11.1 Dataset Limitations

1. **Adversarial response homogeneity** (§3.5): All samples within each adversarial category share identical responses. The model may achieve high adversarial accuracy by memorizing three templates rather than learning contextual rejection reasoning.

2. **Single-standard lifecycle operations:** Lifecycle management is only represented for TMF921. The model cannot be expected to generalize lifecycle operations to other standards.

3. **Synthetic data provenance:** The dataset is synthetically generated (not derived from real network deployments). The distribution of KPI values, region-sector combinations, and use cases may not reflect production telecom traffic patterns.

4. **English-only intents:** All intents are in English. Multilingual intent translation (leveraging Qwen3's 119-language pre-training) is untested.

### 11.2 Methodological Limitations

5. **Substring matching for KPI evaluation** (§8.3): This approach cannot distinguish between a KPI value appearing in the correct JSON field vs. appearing elsewhere in the output (e.g., in a comment field or wrong KPI slot). A more rigorous evaluation would use JSONPath queries against the expected schema.

6. **No semantic equivalence checking:** Two JSON configurations can be structurally different but semantically equivalent (e.g., different field ordering, equivalent but differently formatted values). Our evaluation treats these as incorrect.

7. **Single base model:** We evaluate only Qwen3-8B. Comparative studies with Llama 3.1-8B, Mistral-7B, and Phi-4 would strengthen the generalizability claims.

8. **No inference latency benchmarking:** For production intent translation, latency matters. We do not report inference time per intent.

### 11.3 Threats to External Validity

9. **Standard version drift:** Telecom standards evolve rapidly. The TMF921 v5.0, 3GPP Rel-18, and CAMARA schemas used in this dataset may become outdated as new releases appear.

10. **No closed-loop validation:** The generated configurations are not validated against actual network orchestration systems (e.g., ONAP, O-RAN SMO). A structurally correct JSON may still fail deployment due to semantic constraints not captured in our evaluation.

---

## 12. References

[1] Y. Deng et al., "RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs," *arXiv:2512.00319*, 2024.

[2] E. McMahon et al., "ORION: A Holistic End-to-End AI Framework for Intent-Aware Orchestration in O-RAN," *arXiv:2603.03667*, 2025.

[3] S. Niknam et al., "NEFMind: LLM-Driven 5G Network Exposure — A Fine-Tuning Approach," *arXiv:2508.09240*, 2025.

[4] A. Duarte da Costa et al., "Hermes: A Large Language Model Framework on the Journey to Autonomous Networks," *arXiv:2411.06490*, 2024.

[5] L. N. T. Huynh et al., "When LLMs Meet Network Slicing: An LLM-Based Framework for Intent-Driven Network Slicing Management," *arXiv:2403.13721*, 2024.

[6] Z. Zhou et al., "TelecomGPT: A Framework to Build Telecom-Specific Large Language Models," *arXiv:2407.09424*, 2024.

[7] A. Maatouk et al., "ORANSight-2.0: Refining Open Radio Access Networks Alignment with the RANSTRUCT Dataset," *arXiv:2503.05200*, 2025.

[8] P. Mahi et al., "A Comprehensive Survey on the Role of Generative AI in Network Monitoring and Management," *arXiv:2502.08576*, 2025.

[9] L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec, "TRL: Transformer Reinforcement Learning," GitHub, 2020. [Online]. Available: https://github.com/huggingface/trl

[10] Qwen Team, "Qwen3 Technical Report," *arXiv:2505.09388*, 2025.

[11] N. Shazeer, "GLU Variants Improve Transformer," *arXiv:2002.05202*, 2020.

[12] J. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints," *arXiv:2305.13245*, 2023.

[13] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "QLoRA: Efficient Finetuning of Quantized LLMs," *arXiv:2305.14314*, NeurIPS, 2023.

[14] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "LoRA: Low-Rank Adaptation of Large Language Models," *arXiv:2106.09685*, ICLR, 2022.

[15] I. Loshchilov and F. Hutter, "Decoupled Weight Decay Regularization," *arXiv:1711.05101*, ICLR, 2019.

[16] I. Loshchilov and F. Hutter, "SGDR: Stochastic Gradient Descent with Warm Restarts," *arXiv:1608.03983*, ICLR, 2017.

[17] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, and H. Wu, "Mixed Precision Training," *arXiv:1710.03740*, ICLR, 2018.

[18] D. Kalamkar et al., "A Study of BFLOAT16 for Deep Learning Training," *arXiv:1905.12322*, 2019.

[19] T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness," *arXiv:2205.14135*, NeurIPS, 2022.

[20] T. Dao, "FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning," *arXiv:2307.08691*, ICLR, 2024.

[21] T. Chen, B. Xu, C. Zhang, and C. Guestrin, "Training Deep Nets with Sublinear Memory Cost," *arXiv:1604.06174*, 2016.

[22] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., "Training language models to follow instructions with human feedback," *arXiv:2203.02155*, NeurIPS, 2022.

[23] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, "Stanford Alpaca: An Instruction-Following LLaMA Model," GitHub, 2023. [Online]. Available: https://github.com/tatsu-lab/stanford_alpaca

[24] Z. Shi, A. X. Yang, B. Wu, L. Aitchison, E. Yilmaz, and A. Lipani, "Instruction Tuning With Loss Over Instructions," *arXiv:2405.14394*, 2024.

[25] G. Geng et al., "Generating Structured Outputs from Language Models: Benchmark and Studies," *arXiv:2501.10868*, 2025.

[26] A. Fressancourt and A. Mahi, "SafeCOMM: Evaluating the Impact of Fine-tuning on Safety Capabilities of LLMs for Telecommunications Applications," *arXiv:2506.00062*, 2025.

[27] B. Wang et al., "DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models," *arXiv:2306.11698*, NeurIPS, 2023.

[28] B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li, "Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models," *arXiv:2111.02840*, 2021.

[29] A. Fressancourt and A. Mahi, "SafeCOMM: Evaluating the Impact of Fine-tuning on Safety Capabilities of LLMs for Telecommunications Applications," *arXiv:2506.00062*, 2025.

[30] T. Yu et al., "Give Me FP32 or Give Me Death? Challenges and Solutions for Reproducible Reasoning," *arXiv:2506.09501*, 2025.

[31] Y. Song et al., "The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism," *arXiv:2407.10457*, 2024.

[32] A. Maatouk et al., "ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks," *arXiv:2407.06245*, 2024.

[33] S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, "PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods," GitHub, 2022. [Online]. Available: https://github.com/huggingface/peft

[34] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," *arXiv:2208.07339*, NeurIPS, 2022.

[35] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned Language Models Are Zero-Shot Learners," *arXiv:2109.01652*, ICLR, 2022.

[36] Qwen Team, "Qwen2.5 Technical Report," *arXiv:2412.15115*, 2024.

[37] A. Narayanan et al., "HRL for Intent-Driven O-RAN xApp Orchestration," *arXiv:2307.02754*, 2023.

[38] Y. Liu et al., "An AI/ML-Driven SMO Framework for O-RAN," *arXiv:2409.05092*, 2024.

---

*Document generated for [nraptisss/intent-translation-training](https://huggingface.co/nraptisss/intent-translation-training). Last updated: April 2025.*