File size: 6,992 Bytes

---
license: apache-2.0
tags:
- qlora
- sft
- trl
- peft
- qwen3
- tmf921
- intent-based-networking
- network-slicing
- rtx-6000-ada
- ml-intern
base_model:
- Qwen/Qwen3-8B
datasets:
- nraptisss/TMF921-intent-to-config-research-sota
---

# TMF921 Intent-to-Config Training + Evaluation

Training and evaluation repo for [`nraptisss/TMF921-intent-to-config-research-sota`](https://huggingface.co/datasets/nraptisss/TMF921-intent-to-config-research-sota) on a single **RTX 6000 Ada 48/50GB** server.

The default recipe is **Qwen3-8B + QLoRA NF4 + TRL SFTTrainer + PEFT LoRA**.

## Why this recipe

- Dataset rows were audited with `Qwen/Qwen3-8B` chat-template tokenization.
- Source max length: **1,316 tokens**, p99: **1,300**, so `max_length=2048` is safe.
- QLoRA NF4 + double quant follows the QLoRA recipe for fitting large models on one 48GB-class GPU.
- LoRA uses `target_modules="all-linear"`, recommended for QLoRA-style training.
- `assistant_only_loss=True` trains only the JSON/config response tokens.
- Evaluation is split by in-distribution and OOD splits; do not report only a single merged score.

## Hardware target

Recommended server:

- GPU: NVIDIA RTX 6000 Ada, 48GB/50GB VRAM
- RAM: 64GB+
- Disk: 200GB+ free
- CUDA-compatible PyTorch

Default effective batch size:

```text
per_device_train_batch_size = 2
gradient_accumulation_steps = 8
effective batch size = 16
max_length = 2048
```

If OOM occurs, preserve the effective batch size by changing:

```yaml
per_device_train_batch_size: 1
gradient_accumulation_steps: 16
```

Do **not** reduce `max_length` unless you intentionally want a different training task.

## Quick start with nohup, unique run dirs, and resumable checkpoints

```bash
git clone https://huggingface.co/nraptisss/tmf921-intent-training
cd tmf921-intent-training

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
bash scripts/install_rtx6000ada.sh
python scripts/check_gpu.py

export HF_TOKEN=hf_...
export CUDA_VISIBLE_DEVICES=0
export PYTHONPATH="$PWD/src"
export TOKENIZERS_PARALLELISM=false

bash scripts/nohup_new_run.sh
```

Monitor:

```bash
RUN_DIR=runs/qwen3-8b-qlora-YYYYMMDD-HHMMSS
bash scripts/status_run.sh "$RUN_DIR"
tail -f "$RUN_DIR/logs/train.log"
watch -n 2 nvidia-smi
```

Resume:

```bash
bash scripts/nohup_resume.sh runs/qwen3-8b-qlora-YYYYMMDD-HHMMSS
```

Evaluate:

```bash
bash scripts/nohup_eval.sh runs/qwen3-8b-qlora-YYYYMMDD-HHMMSS
```

## Configs

- `configs/rtx6000ada_qwen3_8b_qlora.yaml` — recommended stage-1 config
- `configs/rtx6000ada_qwen3_14b_qlora_experimental.yaml` — experimental 14B config
- `configs/stage2_weak_layer_qwen3_8b.yaml` — diagnostic weak-layer continuation config

## Evaluation

Raw evaluator:

```bash
python scripts/evaluate_model.py \
  --model Qwen/Qwen3-8B \
  --adapter outputs/qwen3-8b-tmf921-qlora \
  --dataset nraptisss/TMF921-intent-to-config-research-sota \
  --output_dir outputs/qwen3-8b-tmf921-qlora/eval \
  --load_in_4bit
```

Normalize existing predictions:

```bash
python scripts/normalize_eval_metrics.py \
  --eval_dir outputs/qwen3-8b-tmf921-qlora/eval
```

Metrics:

- JSON parse rate
- canonical JSON exact match
- field precision / recall / F1
- normalized field precision / recall / F1
- normalized key precision / recall / F1
- slice/SST diagnostic pass
- KPI text-presence diagnostic pass
- adversarial status pass
- stratified metrics by `target_layer`, `slice_type`, and `lifecycle_operation`

## Merge adapter for deployment/evaluation

```bash
python scripts/merge_adapter.py \
  --base_model Qwen/Qwen3-8B \
  --adapter outputs/qwen3-8b-tmf921-qlora \
  --output_dir outputs/qwen3-8b-tmf921-merged
```

## Stage 2 weak-layer continuation

Stage 2 was implemented and tested as a diagnostic experiment. It is **not promoted** as the main model because it did not materially improve O1/A1 and slightly regressed adversarial performance.

Run if needed:

```bash
bash scripts/nohup_stage2_weak.sh runs/qwen3-8b-qlora-YYYYMMDD-HHMMSS
```

## Results packaging and qualitative failure analysis

After completing stage-1 and stage-2 evaluation plus normalization, package publication artifacts with:

```bash
export PYTHONPATH="$PWD/src"

python scripts/package_results.py \
  --stage1_eval_dir runs/qwen3-8b-qlora-20260501-083834/eval_merged \
  --stage2_eval_dir runs/stage2-weak-20260505-080040/eval \
  --output_dir results
```

This writes:

```text
results/stage1_raw_metrics.json
results/stage1_normalized_metrics.json
results/stage2_raw_metrics.json
results/stage2_normalized_metrics.json
results/metrics_summary.json
results/stage1_vs_stage2_comparison.md
```

Generate qualitative success/failure examples for the paper with:

```bash
python scripts/sample_failure_examples.py \
  --eval_dir runs/qwen3-8b-qlora-20260501-083834/eval_merged \
  --output_dir analysis/stage1_examples
```

Optionally also sample stage-2 examples:

```bash
python scripts/sample_failure_examples.py \
  --eval_dir runs/stage2-weak-20260505-080040/eval \
  --output_dir analysis/stage2_examples
```

The example sampler writes:

```text
analysis/*/failure_examples.md
analysis/*/failure_examples.json
```

These artifacts are intended for paper tables, qualitative error analysis, and reproducibility appendices.

## Scientific reporting protocol

For research papers/reports, report at least:

1. validation loss,
2. `test_in_distribution` metrics,
3. `test_template_ood` metrics,
4. `test_use_case_ood` metrics,
5. `test_sector_ood` metrics,
6. `test_adversarial` metrics,
7. per-target-layer field F1,
8. normalized field/key F1,
9. JSON parse rate,
10. rare-class metrics for lifecycle operations and adversarial categories.

Do **not** claim production standards compliance from JSON validity alone. Official TMF921/3GPP/ETSI/CAMARA/O-RAN validators are still needed for schema-level certification.

## Files

```text
configs/
scripts/
src/tmf921_train/
PROJECT_JOURNAL.md
requirements.txt
```

## References

- QLoRA: https://huggingface.co/papers/2305.14314
- LoRA: https://huggingface.co/papers/2106.09685
- TRL SFTTrainer docs: https://huggingface.co/docs/trl/sft_trainer
- TRL PEFT integration: https://huggingface.co/docs/trl/peft_integration
- Source dataset: https://huggingface.co/datasets/nraptisss/TMF921-intent-to-config-research-sota

<!-- ml-intern-provenance -->
## Generated by ML Intern

This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.

- Try ML Intern: https://smolagents-ml-intern.hf.space
- Source code: https://github.com/huggingface/ml-intern

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'nraptisss/tmf921-intent-training'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
```

For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.