---
license: apache-2.0
language:
- en
- kk
base_model:
- issai/foggen
datasets:
- issai/foggen-data
- issai/KazCulture
- rajpurkar/squad
- truthfulqa/truthful_qa
- openai/gsm8k
pipeline_tag: text-generation
tags:
- edge-cloud-routing
- verbalized-confidence
- self-aware
- routing
- open-ended
- continual-learning
library_name: transformers
---

# FogGen R15-OE: open-ended task-type extension of FogGen

> **One additional round of FogGen training on top of [issai/foggen](https://huggingface.co/issai/foggen) extending the recipe to three open-ended task types: extractive reading comprehension, adversarial factual recall, and chain-of-thought math.**

This is the open-ended ablation from the FogGen paper. It is **not** a chain continuation; it's a parallel branch from the R14 chain endpoint, used to demonstrate that the recipe extends beyond MCQ. For deployment on MCQ tasks the canonical model is [`issai/foggen`](https://huggingface.co/issai/foggen). This ablation model is for reproducibility and open-ended use cases.

## Training

- **Base**: R14 merged (the canonical chain endpoint, [issai/foggen](https://huggingface.co/issai/foggen))
- **Round**: one additional SFT round, same recipe as every R8–R14 chain round
- **SFT pool (1876 rows)**:
  - 133 SQuAD v1.1 train rows (extractive RC, English)
  - 91 TruthfulQA val rows (adversarial factual recall, English)
  - 152 GSM8K train rows (chain-of-thought math, English), using the model's own correct reasoning traces as SFT targets
  - + 1500 R14 replay rows
- **Hyperparameters**: identical to R14 chain (LoRA r=16 α=32 all-linear, bf16, 2 epochs, lr=5e-5)
- **Training cost**: 49 seconds on 4× H100

## Results

Local accuracy gains over R14 zero-shot:

| Task | R14 raw | R15-OE raw | Δ |
|---|---|---|---|
| SQuAD v1.1 | 81.0% | 86.5% | +5.5 |
| TruthfulQA-gen | 36.5% | 40.0% | +3.5 |
| GSM8K (CoT) | 52.0% | 58.0% | +6.0 |
| **Mean** | **56.5%** | **61.5%** | **+5.0** |

MCQ chain capability preservation (mean across 7 in-chain MCQ domains): 55.6% raw at R15-OE vs. 57.3% at R14 (within inter-round variance).

Routing lift over Random at τ=0.5:
- SQuAD CA: R14 +1.4 → R15-OE +0.4 (headroom collapse: raw acc closes to cloud)
- TruthfulQA CA: R14 −0.7 → R15-OE −2.3 (anti-calibrated, dataset-inherent; see Tian et al., 2023)
- GSM8K CoT: R14 +2.2 → **R15-OE +5.5** (routing improves with training)

## How this differs from issai/foggen

| | `issai/foggen` (R14) | `issai/foggen-r15-oe` (this) |
|---|---|---|
| Use case | MCQ deployment | Open-ended task types + reproducibility |
| Training rounds | 14 sequential | 14 + 1 OE extension |
| Trained tasks | 7 MCQ domains | 7 MCQ + SQuAD + TQA + GSM8K |
| MCQ acc (mean) | 57.3% raw | 55.6% raw |
| Chain narrative | canonical endpoint | parallel ablation branch |

## Quick demo

Same usage pattern as `issai/foggen` for MCQ. For open-ended tasks use the per-task system prompts described in the paper.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "issai/foggen-r15-oe", torch_dtype="bfloat16", device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("issai/foggen-r15-oe")

# Example: GSM8K with chain-of-thought
SYSTEM_GSM = """You are a self-aware math assistant.

Rules:
- Do not output <think> tags.
- First, show your step-by-step reasoning to solve the problem.
- Then assess your confidence in your answer.
- Then give your final numeric answer.
- Output format:
  Reasoning: <your step-by-step work>
  Confidence: <0.0|0.25|0.5|0.75|1.0>
  Final answer: <NUMBER>"""
```

SQuAD and TruthfulQA use task-specific system prompts; the exact strings are stored alongside the SFT data in [`issai/foggen-data`](https://huggingface.co/datasets/issai/foggen-data).

## Citation

Paper coming soon.