docs/adversarial_self_play.md

Adversarial Self-Play Training (Kimi-Style + TRL)

This repository now includes a code scaffold for alternating adversarial self-play with Hugging Face TRL.

Goal

Train two policies in alternating rounds:

• Generator policy: proposes hard OSINT tasks (question + answer + supporting edges).
• Answerer policy: solves tasks proposed by the generator.

The loop is intended to move from static evaluation toward on-policy co-evolution.

Kimi-style Objective Mapping

The implementation maps the requested Kimi-style ingredients onto TRL GRPO as follows:

• Grouped rollouts: num_generations in each GRPO phase.
• Relative reward baseline: GRPO group-relative advantages.
• Clipped policy updates: epsilon clipping in GRPO objective.
• KL/reference regularization: beta in GRPOConfig.
• Token-level online RL behavior: GRPO online generation with reward functions.
• Toggle schedule: explicit alternating generator and answerer rounds.

Topology and Scheduling Options

• model_topology: "dual": train separate generator and answerer models.
• model_topology: "shared": train one shared model for both roles.
  • Use shared_model_name_or_path to set the common base checkpoint.
• phase_schedule: "generator_answerer": default two-phase loop per round.
• phase_schedule: "answerer_generator_answerer": solver-first curriculum:
  1. Train answerer on current adversarial pool.
  2. Freeze that answerer snapshot while training generator against it.
  3. Train answerer again on newly generated adversarial tasks.

This directly supports the "train solver, freeze, attack, retrain solver" sequence.

Canonical Graph Mode

• canonical_graph_mode: "generate" (default): generator can propose canonical graph updates in swarm_v2.
• canonical_graph_mode: "fixed": canonical graph candidates are held fixed per prompt, so training focuses on question/answer behavior over stable graph structure.

Tuning Modes

• tuning_mode: "full": full-model GRPO fine-tuning.
• tuning_mode: "lora": PEFT LoRA adapters for GRPO updates.
  • Configure via lora block: r, alpha, dropout, target_modules, bias, task_type.

Reward Design

Generator (adversarial swarm)

GeneratorRewardFunction combines weighted components:

• Validity: checks parsable task fields and bounded support-edge size.
• Hardness: rewards questions the frozen answerer currently gets wrong.
• Diversity: penalizes near-duplicate questions via token-overlap similarity.
• Consistency: rewards edge/answer/question grounding against canonical graph context.

Weights are configurable in generator_reward_weights.

Answerer (existing reward integration)

AnswererRewardFunction wraps existing environment reward logic:

• Reuses compute_answer_reward from src/osint_env/env/reward.py.
• Builds transient TaskInstance objects from training rows.
• Preserves difficulty-aware reward behavior (easy / medium / hard).

Entry Points

• CLI command: osint-env train-self-play
• Main runner: src/osint_env/training/self_play.py
• Config loader: src/osint_env/training/config.py
• Reward functions: src/osint_env/training/rewards.py
• Example config: config/self_play_training_example.json

Dry Run Mode

The example config sets dry_run: true by default.

In dry run mode, the pipeline still:

• Materializes generator/answerer datasets per round.
• Materializes optional answerer_pre_dataset when using solver-first schedule.
• Produces generated-task artifacts (fallback generator path).
• Writes a full run summary.

But it skips expensive GRPO updates.

Compute Mode

When compute is available:

1. Install train dependencies: python -m pip install -e ".[train]"
2. Disable dry run (--dry-run off and/or "dry_run": false in config).
3. Run osint-env train-self-play.

Outputs are written under artifacts/self_play unless overridden.