reports/eval_report.md

# Evaluation Report

Generated at: 2026-04-25T18:12:09.069260+00:00
Base URL: http://localhost:7860
Tasks: clean_claim, contradictory_claim, coordinated_fraud, distribution_shift_claim, identity_fraud
Seeds: 7, 11, 13, 19, 25
Distinct variant_ids: [0, 1, 2, 3, 4]



| Task | Seed | Variant | Steps | Done | Reward | Evidence Quality | Exploit Penalty |

|---|---:|---:|---:|:---:|---:|---:|---:|

| clean_claim | 7 | 2 | 4 | yes | 0.8725 | 1.0000 | 0.0000 |
| clean_claim | 11 | 1 | 4 | yes | 0.8725 | 1.0000 | 0.0000 |

| clean_claim | 13 | 3 | 4 | yes | 0.8725 | 1.0000 | 0.0000 |
| clean_claim | 19 | 4 | 4 | yes | 0.8725 | 1.0000 | 0.0000 |

| clean_claim | 25 | 0 | 4 | yes | 0.8725 | 1.0000 | 0.0000 |
| contradictory_claim | 7 | 2 | 8 | yes | 0.7497 | 1.0000 | 0.0000 |

| contradictory_claim | 11 | 1 | 8 | yes | 0.7497 | 1.0000 | 0.0000 |
| contradictory_claim | 13 | 3 | 8 | yes | 0.7497 | 1.0000 | 0.0000 |

| contradictory_claim | 19 | 4 | 8 | yes | 0.7497 | 1.0000 | 0.0000 |
| contradictory_claim | 25 | 0 | 8 | yes | 0.7497 | 1.0000 | 0.0000 |

| coordinated_fraud | 7 | 2 | 12 | yes | 0.8230 | 1.0000 | 0.0000 |
| coordinated_fraud | 11 | 1 | 12 | yes | 0.8230 | 1.0000 | 0.0000 |

| coordinated_fraud | 13 | 3 | 12 | yes | 0.8230 | 1.0000 | 0.0000 |
| coordinated_fraud | 19 | 4 | 12 | yes | 0.8230 | 1.0000 | 0.0000 |

| coordinated_fraud | 25 | 0 | 12 | yes | 0.8230 | 1.0000 | 0.0000 |
| distribution_shift_claim | 7 | 2 | 12 | yes | 0.7827 | 1.0000 | 0.0000 |
| distribution_shift_claim | 11 | 1 | 12 | yes | 0.7827 | 1.0000 | 0.0000 |
| distribution_shift_claim | 13 | 3 | 12 | yes | 0.7827 | 1.0000 | 0.0000 |
| distribution_shift_claim | 19 | 4 | 12 | yes | 0.7827 | 1.0000 | 0.0000 |
| distribution_shift_claim | 25 | 0 | 12 | yes | 0.7827 | 1.0000 | 0.0000 |
| identity_fraud | 7 | 2 | 10 | yes | 0.8180 | 1.0000 | 0.0000 |

| identity_fraud | 11 | 1 | 10 | yes | 0.8180 | 1.0000 | 0.0000 |
| identity_fraud | 13 | 3 | 10 | yes | 0.8180 | 1.0000 | 0.0000 |

| identity_fraud | 19 | 4 | 10 | yes | 0.8180 | 1.0000 | 0.0000 |
| identity_fraud | 25 | 0 | 10 | yes | 0.8180 | 1.0000 | 0.0000 |



Average Reward: 0.8092

Completion Rate: 100.00%



> **Note on identical rewards within a task.** Each task above shows the same

> reward across all 5 seeds because this run is a *scripted baseline* — the

> evaluation client follows a fixed strategy per `task_id` (e.g. always call
> `validate_document` then `flag_fraud_signal` then `deny_claim` for
> `contradictory_claim`). The seeds vary the generated documents (different

> claimants, amounts, fraud-signal strengths — see the `Variant` column), but

> the scripted strategy is invariant to that variation, so the env returns the

> same scalar reward every time. This is intentional: the scripted baseline

> exists to demonstrate that the environment's reward surface is deterministic

> and reproducible across seeds, *not* to show learning.

>

> The trained GRPO model produces variable rewards across seeds — see the

> held-out component-shift evaluation in

> [`reports/component_shift_summary.json`](component_shift_summary.json) and

> the README "Held-out evaluation" section.