| --- |
| license: apache-2.0 |
| tags: |
| - temporal-graph-learning |
| - fraud-detection |
| - synthetic-data |
| - benchmark |
| - upi |
| - causal-evaluation |
| - matched-controls |
| - neurips |
| --- |
| |
| # Temporal Twins: A Matched-Control Benchmark for Temporal Fraud Detection |
|
|
| Synthetic UPI-style temporal transaction benchmark where fraud and benign trajectories are matched on static and prefix-level summaries but differ in delayed event-order structure. |
|
|
| ## Links |
|
|
| - Dataset repository: [https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins](https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins) |
| - Code repository: [https://huggingface.co/temporal-twins-benchmark/temporal-twins-code](https://huggingface.co/temporal-twins-benchmark/temporal-twins-code) |
| - Croissant metadata URL: [https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins/raw/main/metadata/temporal_twins_croissant.json](https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins/raw/main/metadata/temporal_twins_croissant.json) |
| - Paper or preprint: Not available during double-blind review; to be added after publication. |
|
|
| ## Installation |
|
|
| Recommended Python: `3.11+` |
|
|
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| If you prefer Conda: |
|
|
| ```bash |
| conda env create -f environment.yml |
| conda activate temporal-twins |
| ``` |
|
|
| ## Repository Structure |
|
|
| - `src/`: synthetic user, transaction, risk, fraud, graph, and temporal benchmark generation code |
| - `models/`: SeqGRU, static baselines, audit/probe models, and temporal GNN wrappers |
| - `experiments/`: deterministic benchmark runner and matched-prefix evaluation utilities |
| - `config/`: base YAML configs used by the experiment runner |
| - `configs/`: release-facing config snapshots for calibration and paper-suite reproduction |
| - `docs/`: determinism and supporting documentation |
| - `metadata/`: MLCommons Croissant metadata and validation notes |
| - `results/`: lightweight frozen paper-suite summaries and interpretation notes |
|
|
| ## Quick Smoke Test |
|
|
| ```bash |
| PYTHONPATH=. python3 experiments/run_all.py \ |
| --fast \ |
| --seed 0 \ |
| --benchmark-mode temporal_twins_oracle_calib \ |
| --experiments audit \ |
| --device cpu |
| ``` |
|
|
| ## Exact Paper-Scale Reproduction |
|
|
| The checked-in CLI exposes `--benchmark-mode`, `--seed`, `--seeds`, `--fast`, `--device`, and `--experiments`, but not separate `--difficulty`, `--num-users`, or `--simulation-days` flags. For the exact grouped paper-scale runs, use the helper below from the repository root. |
|
|
| Define this shell helper once: |
|
|
| ```bash |
| run_group() { |
| local group="$1" |
| local seed="$2" |
| local out_json="$3" |
| |
| PYTHONPATH=. python3 - "$group" "$seed" "$out_json" <<'PY' |
| import json |
| import math |
| import sys |
| import time |
| from pathlib import Path |
| |
| from src.core.config_loader import load_config |
| from experiments.run_all import ( |
| build_gate_pool_from_frames, |
| gate_volume_is_sufficient, |
| generate_single_difficulty, |
| offset_gate_namespace, |
| prepare_gate_subset, |
| run_motif_validity_check, |
| set_global_determinism, |
| ) |
| |
| |
| def normalize(value): |
| if isinstance(value, dict): |
| return {k: normalize(v) for k, v in value.items()} |
| if isinstance(value, (list, tuple)): |
| return [normalize(v) for v in value] |
| if hasattr(value, "item"): |
| try: |
| value = value.item() |
| except Exception: |
| pass |
| if isinstance(value, float) and not math.isfinite(value): |
| return None |
| return value |
| |
| |
| group = sys.argv[1] |
| seed = int(sys.argv[2]) |
| out_json = Path(sys.argv[3]) |
| |
| if group == "oracle_calib": |
| benchmark_mode = "temporal_twins_oracle_calib" |
| difficulty = "easy" |
| hard_abort = True |
| else: |
| benchmark_mode = "temporal_twins" |
| difficulty = group |
| hard_abort = False |
| |
| cfg = load_config("config/default.yaml") |
| cfg = cfg.model_copy( |
| update={ |
| "num_users": 350, |
| "simulation_days": 45, |
| "benchmark_mode": benchmark_mode, |
| "random_seed": seed, |
| } |
| ) |
| |
| set_global_determinism(seed) |
| pool = generate_single_difficulty( |
| cfg, |
| difficulty=difficulty, |
| seed=seed, |
| benchmark_mode=benchmark_mode, |
| ) |
| gate = prepare_gate_subset(pool, seed=seed, fast_mode=False) |
| pack_count = 1 |
| |
| while (not gate_volume_is_sufficient(gate["volume"], False)) and pack_count <= 6: |
| extra_seed = seed + pack_count * 10007 |
| extra_pack = generate_single_difficulty( |
| cfg, |
| difficulty=difficulty, |
| seed=extra_seed, |
| benchmark_mode=benchmark_mode, |
| ) |
| extra_pack = offset_gate_namespace(extra_pack, pack_count) |
| pool = build_gate_pool_from_frames([pool, extra_pack]) |
| gate = prepare_gate_subset(pool, seed=seed, fast_mode=False) |
| pack_count += 1 |
| |
| gate["source_pool_events"] = int(len(pool)) |
| gate["source_pool_pairs"] = int(pool.loc[pool["twin_pair_id"] >= 0, "twin_pair_id"].nunique()) if "twin_pair_id" in pool.columns else 0 |
| gate["source_pool_packs"] = int(pack_count) |
| |
| start = time.time() |
| gate_pass, report = run_motif_validity_check( |
| df=pool, |
| config=cfg, |
| seed=seed, |
| device="cpu", |
| num_epochs=3, |
| node_epochs=150, |
| n_checkpoints=8, |
| hard_abort=hard_abort, |
| benchmark_mode=benchmark_mode, |
| fast_mode=False, |
| force_temporal_models=True, |
| prebuilt_gate=gate, |
| ) |
| elapsed = time.time() - start |
| |
| result = { |
| "benchmark_group": group, |
| "benchmark_mode": benchmark_mode, |
| "seed": seed, |
| "primary_metric_label": report["audit_metric_label"], |
| "secondary_metric_label": report["raw_metric_label"], |
| "gate_pass": bool(gate_pass), |
| "run_wall_time_sec": float(elapsed), |
| **report, |
| } |
| |
| out_json.parent.mkdir(parents=True, exist_ok=True) |
| out_json.write_text(json.dumps(normalize(result), indent=2) + "\n") |
| print(f"Wrote {out_json}") |
| PY |
| } |
| ``` |
|
|
| ### Reproduce `oracle_calib` |
| |
| ```bash |
| run_group oracle_calib 0 results/paper_suite_repro/jobs/oracle_calib_0.json |
| ``` |
| |
| ### Reproduce `easy` |
| |
| ```bash |
| run_group easy 0 results/paper_suite_repro/jobs/easy_0.json |
| ``` |
| |
| ### Reproduce `medium` |
| |
| ```bash |
| run_group medium 0 results/paper_suite_repro/jobs/medium_0.json |
| ``` |
| |
| ### Reproduce `hard` |
| |
| ```bash |
| run_group hard 0 results/paper_suite_repro/jobs/hard_0.json |
| ``` |
| |
| ## Reproduce the Full Paper Suite |
| |
| ```bash |
| mkdir -p results/paper_suite_repro/jobs |
| |
| for group in oracle_calib easy medium hard; do |
| for seed in 0 1 2 3 4; do |
| run_group "$group" "$seed" "results/paper_suite_repro/jobs/${group}_${seed}.json" |
| done |
| done |
| ``` |
| |
| The frozen reference outputs for the final deterministic suite are already included in `results/`: |
|
|
| - `paper_suite_summary.csv` |
| - `paper_suite_summary.md` |
| - `paper_suite_runtime.csv` |
| - `paper_suite_meta.json` |
| - `paper_suite_runs.csv` |
| - `PAPER_GATE_INTERPRETATION.md` |
|
|
| ## Expected Headline Results |
|
|
| | Benchmark | XGBoost ROC-AUC | StaticGNN ROC-AUC | SeqGRU ROC-AUC | SeqGRU Shuffle Delta | |
| | --- | ---: | ---: | ---: | ---: | |
| | `oracle_calib` | `0.5000` | `0.5222` | `1.0000` | `-0.5032` | |
| | `easy` | `0.5000` | `0.4946` | `1.0000` | `-0.5003` | |
| | `medium` | `0.5000` | `0.4922` | `0.8391` | `-0.3337` | |
| | `hard` | `0.5000` | `0.5026` | `0.6876` | `-0.1883` | |
|
|
| ## Determinism |
|
|
| CPU deterministic runtime is enabled. The same seed should reproduce identical matched-prefix data and metrics. Deterministic torch settings can slow runtime, especially for the non-fast paper-scale suite. |
|
|
| ## Data Note |
|
|
| This code repository contains source code, metadata, documentation, and lightweight result summaries only. The generated synthetic dataset and full release artifacts are hosted separately at the dataset repository: |
|
|
| - [https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins](https://huggingface.co/datasets/temporal-twins-benchmark/temporal-twins) |
|
|
| ## Privacy Note |
|
|
| - Synthetic data only |
| - No real UPI transactions |
| - No real users |
| - No real bank accounts |
| - No personal financial records |
|
|
| ## License |
|
|
| - Code: `Apache-2.0` |
| - Dataset and generated benchmark artifacts: `CC-BY-4.0` |
|
|
| ## Citation |
|
|
| Anonymous NeurIPS 2026 submission; final citation to be added after review. |
|
|
